Handling Network Issues on Private Location Machines (Docker Installation)

This article covers known network issues with the installation of an agent:

Note: Whereas this guide's purpose is to provide helpful tips for investigating network issues with your agent installation, please be aware that BlazeMeter Support does not provide network troubleshooting assistance.

Crane Does Not Download After Entering docker run Command

When you use the docker run command generated by the agent installation process, the initial crane image download will occur from the public Docker registry site. If this image fails to be downloaded, you may see the following error (or a variation of the same):

docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request cancelled while waiting for connection (Client.Timeout exceeded while awaiting headers).

This indicates that you have not properly configured your machine as described in the requirements page to allow connections from this site. Allowlisting this URL will allow the crane image to be downloaded as expected (either through the proxy you are using or the machine itself).

Crane Downloads but the Other Images Do Not

If you get the initial crane agent image downloaded but are not seeing the rest of the images being downloaded (including the latest crane image), then there is most likely some type of connection issue within the Docker crane container. The following is a list of known error messages and what causes them:

  • __retry_internal WARNING /usr/lib/python2.7/site-packages/retry/api.py HTTPSConnectionPool(host='a.blazemeter.com', port=443): Max retries exceeded with url: /api/v4/private-locations/5c4b99abc7dc97dd0ec487c1/ships/5db73a9cb3b5f7a16a2b5e63/status (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)), retrying in 1 seconds...
    This warning is caused by a bad or missing certificate. If you are implementing your agent to use the CA bundle, this error points to an issue with the certificate when it is trying to communicate to BlazeMeter with the certificate that is provided. To fix this, you will need to fix the certificate you are using, as it is either corrupted or does not have the right data for connecting to BlazeMeter.
  • __retry_internal WARNING /usr/lib/python2.7/site-packages/retry/api.py HTTPSConnectionPool(host='a.blazemeter.com', port=443): Max retries exceeded with url: /api/v4/private-locations/5c4b99abc7dc97dd0ec487c1/ships/5db73a9cb3b5f7a16a2b5e63/status (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff1e42658d0>: Failed to establish a new connection: [Errno 111] Connection refused',))), retrying in 1 seconds...
    This warning is caused by a bad connection to the proxy you are using. If you are implementing your agent to use a proxy, you need to ensure that you are using the correct URL and port for the proxy to ensure that you get valid connections to BlazeMeter. To fix this, see Optional Installation Step - Configuring Private Location Agents To Use a Corporate Proxy and ensure the proxy details you have are correct and ensure that you are using the correct variables.
  • __retry_internal WARNING /usr/lib/python2.7/site-packages/retry/api.py HTTPSConnectionPool(host='a.blazemeter.com',
    port=443): Max retries exceeded with url: /api/v4/private-locations/5a4ce723871f3bf4637b23c7/ships/5cecebb54db7de1bd530370c/status
    (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection
    object at 0x7f6f3b568f50>: Failed to establish a new connection: [Errno
      113] Host is unreachable',)), retrying in 1 seconds...
    This warning is caused by either the Docker container not getting network or the machine is unable to reach BlazeMeter (as listed in the requirements document). If you have verified that the issue is not tied to the machine (you can reach the required network connections), then you will want to dig into the settings on your machine with regards to Docker to ensure you are allowing Docker to access the network on your machine.

Agent Installation Not Running a Test

If you were able to get the agent fully installed with all images but you are running into issues with running tests, then you may have an issue with network connectivity to the container attempting to run your test. This will be signified by the following:

The test status "Servers received, downloading engine image" does not change

If you are running a test on your Private Location and you are seeing the test start status never change from the above, then you may be having issues with the connectivity in the container that is trying to spin up. To check this, use the method described in Checking For Connectivity from the Container to the BlazeMeter API Server - Docker Installation to run a connection check against BlazeMeter. If you get back a connection refused response, or some other response other than the one listed in the article, you will want to ensure that Docker is allowed to pass the network through the Docker bridge network interface (typically labeled docker0 in ifconfig). For the case of Centos/Red Hat Linux, you should do the following:

Edit the /etc/sysctl.conf with net.ipv4.conf.all.forwarding=1 and run sysctl -p /etc/sysctl.conf
OR
Use sysctl net.ipv4.conf.all.forwarding=1

Also, if you are using a proxy, ensure that you followed our documentation for implementing a proxy and not the official Docker documentation, as they will conflict with each other.