As teams start adopting API Monitoring more broadly in an organization, there are certain limits that are good to keep in mind when creating and organizing your buckets and tests.
Currently, our buckets have certain limits when running a high number of tests, which is based on whether your team is using cloud locations and/or on-premise Radar Agents:
- 200 tests can be simultaneously running per cloud location (example: US Virginia) per bucket
- 200 tests can be simultaneously running amongst all on-premise agents (example: multiple tests running from 10 different agents) in a bucket
This is true for tests which are running or being executed concurrently. If you go over these limits, any additional test runs will be added to a queue, and will be executed as soon as the number of concurrent tests is less than 200. This does increase the chances of tests failing for taking more than 10 minutes to execute, which is the system limit for the execution time of a test.
Some example scenarios of how this might happen:
- Scenario 1: let's say you have 1 bucket with 250 tests in it, only one cloud location enabled for them of US Virginia, and every test is scheduled to run every minute. What is going to happen is, 200 of those tests will start running. The remaining 50 tests will be added to a queue, and as soon as the initial 200 tests finish running the remaining tests will start being executed. If your tests are long-running tests for any reason, this can lead to the queues in the system getting bigger, and that could lead to tests expiring because of timeout issues.
- Scenario 2: let's say you have 1 bucket with 150 tests in it, each test has two cloud locations enabled of US Virginia and US Illinois, and every test is scheduled to run every minute. Since each test has two cloud locations, every minute you'll have 300 tests running concurrently. But since each cloud location will only have a maximum of 150 tests running concurrently, as long as your tests finish running in under a minute, you shouldn't see any issues.
- Scenario 3: let's say you have 1 bucket with 100 tests in it, and each test has three Radar Agent locations, and every test is scheduled to run every minute. Since there are 3 locations, every minute 200 tests will start running among the 3 different agents, while 100 tests will be added to the queue and wait for the initial tests to be completed before they can start running. Like we mentioned above, on-premise agents work differently than our public locations. Even though in this scenario there are 3 separate agents representing 3 locations in your environment, they can still only run a maximum of 200 tests at the same time.
To avoid running into any issues, we recommend: