API Monitoring (Runscope) slowness
Incident Report for BlazeMeter
Postmortem

This was related to our AWS->GCP Migration. See https://status.blazemeter.com/incidents/yn3z09p2tvpq for details

Posted Jun 17, 2020 - 06:56 PDT

Resolved
Our engineering team has completed the fixes to resolve this incident. They are continuing to make additional optimizations but API Monitoring should be fully functional.

After we finish reviewing the incident, we will post a postmortem report here. We apologize sincerely for the disruption.
Posted May 29, 2020 - 17:52 PDT
Monitoring
Our engineering team has made a number of changes to the API Monitoring (Runscope) system that we are now monitoring. The platform should be stable and performant now. If you do encounter any issues, please notify support@blazemeter.com. We will provide an additional update by 18:00 PDT.

Sometime next week, once everything is resolved, we will provide a more in depth analysis of what went wrong and how we addressed it.

We appreciate your patience through this process.
Posted May 29, 2020 - 12:56 PDT
Update
API Monitoring (Runscope) should continue to be functioning with some degraded performance. Our engineering team continues to work on optimizing API Monitoring. We are working to provide a more substantive update about the remaining items. We hope to have that by our next post at 13:00 PDT.
Posted May 29, 2020 - 11:59 PDT
Update
We continue to monitor the results of the changes we made in US California and will provide another update by 12:00 PDT.
Posted May 29, 2020 - 10:58 PDT
Update
We've made. some of the changes to our US California location and are monitoring the results while making some additional changes. Based on those results we will determine the next steps. We expect to make our next update by 11:00 PDT.
Posted May 29, 2020 - 09:59 PDT
Update
Our team is working on making changes to our US California location that should improve performance and reduce queue backlog. If you are seeing issues with tests running from US California, please try running tests from a different location.

We will provide the next update by 10:00 PDT.
Posted May 29, 2020 - 09:00 PDT
Update
Sorry for the delay in posting this update! Our team continues to work on the two items referenced last update and we expect those to be resolved today. We hope to have a more substantive update in the next hour or two. Thank you for your continued patience.

We will provide the next update by 9:00 PDT.
Posted May 29, 2020 - 08:14 PDT
Update
Time for the 7am update. Our team continues to work hard to optimize API Monitoring (Runscope) performance. There are four items remaining for resolution and work is in process on two of them. No concrete information yet on ETA for final resolution.

Next update at 8:00 PDT.
Posted May 29, 2020 - 07:01 PDT
Update
Good morning -- our team is continuing to work on the earlier identified issues and we will continue to provide hourly updates. Thank you for your patience.

The next update will be posted by 7:00 PDT
Posted May 29, 2020 - 06:05 PDT
Update
Overnight, our team identified and repaired a number of issues related to database and networking performance. In general, the system should be performing better but there are a few remaining issues we are working on fixing.

We will provide the next update by 6:00 PDT.
Posted May 29, 2020 - 04:43 PDT
Update
Our team is continuing to work over night to resolve the performance issues with the API Monitoring (Runscope) Functionality. If we have any substantive updates we will provide them. Otherwise, we will provide the next update by 6:00 PDT tomorrow morning (May 29).
Posted May 28, 2020 - 18:00 PDT
Update
Our team continues to work to make API Monitoring provide the performance we all expect. To do this, we have moved our Virginia location (which still ran on AWS) to Google Cloud Platform. We have also upgraded one of our databases with a higher performance CPU. We will provide the next update by 18:00 PDT.
Posted May 28, 2020 - 17:00 PDT
Update
There is no additional detail at this time about the progress our team is making, but they continue to work hard to resolve the issues as fast as we can. We will provide another update by 17:00 PDT.
Posted May 28, 2020 - 15:59 PDT
Update
Our team is continuing to work on a resolution to the ongoing issues. Primary behaviors you may see include API Monitoring tests running slowly from US Virginia (US1) location and delays in start of non-scheduled tests. We will provide our next update by 16:00 PDT.
Posted May 28, 2020 - 14:57 PDT
Identified
Our team has identified several items which are leading to the performance issues we are seeing and are working on fixes. We do not have an expected ETA for completion but will provide the next update by 15:00 PDT. Thank you for your patience.
Posted May 28, 2020 - 14:08 PDT
Update
Some say "no news is good news" but you probably don't agree with that right now. Our team is continuing to investigate what is causing the issues. We apologize for the inconvenience.

We will post an update by 14:30 PDT
Posted May 28, 2020 - 13:30 PDT
Update
Our engineering team is continuing to investigate. We will provide the next update by 13:30 PDT.
Posted May 28, 2020 - 12:26 PDT
Update
Our engineering team is continuing to investigate. We will provide the next update by 12:30 PDT.
Posted May 28, 2020 - 11:29 PDT
Update
We are continuing to investigate. There are issues related to the Virginia location as well as queuing API Monitoring (Runscope) tests. We will provide another update by 11:30 PDT.
Posted May 28, 2020 - 10:30 PDT
Investigating
We are seeing some indications of slowness in API Monitoring (Runscope). This may particularly impact tests run from US1 (Virginia) location. We are investigating.

Note — if you are seeing any issues with tests using a local agent, please restart the agent
Posted May 28, 2020 - 09:34 PDT
This incident affected: API Monitoring (Runscope).