Our engineering team has completed the fixes to resolve this incident. They are continuing to make additional optimizations but API Monitoring should be fully functional.
After we finish reviewing the incident, we will post a postmortem report here. We apologize sincerely for the disruption.
May 29, 17:52 PDT
Our engineering team has made a number of changes to the API Monitoring (Runscope) system that we are now monitoring. The platform should be stable and performant now. If you do encounter any issues, please notify email@example.com
. We will provide an additional update by 18:00 PDT.
Sometime next week, once everything is resolved, we will provide a more in depth analysis of what went wrong and how we addressed it.
We appreciate your patience through this process.
May 29, 12:56 PDT
API Monitoring (Runscope) should continue to be functioning with some degraded performance. Our engineering team continues to work on optimizing API Monitoring. We are working to provide a more substantive update about the remaining items. We hope to have that by our next post at 13:00 PDT.
May 29, 11:59 PDT
We continue to monitor the results of the changes we made in US California and will provide another update by 12:00 PDT.
May 29, 10:58 PDT
We've made. some of the changes to our US California location and are monitoring the results while making some additional changes. Based on those results we will determine the next steps. We expect to make our next update by 11:00 PDT.
May 29, 09:59 PDT
Our team is working on making changes to our US California location that should improve performance and reduce queue backlog. If you are seeing issues with tests running from US California, please try running tests from a different location.
We will provide the next update by 10:00 PDT.
May 29, 09:00 PDT
Sorry for the delay in posting this update! Our team continues to work on the two items referenced last update and we expect those to be resolved today. We hope to have a more substantive update in the next hour or two. Thank you for your continued patience.
We will provide the next update by 9:00 PDT.
May 29, 08:14 PDT
Time for the 7am update. Our team continues to work hard to optimize API Monitoring (Runscope) performance. There are four items remaining for resolution and work is in process on two of them. No concrete information yet on ETA for final resolution.
Next update at 8:00 PDT.
May 29, 07:01 PDT
Good morning -- our team is continuing to work on the earlier identified issues and we will continue to provide hourly updates. Thank you for your patience.
The next update will be posted by 7:00 PDT
May 29, 06:05 PDT
Overnight, our team identified and repaired a number of issues related to database and networking performance. In general, the system should be performing better but there are a few remaining issues we are working on fixing.
We will provide the next update by 6:00 PDT.
May 29, 04:43 PDT
Our team is continuing to work over night to resolve the performance issues with the API Monitoring (Runscope) Functionality. If we have any substantive updates we will provide them. Otherwise, we will provide the next update by 6:00 PDT tomorrow morning (May 29).
May 28, 18:00 PDT
Our team continues to work to make API Monitoring provide the performance we all expect. To do this, we have moved our Virginia location (which still ran on AWS) to Google Cloud Platform. We have also upgraded one of our databases with a higher performance CPU. We will provide the next update by 18:00 PDT.
May 28, 17:00 PDT
There is no additional detail at this time about the progress our team is making, but they continue to work hard to resolve the issues as fast as we can. We will provide another update by 17:00 PDT.
May 28, 15:59 PDT
Our team is continuing to work on a resolution to the ongoing issues. Primary behaviors you may see include API Monitoring tests running slowly from US Virginia (US1) location and delays in start of non-scheduled tests. We will provide our next update by 16:00 PDT.
May 28, 14:57 PDT
Our team has identified several items which are leading to the performance issues we are seeing and are working on fixes. We do not have an expected ETA for completion but will provide the next update by 15:00 PDT. Thank you for your patience.
May 28, 14:08 PDT
Some say "no news is good news" but you probably don't agree with that right now. Our team is continuing to investigate what is causing the issues. We apologize for the inconvenience.
We will post an update by 14:30 PDT
May 28, 13:30 PDT
Our engineering team is continuing to investigate. We will provide the next update by 13:30 PDT.
May 28, 12:26 PDT
Our engineering team is continuing to investigate. We will provide the next update by 12:30 PDT.
May 28, 11:29 PDT
We are continuing to investigate. There are issues related to the Virginia location as well as queuing API Monitoring (Runscope) tests. We will provide another update by 11:30 PDT.
May 28, 10:30 PDT
We are seeing some indications of slowness in API Monitoring (Runscope). This may particularly impact tests run from US1 (Virginia) location. We are investigating.
Note — if you are seeing any issues with tests using a local agent, please restart the agent
May 28, 09:34 PDT