Root cause of the problem
As part of BlazeMeter’s business continuity plans, BlazeMeter engineering is preparing redundant stage and DR environments in other locations/providers to ensure business and data continuity. As part of this initiative, they were performing simulation testing in their redundant staging environment with redundant test schedules dataset from production backups. But unfortunately, the simulation test missed turning off the integrations associated with those test schedules, hence, the staging schedules simulation triggered failure notifications to customer’s configured alert/notification mechanism associated with it.
What we are doing to avoid recurrences in future: