Private Location Agents in Error state

Incident Report for BlazeMeter

Resolved

As of 00:30 UTC, the issue experienced by some customers has been addressed and functionality has been restored. This incident is now resolved.
Posted Mar 05, 2025 - 20:38 PST

Update

The root cause of the issue has been identified by the R&D team. Unfortunately an automated fix cannot be rolled out to the affected Agents so they will need to be manually restarted if they are in Error state due to this issue. The steps to restart the Agents are as follows:

Docker Steps:

sudo docker ps -a
sudo docker start {{crane containerId}}

Kubernetes Step:

kubectl rollout restart deployment {{deployment_name}} -n {{namespace}}
Posted Mar 05, 2025 - 07:53 PST

Monitoring

The root cause of the issue has been identified by the R&D team. Unfortunately an automated fix cannot be rolled out to the affected Agents so they will need to be manually restarted if they are in Error state due to this issue. The steps to restart the Agents are as follows:

Docker Steps:

sudo docker ps -a
sudo docker start {{crane containerId}}

Kubernetes Step:

kubectl rollout restart deployment {{deployment_name}} -n {{namespace}}
Posted Mar 05, 2025 - 07:33 PST

Update

We are still continuing to identify the root cause of the issue. However most Agents can be returned to a functional state by restarting them. The steps for restarting an Agent are:

Docker Steps:

sudo docker ps -a

sudo docker start {{crane containerId}}

Kubernetes Step:

kubectl rollout restart deployment {{deployment_name}} -n {{namespace}}
Posted Mar 05, 2025 - 06:58 PST

Update

We are continuing to investigate this issue.
Posted Mar 05, 2025 - 06:27 PST

Investigating

We are aware of an issue with Private Locations at this time that has caused them to go into Error state. We are actively investigating the issue now and will provide updates as soon as possible. Restarting or Regenerating the Agents has resolved the issue for some customers so we recommend attempting that for Agents currently in and Error state.
Posted Mar 05, 2025 - 06:24 PST
This incident affected: BlazeMeter Platform.