This incident has been resolved. No further issues were seen since adjusting the backend configuration on Friday June 20th. (22:22 UTC)
The root cause has been identified as node CPU saturation, causing high latency on ingesters.
Posted Jun 24, 2025 - 16:35 UTC
Monitoring
We are aware of a few intermittent re-occurrences lasting 10-15 minutes. We've made some adjustments to the backend configuration and continue to monitor the situation.
Posted Jun 20, 2025 - 22:22 UTC
Update
We are continuing to investigate this issue.
Posted Jun 20, 2025 - 15:54 UTC
Investigating
Starting at 13:23 UTC and resolved at 13:36 UTC, some hosted Prometheus datasources experience high query latency, which could trigger false alarms or cause timeouts in the UI when querying. The root cause is being investigated.
Posted Jun 20, 2025 - 15:48 UTC
This incident affected: Grafana Cloud: Loki (AWS US East - prod-us-east-0) and Grafana Cloud: Prometheus (AWS US East - prod-us-east-0: Querying).