Stellar incident
History lag and degraded performance on certain Horizon endpoints
Stellar experienced a minor incident on March 11, 2024 affecting SDF Public Network Horizon, lasting 7d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 11, 2024, 11:34 PM UTC
We've identified a high CPU utilization on SDF Horizon nodes due to increased request load that saturated DB connections.
- investigating Mar 11, 2024, 11:37 PM UTC
We reduced global rate limits by half for Horizon.
- monitoring Mar 11, 2024, 11:45 PM UTC
Reports of degraded performance in Discord.
- monitoring Mar 12, 2024, 12:20 AM UTC
We flipped the standby and active endpoint. This temporarily fixed the problem.
- monitoring Mar 12, 2024, 12:22 AM UTC
Performance degradation observed again.
- monitoring Mar 12, 2024, 12:24 AM UTC
Additional reduction of rate limits.
- monitoring Mar 12, 2024, 12:29 AM UTC
A fix has been deployed. We are monitoring to ensure it is stable.
- monitoring Mar 12, 2024, 06:53 PM UTC
Fix appears to have resolved performance issues, but reduced rate limits are still in effect as we continue to monitor.
- resolved Mar 18, 2024, 11:01 PM UTC
Service has remained stable over the past week and thus the incident is resolved. Note that rate limits may dynamically adjust to preserve service health in times of high network activity or volatility.