Stellar incident

Certain validators dropped off the network

Major Resolved View vendor source →

Stellar experienced a major incident on April 6, 2021 affecting Stellar Public Network and SDF Public Network Validators and 1 more component, lasting 10h 40m. The incident has been resolved; the full update timeline is below.

Started
Apr 06, 2021, 08:19 AM UTC
Resolved
Apr 06, 2021, 07:00 PM UTC
Duration
10h 40m
Detected by Pingoru
Apr 06, 2021, 08:19 AM UTC

Affected components

Stellar Public NetworkSDF Public Network ValidatorsSDF Public Network Horizon

Update timeline

  1. investigating Apr 06, 2021, 09:28 AM UTC

    SDF engineers are investigating some validators that dropped off the network

  2. identified Apr 06, 2021, 10:02 AM UTC

    We have identified potential root cause of the problem and are working to restore services

  3. identified Apr 06, 2021, 10:31 AM UTC

    SDF engineers are still working on testing the fix. We are planning to restore the SDF horizon endpoint in first instance

  4. identified Apr 06, 2021, 11:01 AM UTC

    SDF engineers are still working on testing the potential fix to restore SDF horizon service

  5. identified Apr 06, 2021, 11:42 AM UTC

    We are continuing to work on a fix for this issue.

  6. identified Apr 06, 2021, 11:45 AM UTC

    SDF engineers continue investigating and testing a potential fix but we're unable to provide an ETA for a fix at this time.

  7. identified Apr 06, 2021, 12:22 PM UTC

    We are continuing to work on a fix for this issue.

  8. identified Apr 06, 2021, 12:42 PM UTC

    We are continuing to work on a fix for this issue.

  9. identified Apr 06, 2021, 12:42 PM UTC

    SDF engineers are continuing to investigate this incident

  10. identified Apr 06, 2021, 01:26 PM UTC

    SDF engineers are continuing to investigate this incident. At this moment we are unable to provide ETA for a fix

  11. identified Apr 06, 2021, 02:32 PM UTC

    We are continuing to work on a fix for this issue

  12. identified Apr 06, 2021, 03:28 PM UTC

    SDF engineers are continuing to investigate this incident. At this moment we are unable to provide ETA for a fix

  13. identified Apr 06, 2021, 05:03 PM UTC

    We continue to investigate this issue. In parallel we are bringing a new SDF Horizon cluster up. We will update as soon as that’s complete.

  14. monitoring Apr 06, 2021, 06:28 PM UTC

    We restored SDF horizon service and brought the SDF validators online. We continue to monitor the situation

  15. resolved Apr 06, 2021, 07:00 PM UTC

    The SDF horizon cluster and validators are back up. We will be sharing more information on the root cause in the near future