Logit.io incident

Elastic Search - Clusters Yellow and Red

Major Resolved View vendor source →

Logit.io experienced a major incident on August 10, 2020 affecting Search Hosts, lasting 1h 51m. The incident has been resolved; the full update timeline is below.

Started
Aug 10, 2020, 10:42 PM UTC
Resolved
Aug 11, 2020, 12:34 AM UTC
Duration
1h 51m
Detected by Pingoru
Aug 10, 2020, 10:42 PM UTC

Affected components

Search Hosts

Update timeline

  1. investigating Aug 10, 2020, 10:42 PM UTC

    We are investigating the root cause, we have a range of yellow and red clusters

  2. identified Aug 10, 2020, 10:58 PM UTC

    We are working with out hosting provider to restore Network connectivity to the effected hosts, we will provide an update in 30 minutes

  3. identified Aug 10, 2020, 11:30 PM UTC

    We are working with our hosting provider to fix the networking, they are replacing the hardware. Further update in 1 hour

  4. identified Aug 10, 2020, 11:45 PM UTC

    We have connectivity back to a number of servers, we are working on ensuring all stacks on the effected hosts are operating as expected

  5. monitoring Aug 11, 2020, 12:09 AM UTC

    All servers are operational, we are monitoring clusters health from Yellow to Green

  6. resolved Aug 11, 2020, 12:34 AM UTC

    All clusters have recovered and are green