Pypestream incident

Reports intermittent connectivity issue with EU DC

Critical Resolved View vendor source →

Pypestream experienced a critical incident on August 26, 2019, lasting 5h 11m. The incident has been resolved; the full update timeline is below.

Started
Aug 26, 2019, 05:19 PM UTC
Resolved
Aug 26, 2019, 10:31 PM UTC
Duration
5h 11m
Detected by Pingoru
Aug 26, 2019, 05:19 PM UTC

Update timeline

  1. investigating Aug 26, 2019, 05:19 PM UTC

    We are currently investigating the issue.

  2. identified Aug 26, 2019, 06:38 PM UTC

    AWS Infrastructure in our VPC collapsed and took down 3 of our RIAK DB VMs we are now working to recover from backups and resyncing nodes.

  3. identified Aug 26, 2019, 07:53 PM UTC

    We are continuing to work on rebuilding 3 nodes.

  4. identified Aug 26, 2019, 09:36 PM UTC

    We are still working on recovering data machines were rebuilt and working on synchronizations.

  5. monitoring Aug 26, 2019, 10:23 PM UTC

    A fix has been implemented and we are monitoring the results.

  6. monitoring Aug 26, 2019, 10:24 PM UTC

    Pypes are all restored and Solutions are all back and loading we are now monitoring as system rebalances

  7. resolved Aug 26, 2019, 10:31 PM UTC

    Solutions are back up please work with your Customer Success or Customer Experience Contact if you are seeing issues.