Hive incident

Database performance issues causing partial service outages across the platform

Major Resolved View vendor source →

Hive experienced a major incident on October 13, 2021 affecting Document Proofing and Desktop Application and 1 more component, lasting 5h 52m. The incident has been resolved; the full update timeline is below.

Started
Oct 13, 2021, 02:28 PM UTC
Resolved
Oct 13, 2021, 08:20 PM UTC
Duration
5h 52m
Detected by Pingoru
Oct 13, 2021, 02:28 PM UTC

Affected components

Document ProofingDesktop ApplicationMessaging and NotificationsMobile Application

Update timeline

  1. investigating Oct 13, 2021, 02:28 PM UTC

    We are currently investigating this issue.

  2. investigating Oct 13, 2021, 02:28 PM UTC

    We are continuing to investigate this issue.

  3. investigating Oct 13, 2021, 02:29 PM UTC

    We are continuing to investigate this issue.

  4. investigating Oct 13, 2021, 02:31 PM UTC

    We have identified the likely source and are currently working to remediate and restore service.

  5. investigating Oct 13, 2021, 04:26 PM UTC

    Our DB PaaS provider is currently experiencing an outage that has been preventing our normal scale-up process (https://status.cloud.mongodb.com/incidents/bkzhxk9db0nr) and we are continuing to work on other mitigating options while we wait for them to fully restore service.

  6. investigating Oct 13, 2021, 05:03 PM UTC

    We continuing to work on mitigating efforts while we work with our DB provider to restore our scaling functionality.

  7. investigating Oct 13, 2021, 05:37 PM UTC

    We have completed a manual scaling increase and are currently redeploying our services to re-sync and restore service.

  8. monitoring Oct 13, 2021, 07:13 PM UTC

    A fix has been implemented and we are monitoring the results.

  9. resolved Oct 13, 2021, 08:20 PM UTC

    This incident has been resolved.