Pitcher incident

Elevated API Errors

Notice Resolved View vendor source →

Pitcher experienced a notice incident on May 18, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
May 18, 2022, 05:42 PM UTC
Resolved
May 18, 2022, 02:00 PM UTC
Duration
Detected by Pingoru
May 18, 2022, 05:42 PM UTC

Update timeline

  1. resolved May 18, 2022, 05:42 PM UTC

    There was a cloud driven AWS availability issue that caused the Group Queries to be empty. We restored the connectivity and the Group Keywords were restored.

  2. postmortem May 18, 2022, 06:31 PM UTC

    ### Incident Post Mortem: We have made a number of improvements to increase availability in the event of future similar failures. First, have rolled out safeguard multi-az failovers to prevent similar outages. Second, though this incident impacted API based services, we’re rolling out improved tooling to ensure we can quickly identify similar problems. Finally, we’re rolling out safeguards that will allow us to contain the impact of future API failures using cached data.