Simplr incident

Platform stopped processing events after new release deployment

Major Resolved View vendor source →

Simplr experienced a major incident on September 21, 2021, lasting —. The incident has been resolved; the full update timeline is below.

Started
Sep 21, 2021, 07:15 PM UTC
Resolved
Sep 21, 2021, 07:15 PM UTC
Duration
—
Detected by Pingoru
Sep 21, 2021, 07:15 PM UTC

Update timeline

  1. resolved Sep 21, 2021, 07:24 PM UTC

    Release reverted back

  2. postmortem Sep 21, 2021, 07:25 PM UTC

    **Incident:** Platform unusable on 11/21/2021 between 11:00am - 12:15pm PDT **Timeline:** * Operations team reported platform sluggishness around 11:00 AM PDT * Problem resolved at 12:15 pm PDT after rolling back the release and killing long running DB queries **RCA:** A new slow query caused database to back up and platform stopped processing new events. Release was reverted back.