Athennian incident

Intermittent performance issues

Resolved after 1 hour 31 minutes, on September 26, 2022.

postmortem Sep 28, 2022, 05:10 PM UTC

On Monday, September 26th, 2022, we experienced intermittent outages for some Canadian customers between 12:51 pm to 1:31 pm MST. A recent change involving the transition between Web Application Firewall \(WAF\) providers adversely impacted the production environment causing other application health safeguards to kick in to action creating the intermittent access issues for customers hosted in our Canadian data centers. Systems in place immediately notified our team of the issue. We then worked to restore the system back to full operation for all customers. Further, steps have been taken to ensure our change management procedures consider potential spill over issues while working with third-party security infrastructure providers.

Started
Sep 26, 2022, 07:00 PM UTC
Resolved
Sep 26, 2022, 08:31 PM UTC
Duration
1h 31m
Detected by Pingoru
Sep 26, 2022, 07:00 PM UTC

Update timeline

  1. resolved Sep 28, 2022, 05:10 PM UTC

    Users are experiencing "503" errors and are unable to reach the application.

  2. postmortem Sep 28, 2022, 05:10 PM UTC

    On Monday, September 26th, 2022, we experienced intermittent outages for some Canadian customers between 12:51 pm to 1:31 pm MST. A recent change involving the transition between Web Application Firewall \(WAF\) providers adversely impacted the production environment causing other application health safeguards to kick in to action creating the intermittent access issues for customers hosted in our Canadian data centers. Systems in place immediately notified our team of the issue. We then worked to restore the system back to full operation for all customers. Further, steps have been taken to ensure our change management procedures consider potential spill over issues while working with third-party security infrastructure providers.

Your product runs on more than Athennian

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →