Akana incident

Global environment; Developer Portal and OAuth impaired

Notice Resolved View vendor source →

Akana experienced a notice incident on October 31, 2021, lasting —. The incident has been resolved; the full update timeline is below.

Started
Oct 31, 2021, 03:09 AM UTC
Resolved
Oct 31, 2021, 02:56 AM UTC
Duration
—
Detected by Pingoru
Oct 31, 2021, 03:09 AM UTC

Update timeline

  1. resolved Oct 31, 2021, 03:09 AM UTC

    Beginning at 22:30UTC; the Gobal API Portal (Developler Portal) cluster was at reduced functionality. Full functionality was restored by 02:56UTC. Initial determination of the cause was due to multi-regional AWS EBS service issues impacting the io2 (provisioned IOPS) volume types, used by the Akana Mongo DB clusters. The EBS issues caused nvme kernel driver "panics", rendering the systems unresponsive. This resulted in increase API Portal java heap utilization and eventual exhaustion, and had a cascading affect across all the active API Portal JVMs with JVM Out of Memory conditions that induced rolling JVM restarts, and periodic loss of connectivity to the Developer Portals, and OAuth services.