Akana incident

Global environment; Developer Portal and OAuth impaired

Resolved after 23 hours 59 minutes, on October 31, 2021.

resolved Oct 31, 2021, 03:09 AM UTC

Beginning at 22:30UTC; the Gobal API Portal (Developler Portal) cluster was at reduced functionality. Full functionality was restored by 02:56UTC. Initial determination of the cause was due to multi-regional AWS EBS service issues impacting the io2 (provisioned IOPS) volume types, used by the Akana Mongo DB clusters. The EBS issues caused nvme kernel driver "panics", rendering the systems unresponsive. This resulted in increase API Portal java heap utilization and eventual exhaustion, and had a cascading affect across all the active API Portal JVMs with JVM Out of Memory conditions that induced rolling JVM restarts, and periodic loss of connectivity to the Developer Portals, and OAuth services.

Started: Oct 31, 2021, 02:56 AM UTC
Resolved: Nov 01, 2021, 02:56 AM UTC
Duration: 23h 59m
Detected by Pingoru: Oct 31, 2021, 02:56 AM UTC

Update timeline

resolved Oct 31, 2021, 03:09 AM UTC

Beginning at 22:30UTC; the Gobal API Portal (Developler Portal) cluster was at reduced functionality. Full functionality was restored by 02:56UTC. Initial determination of the cause was due to multi-regional AWS EBS service issues impacting the io2 (provisioned IOPS) volume types, used by the Akana Mongo DB clusters. The EBS issues caused nvme kernel driver "panics", rendering the systems unresponsive. This resulted in increase API Portal java heap utilization and eventual exhaustion, and had a cascading affect across all the active API Portal JVMs with JVM Out of Memory conditions that induced rolling JVM restarts, and periodic loss of connectivity to the Developer Portals, and OAuth services.

Your product runs on more than Akana

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.