Osano experienced a major incident on November 25, 2020 affecting Customer Authentication, lasting 6h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Nov 25, 2020, 05:19 PM UTC
Osano is currently experiencing an outage affecting the ability to log into the Osano dashboard. This outage does not impact visitor facing services such as consent management and subject rights management. The root cause is Osano's reliance on Amazon Web Service's Cognito authentication solution for processing customer credentials in the web app. The engineering team is monitoring the situation and will update this status as soon as it is resolved.
- resolved Nov 25, 2020, 11:57 PM UTC
Customer authentication issues have been resolved. The root cause of this outage was due to Osano's reliance on AWS Cognito identity stores which began experiencing increased API failure rates due to an issue with Kinesis Data Streams. AWS have implemented a mitigation to this issue.
- postmortem Nov 25, 2020, 11:57 PM UTC
Osano relies heavily on Amazon Web Services for all infrastructure. Cognito which is fault tolerant powers the authentication into the Osano dashboard experienced a major outage for approximately 6 hours. Osano engineering confirmed that the errors were not due to Osano configuration issues but rather were caused by Cognito having an outage. The Cognito outage was the result of failures on a heavily used streaming component on Amazon called Kinesis. Kinesis had a global outage across many data centers that impacted AWS Cognito along with numerous other AWS services. Outages of Kinesis on this scale are extremely rare and Cognito has had nearly 2 years without a single incident, so while this outage was inconvenient to customers, it is unlikely to recur and AWS has reassured us that mitigations are now in place to prevent this issue in the future.