Aqua experienced a major incident on November 25, 2020 affecting Cloud Account Scanning and Real-Time Events Service and 1 more component, lasting 8h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Nov 25, 2020, 03:02 PM UTC
Aqua Wave is currently monitoring elevated error rates from some AWS APIs. These errors are preventing Aqua Wave from running complete scans, so we have temporarily disabled scheduled scans.
- monitoring Nov 25, 2020, 03:22 PM UTC
The AWS outage is also impacting Cognito, which may affect the ability to sign in using SSO. If you require a break-glass user and do not currently have one, please contact our support.
- monitoring Nov 25, 2020, 03:40 PM UTC
We are continuing to monitor elevated AWS error rates affecting CloudWatch, Cognito, Kinesis, and other services (see the AWS status page for more information: https://status.aws.amazon.com/). This is currently impacting Aqua Wave in the following ways: - Sign in via SAML/SSO may be degraded. Standard sign in is not impacted. - Scheduled background scans may be delayed - Scans may produce "new risk" or "unknown" results for impacted services
- monitoring Nov 25, 2020, 03:49 PM UTC
We are continuing to monitor for any further issues.
- monitoring Nov 25, 2020, 05:08 PM UTC
We are continuing to monitor increased error rates from AWS in us-east-1. At this time, Aqua Wave's core APIs are not impacted, but background scanning continues to remain affected due to the use of AWS CloudWatch APIs.
- monitoring Nov 25, 2020, 06:47 PM UTC
We are continuing to monitor elevated AWS error rates across numerous AWS services. All Aqua Wave APIs are currently functioning as expected. CSPM background scans continue to run at a reduced interval. Technical details: CSPM background scans are normally triggered by CloudWatch Event rules, which are currently not operational due to an outage in AWS us-east-1. However, our failover workflow has been implemented which will continue to trigger these scans at reduced capacity until AWS CloudWatch is fully restored in us-east-1. During this time, you may find that scan reports are delayed. For example, if your scan is scheduled to run hourly, it may run 10-15 minutes after the hour at staggered intervals instead. CSPM Real-Time Events are typically triggered via CloudWatch Event rules deployed in the user environment. You may see delayed or missing events in us-east-1 during the outage window as these rules are unlikely to be triggered. The CSPM Real-Time Events service is still operational and available to receive the events that are sent. AWS CloudTrail is also experiencing a partial outage which may result in delayed or missing delivery of events from source AWS accounts to Aqua Wave.
- monitoring Nov 25, 2020, 08:06 PM UTC
Aqua Wave is continuing to monitor an ongoing incident in AWS. The impact is currently limited to CSPM background scan schedules and the triggering of real-time events from user accounts in us-east-1. We will continue to provide updates when we have new information to share, but at this point, will not publish any new updates unless we receive a status change from AWS or different issues are detected.
- resolved Nov 25, 2020, 11:32 PM UTC
AWS services appear to be recovering and at this time, all core Aqua Wave functionality has returned to normal operation.