VergeSense incident

Delayed Space Type and Space Group metrics and degraded performance

Major Resolved View vendor source →

VergeSense experienced a major incident on January 11, 2024 affecting AWS and API, lasting 5d 5h. The incident has been resolved; the full update timeline is below.

Started
Jan 11, 2024, 04:11 PM UTC
Resolved
Jan 16, 2024, 09:59 PM UTC
Duration
5d 5h
Detected by Pingoru
Jan 11, 2024, 04:11 PM UTC

Affected components

AWSAPI

Update timeline

  1. monitoring Jan 11, 2024, 04:11 PM UTC

    Our hourly Space Type and Space Group metrics are currently delayed and catching up has been causing additional performance issues with the main application. A fix is ongoing.

  2. monitoring Jan 11, 2024, 04:20 PM UTC

    We are continuing to monitor for any further issues.

  3. monitoring Jan 12, 2024, 03:48 PM UTC

    Hourly Space Type and Space Group metrics are mostly caught up and should be fully caught up later today, performance issues are resolved.

  4. monitoring Jan 12, 2024, 11:19 PM UTC

    Space Group and Space Type metrics are now caught back up and we are continuing to ensure they continue to keep up.

  5. monitoring Jan 16, 2024, 04:01 PM UTC

    We've hit a small slowdown of Space Group and Space Type metrics over the weekend are working on getting these back up to date

  6. monitoring Jan 16, 2024, 07:44 PM UTC

    We are caught up with Space Type and Group metrics through 1/15 and are continuing to work toward full remediation.

  7. resolved Jan 16, 2024, 09:59 PM UTC

    Space Group and Space Type metrics are back up to date