TeamDynamix incident

Elevated Email Monitor Processing Error Rate

Major Resolved View vendor source →

TeamDynamix experienced a major incident on March 6, 2024 affecting Email, lasting 6h 55m. The incident has been resolved; the full update timeline is below.

Started
Mar 06, 2024, 02:58 PM UTC
Resolved
Mar 06, 2024, 09:54 PM UTC
Duration
6h 55m
Detected by Pingoru
Mar 06, 2024, 02:58 PM UTC

Affected components

Email

Update timeline

  1. investigating Mar 06, 2024, 02:58 PM UTC

    We have identified increased error rates in the services that support the processing of email monitors. We are working to identify the issue and will continue to provide updates as progress is made.

  2. investigating Mar 06, 2024, 06:02 PM UTC

    Error rates for processing new emails have returned to normal levels. We are now seeing delays in outbound email and are monitoring resource levels.

  3. resolved Mar 06, 2024, 09:54 PM UTC

    Email Monitors and Outbound Email are processing as expected since 4:16pm ET. We have identified a bug in the logic used to persist emails as a contributing factor and have deployed a fix. We will continue analysis of the problem to ensure proper resolution.