Parade incident

Delay in API and CSV updates from the TMS

Notice Resolved View vendor source →

Parade experienced a notice incident on May 19, 2023 affecting Internal Integration APIs, lasting 2h 37m. The incident has been resolved; the full update timeline is below.

Started
May 19, 2023, 05:28 PM UTC
Resolved
May 19, 2023, 08:06 PM UTC
Duration
2h 37m
Detected by Pingoru
May 19, 2023, 05:28 PM UTC

Affected components

Internal Integration APIs

Update timeline

  1. monitoring May 19, 2023, 05:28 PM UTC

    We experienced a slowdown in processing both API and CSV updates from the TMS, starting around 7:08 AM PST. We have already implemented a fix to API integrations, which was put into place in production around 9:00AM PST. We have been monitoring for over an hour, and we are processing updates in real-time. For CSV integrations, we have already implemented a series of optimizations that have brought the processing delay to under 20 minutes, but have not fully resolved the issue. We are implementing an additional fix at 10:30AM PST that should bring updates back to a non-delayed state. No data was lost during this time period, and API integrations are already in a consistent state. We will continue to monitor CSV integrations and provide updates as needed. As part of the investigation, we have also identified further improvements to our processing pipeline for both API and CSV integrations. These improvements will be implemented in the near future.

  2. resolved May 19, 2023, 08:06 PM UTC

    After monitoring this for the last 2 hours, no further delays have occurred. This incident has been resolved