Parade experienced a major incident on January 10, 2023 affecting Broker Portal APIs and Aljex Integration and 1 more component, lasting 1h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 10, 2023, 05:02 PM UTC
We are currently investigating a network latency issue. This is caused by downtime at one of our cloud providers, specifically Google Cloud Platform, which is affecting their global customers. Here is a summary of what is being investigated on their end: https://status.cloud.google.com/incidents/FRpfyvgG3MUeSTTuX1Dx https://status.cloudamqp.com/ During this time, the following services are affected: - All Parade applications will see slowdowns in terms of network requests - Parade data processing will be delayed (this includes load and carrier data processing from the TMS) According to their site, this problem is intermittent, and seems to be causing intermittent timeouts and network packet loss. Pending their investigation, we are planning workarounds on our end to bring our cluster back to full health.
- resolved Jan 10, 2023, 06:44 PM UTC
This incident has been resolved as of 9:25 AM PST. Our cloud platform provider (GCP) has notified us that the network issues should no longer be happening. We have verified that all products have recovered. There is minimal data backup, and we will be closely monitoring data pipelines in the next few hours. A full incident post-mortem will be published soon.