GumGum incident
Increased Classification Latency in page and image endpoints
GumGum experienced a critical incident on October 24, 2023 affecting Verity Api, lasting 10h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 24, 2023, 08:33 PM UTC
We are facing an issue with one of our vendors which has resulted in increased latency for image classification. This affects both image and page endpoint classification latencies.
- investigating Oct 24, 2023, 10:41 PM UTC
We are still investing the root cause of our vendor. At the moment, we are seeing the following delay in classifications: Pages: 45 to 50 minute delay Videos: 3.5 to 4 hour delay Images: 45 to 50 minute delay This mostly affects content that has image or visual components.
- investigating Oct 25, 2023, 12:10 AM UTC
We are currently online with our cloud vendor to help assist with identifying the issue affecting our image services.
- investigating Oct 25, 2023, 02:11 AM UTC
The issue is still not resolved. At this moment we are experiencing the following effects to clients: Pages: Outage affecting content with image data Videos: Outage Images: Outage Text: No impact
- monitoring Oct 25, 2023, 03:39 AM UTC
We have implemented a fix to our image inference services and we are seeing our backlog of messages being processed. This has been done for Pages and Images with Videos coming shortly. For Pages and Images we are expecting the following delay in results: Pages: up-to a 4hr delay in data Images: up-to a 4hr delay in data These latency are expected to decrease as we scale out our backend.
- resolved Oct 25, 2023, 06:43 AM UTC
We have applied a fix to our image inference services for videos as well. We will continue monitoring to ensure stability. We expect the backlog of messages for Pages and Images to be processed between 2 to 3 hours