GoSpotCheck incident

Delays in data pipeline and impacts to photo rendering

Minor Resolved View vendor source →

GoSpotCheck experienced a minor incident on December 7, 2021 affecting Web Application and Insights and 1 more component, lasting 9h 10m. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 04:00 PM UTC
Resolved
Dec 08, 2021, 01:10 AM UTC
Duration
9h 10m
Detected by Pingoru
Dec 07, 2021, 04:00 PM UTC

Affected components

Web ApplicationInsightsPhotoWorks

Update timeline

  1. investigating Dec 07, 2021, 04:00 PM UTC

    We are currently investigating an issue with our upstream infrastructure provider that is preventing execution of our data pipeline

  2. investigating Dec 07, 2021, 04:33 PM UTC

    We are continuing to investigate this issue. Our data pipelines have been paused since 8:30am MT - Insights does not currently reflect any new or updated data after this timestamp

  3. investigating Dec 07, 2021, 05:07 PM UTC

    We have identified that the upstream infrastructure issue is causing some photos and thumbnails to fail to load in the GoSpotCheck web application and PhotoWorks. We are currently waiting on updates from the upstream provider for an expected time of resolution

  4. identified Dec 07, 2021, 05:32 PM UTC

    Photo and thumbnail rendering in the web application and PhotoWorks has been restored. We are continuing to monitor for resolution from our upstream infrastructure provider for restoration of our data pipeline for Insights.

  5. identified Dec 07, 2021, 07:58 PM UTC

    We are continuing to work with our upstream infrastructure provider for restoration of our data pipeline for Insights.

  6. monitoring Dec 07, 2021, 11:47 PM UTC

    Our infrastructure provider has restored service. We are working now to bring current all data in our pipeline for Insights reporting and related APIs

  7. resolved Dec 08, 2021, 01:10 AM UTC

    This incident has been resolved.