Webshipper incident

Drop point service down

Notice Resolved View vendor source →

Webshipper experienced a notice incident on November 13, 2022, lasting 17m. The incident has been resolved; the full update timeline is below.

Started
Nov 13, 2022, 09:03 AM UTC
Resolved
Nov 13, 2022, 09:21 AM UTC
Duration
17m
Detected by Pingoru
Nov 13, 2022, 09:03 AM UTC

Update timeline

  1. investigating Nov 13, 2022, 09:03 AM UTC

    Unfortunately our drop point service is currently unavailable

  2. resolved Nov 13, 2022, 09:21 AM UTC

    This incident has been resolved.

  3. postmortem Nov 13, 2022, 10:00 AM UTC

    Our Amazon Webservice ECR repository which holds our build images of all our services, unfortunately, had a rule set up on drop point services to delete images older than 90 days. And drop point service had not been updated for 90 days since this weekend. Thus the ECS \(Which uses these images\) had nothing to pull since the old ones were deleted, this, therefore, resulted in no drop point services. We have now changed the policy rules to only delete if there are more than 5 images instead of 90 days. This will ensure that we will not run into this issue again.