Snapsheet incident
Issue when trying to load photos and documents within a claim (AWS)
Snapsheet experienced a minor incident on December 5, 2022 affecting US and EU, lasting 2d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 05, 2022, 04:07 PM UTC
Some users are unable to load photos and documents within a claim. We are currently investigating and following up with AWS.
- investigating Dec 05, 2022, 08:36 PM UTC
The Snapsheet Team is still troubleshooting with AWS to resolve the issue. AWS has escalated the issue, and we are awaiting feedback from their Engineers.
- investigating Dec 05, 2022, 10:07 PM UTC
We've found more information that suggests the problem is in the AWS network handling some IPv6 requests to files in vice-backend production. The AWS S3 team is helping to investigate. No ETA at this time.
- investigating Dec 06, 2022, 03:58 AM UTC
We spoke with a network specialist at AWS who will be working with his team to bring all eyes in AWS on this issue as we continue to try and identify the source of the disruption we are seeing when accessing files in vice-backend production. They will continue the investigation later tonight and also tomorrow morning.
- investigating Dec 06, 2022, 04:00 AM UTC
We are continuing to investigate this issue.
- identified Dec 06, 2022, 02:34 PM UTC
We have identified and are currently testing a workaround. We are still working with AWS to obtain an RCA and resolution to the original issue. We will continue to provide updates as they become available.
- monitoring Dec 06, 2022, 06:48 PM UTC
We’ve rolled out a workaround to mitigate this issue as we are still working with AWS.
- resolved Dec 07, 2022, 04:09 PM UTC
Our workaround has resolved this issue, and we are continuing to work with AWS for an RCA