CrystalCommerce experienced a critical incident on March 13, 2023 affecting Frontend, lasting 2d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 13, 2023, 05:37 PM UTC
We've seen periodic spikes in response time for frontend websites. Some of them lasting significantly longer than others. Our engineering team is currently working on identifying the cause of the issue so we can fix it ASAP.
- identified Mar 13, 2023, 08:48 PM UTC
The issue has been identified and a fix is being worked on.
- identified Mar 14, 2023, 03:24 AM UTC
We are still seeing outages every 90 minutes or so, and are predicting that we'll have another outage from around 7AM-11AM Pacific for frontend websites. We know that the issue is with our NAS filesystem, but haven't yet been able to find what specifically it is. Our developers are sticking with it until we can get this resolved.
- resolved Mar 15, 2023, 10:00 PM UTC
We found the issue and deployed a fix. We don't expect it to return, but are still monitoring things just to make sure.