Eptura Workplace experienced a minor incident on January 19, 2024 affecting Space Module, lasting 2d 16h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 19, 2024, 09:21 PM UTC
We are currently investigating an issue with floor plan tiles not loading.
- identified Jan 19, 2024, 10:13 PM UTC
We've identified the issue and are working towards a resolution. We appreciate your patience during this disruption.
- resolved Jan 22, 2024, 01:34 PM UTC
On the morning of Saturday, January 20, 2023, at approximately 6:40am MST, our engineering team had released a Hot fix to resolve the issue of "Floor plan tiles are not loading". Monitoring continued throughout the weekend. No additional reports have been made to our support team. A detailed RCA will be released within 10 business days. We appreciate your continued patience as we worked through this disruption.
- postmortem Feb 22, 2024, 03:37 PM UTC
**Eptura Workplace Detailed Root Cause Analysis – Severity 2 – January 22, 2024** **Floor Plan Tiles Not Loading** We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** On Friday, January 19, 2024, at approximately 9:00am EST a report came into support advising that floor plan tiles are not loading. After an initial investigation from the support team, an internal ticket was created to further investigate the incident. All customers were made aware of the Severity 2 through the Eptura Workplace Status Page. **Type of Event:** Performance Degradation in the Space Module **Services/Modules Impacted:** Production/ Space Module **Timeline:** On Friday, January 19, 2024, at approximately 9:00am EST an initial report came into support advising that floor plan tiles are not loading. After an initial investigation from the support team, an internal ticket was created to further investigate the incident. All customers were made aware of the Severity 2 through the Eptura Workplace Status Page at 4:16pm EST. Investigation continued through the evening and our Engineering team found the resolution and the fix was implemented on Saturday morning January 20th around 8:40am EST. Monitoring continued for the rest of the day and on Sunday. All customers were notified that the incident has moved into a resolved phase on Monday, January 22, 2024, at 8:35am EST. **Total Duration of Event:** 16hrs 24mins **Remediation:** The team reverted recent changes that updated a third-party library and re-populated an image cache. **Root Cause Analysis:** Our image processing service started crashing when under heavy load after updating a third-party library. These crashes corrupted or removed some images stored in our local image cache. **Preventative Action:** Made changes to the code to make it more resilient to missing images by regenerating images on-demand. Adding tests to verify the tiles are still loading even if cache doesn’t exist.