Eptura Workplace incident
S2 - Space - Not able to upload large floor plans
Eptura Workplace experienced a minor incident on June 8, 2023 affecting Space Module, lasting 13d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 08, 2023, 06:13 PM UTC
We are currently investigating an issue with Space - Not able to upload large floor plans. Our Engineering team is currently investigating to determine the cause of the disruption.
- investigating Jun 08, 2023, 08:21 PM UTC
We are continuing to investigate this issue.
- investigating Jun 08, 2023, 10:45 PM UTC
We are continuing to investigate this issue.
- investigating Jun 09, 2023, 12:15 PM UTC
A fix has been implemented for "Space - Not able to upload large floor plans". We are moving into the Monitoring Phase for the next 2 hours.
- monitoring Jun 09, 2023, 12:20 PM UTC
A fix has been implemented and we are monitoring the results.
- investigating Jun 09, 2023, 01:59 PM UTC
After monitoring, we are moving back into an Investigating phase. Our internal teams continue to navigate the root cause of this issue, "Space - Not able to upload large floor plans". We appreciate your continued patience.
- investigating Jun 09, 2023, 05:54 PM UTC
We are continuing to investigate this issue.
- identified Jun 09, 2023, 06:50 PM UTC
The team has identified the issue and are working on a implementing a fix for S2 - Space - Not able to upload large floor plans.
- identified Jun 09, 2023, 09:13 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Jun 09, 2023, 10:44 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jun 10, 2023, 03:10 AM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 12, 2023, 02:54 PM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 12, 2023, 06:35 PM UTC
On Friday, June 9th, there were upgrades that were made to improve internal systems. Our internal team continue to monitor the initial "S2 - Space - Not able to upload large floor plans". This disruption to floorplans continues to be a work in progress. We appreciate your patience while we work through this.
- monitoring Jun 12, 2023, 10:25 PM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 13, 2023, 03:02 PM UTC
Monitoring continues today for full resolution. Thank you for your continued patience as we work through this disruption.
- monitoring Jun 14, 2023, 01:05 AM UTC
Monitoring continues for Space - Not able to upload large floor plans through this evening. We will continue to post further updates in the morning on the progress our internal teams have made. Thank you for your continued patience.
- monitoring Jun 14, 2023, 03:16 PM UTC
After days of monitoring we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.
- resolved Jun 21, 2023, 09:35 PM UTC
This incident has been resolved.
- postmortem Jun 22, 2023, 09:18 PM UTC
**iOFFICE Detailed Root Cause Analysis – Severity 2 – June 8, 2023** **Space - Unable to Load Large Floor Plans** **Description:** On June 8, 2023, at approximately 11:46am MDT, the Support team received reports of Space - Not able to upload large floor plans through SoftSpace or Admin Space. **Type of Event:** Service Disruption **Services\\Modules Impacted:** Production – SWF Drawing Service **Timeline:** Thursday June 8, 2023 On June 8, 2023, at approximately 11:46am MDT, the Support team received reports of Space - Not able to upload large floor plans through SoftSpace or Admin Space. The issue was reported to engineering and they acknowledged that they are investigating the issue. All customers were alerted about the disruption at 12:13pm MDT through our Status Page that we are in an Investigation Phase on our Status Page. 7:28pm MDT Support is notified by our dev ops team for testing on a possible resolution. Friday June 9, 2023 Our support team begins to test the possibility of the fix at 6:15am MDT and the Status Page is updated to Monitoring Phase. At 7:13am MDT support confirms the disruption is ongoing. The engineering team continues to investigate, and customers are notified that we are moving back into an Investigation Phase which continues till approximately 3:15pm MDT. Our engineering team has found a temporary fix by upgrading internal systems, so that customers will be able to upload plans through the weekend. Meanwhile, the engineering team works on a permanent fix through the weekend. They will also monitor the temporary fix for any issues. Customers were alerted that monitoring will continue through the weekend. Monday June 12, 2023 Investigation and Monitoring continues. Tuesday June 13, 2023 At 8:09am MDT the Engineering team confirms that they have identified a more permanent solution for the issue and a hotfix should be released as soon as it passed QA. At 10:51pm MDT Dev Ops confirms that the hot fix has been released to production. Wednesday June 13, 2023 At 8:55am MDT the Support and Drafting Teams have confirmed the fix and at 9:16am MDT Support confirms that as of June 9th, no additional reports were made to the support team about this issue. All customers have confirmed that this issue has been resolved and the Status Page is moved into a Resolved Phase. **Total Duration of Event:** 5 days, 21 hours, 30 mins **Remediation:** We upgraded to the latest version of Java SWF drawing service. We also increased the memory and CPU power for the cluster.