Spatial Cloud Service Degradation for a subset of customers across all spatial releases

Update timeline

1. investigating Nov 27, 2024, 02:52 AM UTC

   We are investigating an issue impacting service for 2024A Spatial cloud. A subset of customers for 2024A Spatial Cloud are experiencing sub-optimal performance. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Nov 27, 2024, 03:49 AM UTC

   Our team has identified the issue causing sub-optimal performance and are now working on a fix. We will provide another update in 60 minutes, or sooner if new information becomes available.

3. identified Nov 27, 2024, 04:48 AM UTC

   Further investigation has been undertaken and this issue is impacting a subset of customers on spatial cloud across ALL releases. We have identified the issue on our data layer and continuing to work on remediation. Intramaps appears to be functional however Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance. We will provide a further update in 2 hours, or sooner if new information becomes available.

4. identified Nov 27, 2024, 06:53 AM UTC

   We are continuing to work on remediation. Intramaps continues to be functional. Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance and are likely to present with errors at times that can often be clicked through. We will provide a further update at 9am AEST, or sooner if new information becomes available.

5. identified Nov 27, 2024, 10:25 PM UTC

   A fix is currently being implemented. We expect the fix to be completed at approximately 3pm AEST today. Intramaps continues to be functional. Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance and are likely to present with errors at times that can often be clicked through. We will provide a further update at 1pm AEST, or sooner if new information becomes available.

6. identified Nov 28, 2024, 02:31 AM UTC

   The fix that is currently being implemented remains on track for completion by 3pm AEST today. Intramaps continues to be functional. Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance and are likely to present with errors at times that can often be clicked through. We will provide a further update at 3pm AEST, or sooner if new information becomes available.

7. monitoring Nov 28, 2024, 03:27 AM UTC

   Our team has verified the implementation of a fix is complete for the subset of customers on Spatial Cloud that have been affected by sub-optimal performance. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

8. resolved Nov 28, 2024, 06:07 AM UTC

   After 2 hours monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

9. postmortem Dec 10, 2024, 06:02 AM UTC

   **Issue Summary:** On Wednesday 27 November 2024 a subset of Spatial Cloud environments experienced performance degradation for Map Manager, Tile Engine and Config Manager. Often presenting with errors at times that could be clicked through. **Root Cause Analysis:** At 08.50am the storage was increased on the secondary Storage system as a review indicated it was running out of space. This caused the throughput to sit at 100%, leading to performance and load issues. The throughput was high because the storage increase took place during Business-As-Usual \(BAU\) operations. **Corrective Measures:** Monitored the storage optimization until completion, which completed at 01:23 PM on 28 November, after which performance and loading returned to normal. **Preventive Measures:** Reviewed the standard operating procedure in line with alert refinement.