Squiz experienced a minor incident on June 23, 2025 affecting Squiz SaaS Hosted Instances, lasting 1h 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 23, 2025, 01:34 AM UTC
We are currently investigating a performance issue affecting DXP customers in Australia. We have identified the issues and are working to resolve it.
- investigating Jun 23, 2025, 01:55 AM UTC
We are continuing to investigate this issue with the DXP hosting environment and will post more updates once we have any more information on this issue.
- identified Jun 23, 2025, 02:01 AM UTC
We've deployed a change to resolve the the immediate issue and are monitoring while working on additional fixes.
- monitoring Jun 23, 2025, 03:00 AM UTC
We are continuing to monitor the fix in place to see what errors are solved and if any further issues appear. We shall post further once we have more information.
- resolved Jun 23, 2025, 03:31 AM UTC
We have now resolved the issue here and performance should be back to normal. Feel free to get in touch via My.Squiz.Net to let us know if you have any issues you believe are related.
- postmortem Jun 26, 2025, 11:36 PM UTC
# **Summary** **On Monday, June 23, 2025, customers may have experienced intermittent periods of degraded performance. This disruption was attributed to the exhaustion of specific IP address ranges within our private Amazon Web Services \(AWS\) networks. This was promptly detected, and our engineering teams swiftly initiated remediation efforts. The issue was identified and resolved within a short timeframe, restoring full service functionality and optimal performance levels.** #### **Customer Impact:** * **Incident Duration: 23 Jun 2025, 11:06- 23 Jun 11:33 \(AEST\)** * **Impact: We have noticed that only a small number of clients have been affected by this incident.** ## **Issue, Resolution, and Mitigation** **Root cause Analysis** **The underlying cause of this incident was the unanticipated depletion of available IP addresses within designated private subnets in our AWS environment. A confluence of factors, including a surge in specific resource utilization and the rapid provisioning of new instances, led to a temporary bottleneck in IP availability.** **Resolution Actions** ### **1. Identification:** * **Squiz Support Teams proactively identified errors relating to this issue** ### **2. Action:** * **Releasing recently provisioned IP addresses** ### **Follow-up Actions** ### **1. Squiz Actions:** * **To prevent future occurrences, Squiz has completed the migration of specific infrastructure to a larger subnet.**