Streamtime experienced a major incident on May 22, 2025 affecting Frontend and API, lasting 9h 44m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 22, 2025, 09:20 AM UTC
We have been made aware of some customer receiving a 'Your connection is not private' browser message when navigating to their Streamtime URL. Our team are investigating as a matter of urgency.
- identified May 22, 2025, 09:29 AM UTC
Our technical team have identified a domain issue that they are investigating a solution for.
- monitoring May 22, 2025, 12:10 PM UTC
Our continued monitoring of the propogation is showing a decline in the small number of customers that have been affected. We are hopeful that the changes made will continue to update DNS servers and resolve access for those still affected. We will continue to closely watch progress and exhaust all avenues available to us to fully resolve the issue as quickly as possible. Our team are available via email ([email protected]) should you have any additional concerns. We apologise and do appreciate the impact this continues to cause.
- monitoring May 22, 2025, 01:28 PM UTC
We’re continuing to see a steady decline in the number of customers affected. Our team are actively exploring every option and are monitoring closely to ensure progress continues. Thank you for your continued patience.
- monitoring May 22, 2025, 02:47 PM UTC
While there are no major changes to report just yet, we’re continuing to monitor things closely. Our team remains on hand to help so please reach out to [email protected] if there’s anything you need.
- monitoring May 22, 2025, 04:41 PM UTC
We have now exhausted all technical paths on our side. There are third parties involved that, unfortunately, we cannot control but we will be working closely with them to get this resolved as quickly as possible. We're still keeping an eye on things and are here if you need any support (feel free to reach out via [email protected]).
- resolved May 22, 2025, 11:12 PM UTC
Our technical team have confirmed with all relevant 3rd parties that the change made at 10.10pm AEST yesterday evening did in fact resolve the core issue. The subsequent delay in the solution reaching some of the affected parties was caused by the propagation process. If your access to Streamtime remains affected, please contact [email protected] for our technical team to investigate your specific situation. Our focus now turns to reviewing the cause and effect of this issue so we can ensure any critical failure points are resolved and future scenarios are avoided. We will endeavor to post an update here in the coming days. Once again, we apologise for the inconvenience this undoubtedly caused a small number of customers. Michael O'Riley Head of Operations
- postmortem May 23, 2025, 02:54 AM UTC
# **Streamtime Technical Postmortem - Domain Suspension** **Incident ID:** STM-2025-0522-001 **Date:** May 27, 2025 **Duration:** 15\+ hours \(ongoing propagation\) **Severity:** High **Impact:** Customer access disruption due to domain suspension. ## **Summary** On May 22, 2025, at 18:00 AEST, customers began experiencing issues accessing Streamtime presenting as either a verification of domain request page, or an SSL certificate validation failure "Your connection is not private". The incident was caused by an incomplete ICANN registration update following domain technical contact changes during routine domain renewal. ## **Timeline \(AEST\)** * **May 7, 2025:** Domain renewal completed. Technical contact details updated to new email address. Key account owner details unchanged. * **May 10, 2025:** Automated email sent from domain provider to a generic, non-specific, non-technical email address asking for new technical contact details to be confirmed. No other contact received. * **May 22, 2025 18:00:** First customer report received of certificate validation issues. * **May 22, 2025 20:30:** Single email requesting confirmation of technical contact details found and actioned. Investigations continued for other possible root causes. * **May 23, 2025 06:00:** Technical support for domain provider available - discussions begin to confirm root cause. * **May 23, 2025:** DNS propagation ongoing. * **May 23, 2025:** Northern hemisphere timezone online with confirmation from majority of previously affected customers that the issue had resolved. Some customers remain affected. * **May 24, 2025:** Propagation process reaches 48hrs at 20:30 AEST. * **May 26, 2025:** Confirmation from all previously affected Southern Hemisphere customers that the issue is resolved. * **May 26, 2025:** No sign of domain related issues in our network monitoring system. * **May 26, 2025:** Confirmation from domain host that domain was suspended due to the updated technical contact email address not being verified. ## **Root Cause Analysis** The incident originated from an ‘incomplete’ domain registration update process: 1. Technical contact information was successfully updated in the domain host administration panel during renewal. 2. Automated email received in a generic email inbox asking for updated technical contact details to be confirmed. 3. Timeframe for validation of details expired without additional warning or contact from the domain provider. Domain suspended without notification. 4. Due to lack of confirmation, ICANN registration records were not updated, leading to a domain suspension. ## **Resolution** 1. Located and verified the legitimacy of the confirmation email from the domain host. 2. Completed the technical contact confirmation process. 3. ICANN registration records subsequently updated. 4. Initiated DNS propagation across global DNS infrastructure & monitored progress. ## **Customer Impact** Due to the distributed nature of DNS propagation, impact varies by customer location and ISP DNS configuration. Some customers may experience delayed resolution or temporary new occurrences of the issue as DNS updates propagate globally. It is generally accepted that global propagation can take up to 48hrs. ## **Immediate Actions Taken** * Troubleshooting of root cause. * Incident created at [https://streamtime.statuspage.io/incidents/hk5ysgq63z9y](https://streamtime.statuspage.io/incidents/hk5ysgq63z9y) with incident subscription option to receive updates automatically. * Consistent, proactive communication to customers on progress. * Confirmed technical contact details with domain registrar. * Confirmed the ICANN registration records would be automatically updated. * Established monitoring for continued customer reports. * Provided customer workaround guidance \(alternative network connections\). * Contacted all 3rd party stakeholders and continued to push for information to confirm root cause. ## **Process Improvements** 1. **Domain Management Consolidation:** We have initiated the process to transfer domain management from our existing provider to GoDaddy for 24/7 domain hosting support availability. 2. **Notification Enhancement:** Update notification systems to target both individual team members and shared support inboxes. ## **Next Steps** * Continue to monitor support channels for customer responses as to their connectivity. * Continue to monitor relevant errors received by customers via our network monitoring system. * A full review of all critical accounts to be scheduled and completed internally. Confirming correct notifications are enabled and are sent to monitored email accounts. * Review and ensure existing internal disaster recovery plan is updated based on any account changes.