Apwide Outage History

Started Mar 10, 2026, 07:04 PM UTC· resolved Mar 10, 2026, 08:27 PM UTC

1. investigating Mar 10, 2026, 07:04 PM UTC

   We are currently investigating the issue.

2. investigating Mar 10, 2026, 07:32 PM UTC

   We identified a DNS resolution issue on one of our production nodes. The Golive service has been moved to another node and is now available again. We are currently working with our cloud provider to determine the root cause of the issue and implement a fix.

3. resolved Mar 10, 2026, 08:27 PM UTC

   We have identified and resolved the root cause of the issue. One of our nodes was unable to perform DNS resolution due to routing errors. The routing issues have now been fixed, and DNS resolution is working correctly. We apologize for the inconvenience caused.

Read the full incident report →

Started Feb 05, 2026, 07:46 AM UTC· resolved Feb 05, 2026, 02:56 PM UTC

1. investigating Feb 05, 2026, 07:46 AM UTC

   We are currently experiencing a performance issue on Golive that is making the platform inaccessible. We are actively investigating the root cause. We apologize for the inconvenience.

2. investigating Feb 05, 2026, 08:14 AM UTC

   We are continuing to investigate this issue.

3. investigating Feb 05, 2026, 08:23 AM UTC

   We have disabled traffic from Atlassian webhooks (issue created/updated/deleted events). As a result, Go-Live automation and scheduling conflict checks are currently unavailable. We are continuing to investigate the issue.

4. investigating Feb 05, 2026, 10:18 AM UTC

   Disabling webhook traffic has restored Golive for app navigation. We are currently working on performance fixes so that we can re-enable webhooks (Golive automation and the Scheduling Conflict Checker). We will provide an update soon.

5. identified Feb 05, 2026, 11:20 AM UTC

   A fix has been implemented and deployed to production. We are reopening webhook traffic to assess whether the solution meets performance requirements. This may result in some instability over the next hour.

6. identified Feb 05, 2026, 11:44 AM UTC

   Although the fix helped mitigate the performance issue on one of our components, it is not sufficient to handle the current webhook traffic load. We are continuing to work on performance improvements, but in the meantime, we have had to disable webhook traffic again.

7. monitoring Feb 05, 2026, 02:01 PM UTC

   A second fix has been deployed to production. Webhook traffic has been re-enabled, and the load appears to be handled correctly. Capabilities such as conflict checking and automation seem to be fully functional. We are continuing to monitor the situation.

8. resolved Feb 05, 2026, 02:56 PM UTC

   Over the past hour, we have experienced several traffic peaks on Atlassian webhook calls, and our updated systems have handled the load without any issues. We now consider this issue to be resolved. We apologize for the inconvenience caused.

9. postmortem Feb 05, 2026, 02:58 PM UTC

   **07:23 UTC** – The support team was alerted by monitoring that some database queries were taking longer than expected. **07:34 UTC** – Our uptime probes began raising alerts indicating that Golive was no longer accessible to end users. **07:46 UTC** – The support team updated the status page to notify users of a Golive outage. **08:23 UTC** – The support team identified an abnormal traffic spike originating from Atlassian webhooks. Under normal conditions, traffic peaks reach around 1,000 webhook calls per minute; however, we experienced a sustained spike exceeding 9,000 calls per minute. This caused one of our components to open more database connections than it was designed to handle, forcing it to restart repeatedly. To protect the Golive application, webhook traffic was temporarily disabled. The component stopped restarting and access to Golive was restored. **11:20 UTC** – The engineering team implemented an initial performance improvement on the affected component. The fix was deployed to production, and webhook traffic was re-enabled to evaluate performance. **11:44 UTC** – Webhook traffic was suspended again as the initial fix proved insufficient. Although database connections were no longer an issue, the component was still receiving excessive request traffic. **14:01 UTC** – A second improvement was implemented, this time at the gateway level. The fix was deployed to production and webhook traffic was re-enabled. No further restarts were observed, and Golive appeared to be fully operational. **14:45 UTC** – After several traffic peaks reaching up to 5,000 webhook calls per minute, all components continued to operate normally, with sufficient margins in terms of database connections and execution threads. The incident was closed. With the additional performance improvements, the infrastructure now appears capable of handling a significantly higher load than originally designed. Furthermore, traffic should be better segregated to isolate asynchronous workloads \(webhooks, conflict checking, automation, etc.\) from transactional traffic \(UI application usage\). This separation should help mitigate the risk of asynchronous traffic impacting normal user navigation.

Read the full incident report →

Started Nov 05, 2025, 06:20 AM UTC· resolved Nov 05, 2025, 10:07 AM UTC

1. investigating Nov 05, 2025, 06:20 AM UTC

   We are currently experiencing an outage affecting Golive and Time Squad since 6:00 AM UTC. Our team is actively working to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience.

2. investigating Nov 05, 2025, 06:58 AM UTC

   We’ve identified that the issue is coming from our hosting provider. We’re currently in contact with their teams and are working together to resolve the problem as soon as possible. Thank you for your continued patience and understanding.

3. monitoring Nov 05, 2025, 08:19 AM UTC

   Our hosting provider has resolved the issue. You can find details about the incident on their status page: https://public-cloud.status-ovhcloud.com/incidents/9qmxv0gbky43 All systems are currently being monitored, and both Golive and Time Squad are now fully operational. Thank you for your patience and understanding throughout this incident.

4. resolved Nov 05, 2025, 10:07 AM UTC

   This incident has been resolved.

5. postmortem Nov 05, 2025, 01:15 PM UTC

   **Outage Recap** * **2025-11-05 at 5:56 AM UTC**, our monitoring system began raising alerts indicating that some of our services were unable to communicate with the database. * **6:20 AM UTC:** We reported an outage affecting the **Golive** and **Time Squad** services on our status page. * **6:26 AM UTC:** We suspected the issue originated from our hosting provider’s managed database service. We contacted their support team and opened a ticket. * **6:57 AM UTC:** The hosting provider confirmed the issue on their side and announced a general outage on their platform. [Incident link](https://public-cloud.status-ovhcloud.com/incidents/9qmxv0gbky43) * **7:58 AM UTC:** The hosting provider reported that the issue had been resolved. We restarted our **Golive** and **Time Squad** services. * **8:19 AM UTC:** All of our services were fully restored, and we updated our status page accordingly. * **8:31 AM UTC:** The hosting provider informed us that the root cause was a **DNS issue** on their end. We apologize for the inconvenience caused and appreciate your patience and understanding.

Read the full incident report →

Started Apr 29, 2025, 08:00 AM UTC· resolved Apr 29, 2025, 08:30 AM UTC

1. resolved Apr 29, 2025, 08:49 PM UTC

   Golive & Time Squad were unavailable this morning for a duration of approximately 30 minutes. We are currently conducting an investigation to identify the root cause.

2. postmortem May 01, 2025, 01:35 PM UTC

   **After a thorough analysis, we identified that one of our components had reached the maximum number of database connections it was allowed to establish.** As traffic continued to grow, our health check system marked the component as unavailable and automatically restarted it in an attempt to restore normal operation. It took approximately 30 minutes for the component to come back online with a sufficient number of available connections to handle the load. **Remediation:** We have increased the maximum number of allowed connections for this particular component. We sincerely apologize for any inconvenience this outage may have caused. Please rest assured that we are committed to providing the best service possible.

Read the full incident report →

Started Apr 17, 2025, 01:30 AM UTC· resolved Apr 17, 2025, 02:30 AM UTC

1. resolved Apr 17, 2025, 01:55 PM UTC

   Golive & Time Squad were unavailable this morning for a duration of 1 hour and 10 minutes. The root cause of the issue was a problem accessing the application databases. After contacting our hosting provider, they confirmed that a network maintenance was scheduled for this morning. Traffic was supposed to be rerouted, but a configuration issue caused a routing error. We apologize for the inconvenience caused.

Read the full incident report →