Cigo Tracker Outage History

Started Jul 27, 2026, 01:46 PM UTC· resolved Jul 27, 2026, 01:51 PM UTC

1. investigating Jul 27, 2026, 01:46 PM UTC

   We are investigating reports of delays and errors when importing jobs via the Import Tool. Some users may experience failed or slow-processing imports during this time.

2. investigating Jul 27, 2026, 01:47 PM UTC

   9:15 AM EST — Investigating We are investigating reports of delays and errors when importing jobs via the Import Tool. Some users may experience failed or slow-processing imports during this time. 9:41 AM EST — Resolved This incident has been resolved. Import Tool functionality has returned to normal and all delayed imports have been processed. We apologize for any inconvenience. Duration: 26 minutes

3. resolved Jul 27, 2026, 01:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 29, 2025, 03:58 PM UTC· resolved Oct 30, 2025, 01:24 AM UTC

1. investigating Oct 29, 2025, 03:58 PM UTC

   We are currently experiencing service disruptions due to a Microsoft Azure outage affecting Cigo Tracker’s infrastructure. Our team is actively monitoring the situation and working closely with Microsoft to restore full functionality as soon as possible. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding.

2. investigating Oct 29, 2025, 04:09 PM UTC

   We are continuing to investigate this issue.

3. identified Oct 29, 2025, 05:10 PM UTC

   We’re aware of a Microsoft Azure service disruption that began around 16:00 UTC (12:00 PM EST) and is currently affecting the availability of some Cigo Tracker services. The issue is related to Azure Front Door (AFD), which is also impacting access to the Azure Portal and other Azure-hosted systems globally. Microsoft has acknowledged the issue and is actively working on mitigation steps, including failover options for affected infrastructure. Their latest update was posted at 16:57 UTC (12:57 PM EST) on October 29, 2025, confirming that investigations are ongoing. Our team continues to monitor the situation closely and will provide additional updates as Microsoft releases more information or once services stabilize. For live updates directly from Microsoft, you can check the Azure Status Page here: 🔗 https://azure.status.microsoft/en-us/status Thank you for your patience and understanding.

4. identified Oct 29, 2025, 07:02 PM UTC

   To ensure our clients’ operations remain uninterrupted, we have activated our failover. You can access the Cigo Tracker platform directly at https://app.cigotracker.com . Please note that our main website (home page) will remain temporarily inaccessible during this period. This decision was made intentionally to prioritize platform uptime and maintain delivery operations for all clients. Microsoft is actively deploying their last known good configuration and has begun recovering nodes to restore full connectivity. While progress is ongoing, Microsoft has indicated that protective safeguards are extending the overall recovery time. We will keep the system status as Degraded until we have collected sufficient data to confirm full stability, at which point we will mark the system as Operational. You can monitor Azure’s live updates here: https://azure.status.microsoft/en-us/status All times are reported in Eastern Standard Time (EST). Thank you for your patience and understanding as we continue to monitor and provide updates in real time.

5. identified Oct 29, 2025, 07:12 PM UTC

   We are continuing to work on a fix for this issue.

6. identified Oct 29, 2025, 07:15 PM UTC

   We are continuing to work on a fix for this issue.

7. monitoring Oct 29, 2025, 08:59 PM UTC

   We have successfully bypassed the Azure Front Door issue. While Azure continues to experience a global outage, Cigo Tracker remains fully operational through our failover. Please note that Cigo Tracker is currently accessible only via https://app.cigotracker.com . We will continue to closely monitor system performance and provide updates as the situation with Microsoft Azure progresses. Thank you for your continued patience and understanding.

8. resolved Oct 30, 2025, 01:24 AM UTC

   Microsoft has confirmed that all affected Azure services have been fully restored. Cigo Tracker systems were back to normal operation as of 16:59 EST. Following service stabilization, we have now reverted from our backup environment back to our primary infrastructure. All systems are operating normally, and performance remains stable. If you wish to receive the official Microsoft incident report once it is published, please email us at [email protected] , and we will make it available to you as soon as we receive it. Thank you for your patience and understanding throughout this Azure service disruption.

Read the full incident report →

Started Sep 15, 2025, 06:12 PM UTC· resolved Sep 15, 2025, 08:24 PM UTC

1. identified Sep 15, 2025, 06:12 PM UTC

   We recently experienced unexpected issues with SSL certification that disrupted database connections for several of our internal services. The Import Tool was initially affected but has since been fully restored. During our investigation, we also identified that the issue impacted the notifications subsystem for the Confirmation module, which relies on SMS and email sending. While the Import Tool is now operating normally, we are still working to restore full connectivity for Confirmation module notifications, along with a few other automated functions that may currently be degraded. Our engineering team has identified the root cause and is implementing safeguards to prevent similar issues during future SSL certification rotations. We sincerely apologize for the disruption and will continue to provide updates until all affected services are fully restored.

2. identified Sep 15, 2025, 06:52 PM UTC

   We have resolved the issues affecting the Import Tool and the Confirmation module notifications. Both are now fully operational. Our team is currently working on finalizing the recovery of a few remaining automated functions that may still be degraded. We will provide another update once all affected services are fully restored.

3. resolved Sep 15, 2025, 08:24 PM UTC

   The SSL certification issue that impacted several internal services, including the Import Tool and the Confirmation module notifications, has been fully resolved. All affected services are now fully operational. We have implemented safeguards to prevent similar issues during future SSL certification rotations. Thank you for your patience and understanding throughout this incident.

4. postmortem Sep 15, 2025, 08:28 PM UTC

   ### Summary On **September 15th, 2025**, we experienced service disruptions affecting the Import Tool and Confirmation module notifications. The root cause was a misidentified SSL certificate dependency during the scheduled **Azure Database for MySQL maintenance**, which included the rotation of the MySQL server’s SSL certificates. ### Impact * The Import Tool was unavailable for a period of time. * Confirmation module notifications \(via SMS and email\) were disrupted. * Some other automated functions experienced degraded performance. ### Root Cause While we were aware of the scheduled SSL certificate rotation as part of Azure’s maintenance process, our evaluation process incorrectly identified which certificates were being used by certain subsystems. As a result, several services lost connectivity when the certificates were rotated. ### Mitigation and Resolution * Our engineering team corrected the certificate configuration for the affected subsystems. * The Import Tool and Confirmation module notifications were restored to full functionality. * Safeguards are being added to ensure that future SSL rotations do not cause similar disruptions. ### Contributing Factors At the onset of the incident, we initially believed the outages were linked to updates and fixes deployed on Thursday and Friday of the previous week. This led to rollbacks that did not address the issue. Although unrelated to the outage, some of those updates remain rolled back and will be re-published in the coming days. ### Next Steps * We are reworking our dependency system analysis to more accurately map SSL and package dependencies across all services. * We are improving our monitoring and review processes for certificate rotations to ensure full coverage across all subsystems. * We will re-publish rolled-back updates after additional validation. ### Conclusion We sincerely apologize for the operational problems this incident caused. We take these issues seriously and are committed to ensuring that similar outages do not reoccur in the future.

Read the full incident report →

Started Jul 18, 2025, 04:28 PM UTC· resolved Jul 18, 2025, 07:10 PM UTC

1. investigating Jul 18, 2025, 04:28 PM UTC

   Our web application is currently unavailable. We’re actively investigating the cause of the outage. Earlier today, we performed some database changes which have since been reverted. We're currently assessing whether these changes contributed to the issue and working to restore full functionality as quickly as possible. We sincerely apologize for the inconvenience this is causing and appreciate your patience while we work to resolve the problem. Further updates will follow shortly.

2. monitoring Jul 18, 2025, 05:59 PM UTC

   The web application is now accessible again at https://app.cigotracker.com. We are closely monitoring system performance to ensure stability and confirm that all services are functioning as expected. We were also seeing some issues accessing the Cigo Tracker landing page (https://cigotracker.com), but this was also just reestablished. Thank you for your continued patience. We’ll provide another update once the incident is fully resolved.

3. resolved Jul 18, 2025, 07:10 PM UTC

   The web application outage has been resolved, and all systems are currently operating normally. While we've successfully isolated the scope of the issue, there are still some contributing factors that remain under investigation. Our team is continuing to analyze the event to determine the full root cause of today's outage and to implement measures that prevent recurrence. We sincerely appreciate your patience and apologize again for the disruption.

Read the full incident report →

Started Jun 23, 2025, 08:08 PM UTC· resolved Jun 23, 2025, 09:00 PM UTC

1. investigating Jun 23, 2025, 08:08 PM UTC

   We are currently investigating an issue related to an ongoing system update that is impacting platform performance and availability. Users may experience slow response times or intermittent access issues. Our engineering team is working to identify the root cause and restore full functionality as quickly as possible. We will share updates as progress is made.

2. identified Jun 23, 2025, 08:16 PM UTC

   We’ve identified the cause of the issue. While platform performance has recovered, the underlying system update did not complete successfully. Our team is conducting a thorough review of the root cause and will provide further updates as we determine the next steps.

3. resolved Jun 24, 2025, 02:02 AM UTC

   The issue has been fully resolved. We reverted the deployment that caused the disruption, and the related changes will be revised and rescheduled for a future release. Thank you for your patience and understanding throughout the investigation.

Read the full incident report →

Started Mar 19, 2025, 12:44 AM UTC· resolved Mar 19, 2025, 03:35 AM UTC

1. monitoring Mar 19, 2025, 12:44 AM UTC

   We want to inform you that we recently experienced network degradation affecting our services due to an ongoing issue within Microsoft's Azure infrastructure in the East US region. What Happened? According to Azure, between 13:09 UTC and 18:51 UTC, a fiber cut impacted network capacity in the region, leading to intermittent connectivity loss and increased latency. While Azure has since mitigated the issue, we observed disruptions in our own services between 7:20 PM and 8:27 PM (Eastern Time), specifically affecting connections between the Cigo Tracker web app and our Redis service. Current Status As of 8:27 PM UTC, network latencies have returned to normal, and service stability has been restored. However, to ensure a prompt and complete resolution, we have escalated this matter to Azure's Operations Support with a critical priority. We appreciate your patience and will continue monitoring the situation closely. If you experience any further issues, please reach out to our support team.

2. resolved Mar 19, 2025, 03:35 AM UTC

   After monitoring the situation over the past few hours, we can confirm that the immediate impact of the outage has been fully mitigated. Our services are stable, and network performance has returned to normal. We will provide a more detailed post-mortem once we receive a conclusive report from Microsoft's Azure Operations Support (OSS) team. Thank you for your patience and understanding. We sincerely apologize for any inconvenience this may have caused.

3. postmortem Apr 11, 2025, 03:40 PM UTC

   ## 📣 Incident Summary – March 18–19, 2025 **Service Impact on Cigo Tracker due to Azure Regional Outage** On March 18, 2025, Cigo Tracker experienced intermittent service disruption due to a regional outage within Microsoft Azure’s East US data center region. Below is a summary of the root cause, impact, and the steps being taken to prevent future occurrences. ### 🕒 What Happened? Azure’s East US region suffered two separate impact windows: * March 18, 13:37 to 16:52 UTC * March 18, 23:20 to March 19, 00:30 UTC The incident was triggered by a **third-party fiber cut** during external drilling work, which caused reduced network capacity in one of Azure’s Availability Zones. A **tooling failure** during Azure's recovery efforts later reintroduced traffic prematurely, leading to congestion and a second round of intermittent connectivity issues. ### 🔍 Root Cause 1. **Fiber Cut**: A construction-related accident physically damaged fiber cabling serving the East US datacenter, degrading network capacity. 2. **Router Maintenance**: A key router in the same zone was already under repair, limiting redundancy. 3. **Tooling Error**: Azure’s automated recovery system failed to fully isolate damaged infrastructure, inadvertently reintroducing traffic too early. 4. **Congestion Spillover**: The unexpected traffic load caused congestion to spread beyond AZ03 into neighboring zones. ### 🎯 Impact on Cigo Tracker While the Azure issue only affected a subset of inter-zone traffic in East US, this included infrastructure we rely on, resulting in **intermittent connectivity issues** for some customers during the incident windows. Core services were restored once Azure manually completed isolation and fiber recovery work. ### 🛠 Resolution Timeline * **13:37 UTC, Mar 18** – Outage begins due to fiber cut * **13:55 UTC** – Initial mitigation starts; traffic rerouted * **16:52 UTC** – First impact window ends * **23:20 UTC** – Second outage begins due to tooling error during recovery * **00:30 UTC, Mar 19** – Final mitigation complete * **06:50 UTC** – Full restoration of all infrastructure ### ✅ What Azure is Doing to Prevent Recurrence * Fixing tooling failures that allowed reintroduction of unready capacity \(by May 2025\) * Accelerating a capacity upgrade for the East US datacenter \(by July 2025\) * Architecting better safeguards to prevent impact from spreading across zones \(by February 2026\) We apologize for the inconvenience caused. Please rest assured that our team is working closely with Azure and continuing to invest in the resiliency of our platform. If you have any questions or would like help designing a more resilient setup, feel free to reach out to our support team. Thank you for your continued trust

Read the full incident report →

Started Nov 15, 2024, 03:15 AM UTC· resolved Nov 15, 2024, 03:15 AM UTC

1. resolved Nov 15, 2024, 05:00 AM UTC

   On November 14th, at approximately 10:15 PM, during a planned network maintenance session, we undertook a migration of core network routing resources. Unfortunately, this migration inadvertently impacted our DNS records, causing DNS Probe errors for some customers attempting to access our services. To address the issue, our operations team promptly remapped the affected DNS records and adjusted network restriction rules to restore connectivity. The total downtime was less than one hour. The root cause of this incident was an unforeseen interaction between the migration process and automatic changes to both networking rules and DNS records. While this was an unanticipated challenge, we’re grateful for the swift recovery efforts of our operations team. The issue has been fully resolved, and measures have been implemented to mitigate the risk of similar occurrences in the future. If you are still experiencing connectivity issues, we recommend: 1. Allowing additional time for DNS propagation to reach your network. 2. Performing a DNS flush on your network or operating system. If the issue persists, please contact us at [email protected], and we’ll be happy to assist you. We sincerely apologize for the inconvenience caused and appreciate your patience and understanding during this time.

Read the full incident report →