HyperTrack Outage History

Started Oct 29, 2026, 10:21 PM UTC· resolved Oct 30, 2026, 12:26 AM UTC

1. investigating Oct 29, 2026, 10:21 PM UTC

   Currently AWS is experiencing issues with AWS Gateway functionality that directly affects HyperTrack platform components. We are experiencing elevated API errors and timeouts affecting API/ connectivity and dashboard access. Data processing is delayed but continues. We’ll provide updates as we learn more.

2. identified Oct 29, 2026, 10:22 PM UTC

   An AWS issue identified as the root cause of performance degradation.

3. monitoring Oct 29, 2026, 10:34 PM UTC

   Update from AWS: [3:30 PM PDT] Our efforts to mitigate the increased invoke error rates and latencies are still underway. Our continued advice is to retry any failed requests with exponential backoff. While we don't have an estimate for how long our efforts will take we will keep you informed of our progress by providing you with another update by 4:30 PM PDT or sooner as new information becomes available.

4. monitoring Oct 29, 2026, 11:29 PM UTC

   Our API error rates and SDK traffic have returned to normal. We’re monitoring for continued stability while AWS’s regional incident remains open.

5. resolved Oct 30, 2026, 12:26 AM UTC

   HyperTrack systems are stable while the AWS issue remains formally open. We are resolving this incident and continue to monitor for the latest developments from AWS.

Read the full incident report →

Started Jul 09, 2026, 10:00 AM UTC· resolved Jul 09, 2026, 07:25 PM UTC

1. resolved Jul 15, 2026, 04:10 AM UTC

   Summary On July 9, 2026, beginning at 10:00 UTC, devices attempting to establish or re-establish live connections to the HyperTrack platform experienced failures and delays. Devices that were already connected and tracking were largely unaffected, but newly starting devices — including workers beginning shifts during the window — experienced delayed or missing location data until service was fully restored at 19:25 UTC. What happened An unusual pattern in a production customer's orders API usage caused a large number of devices to stop tracking simultaneously, which triggered an internal data-reprocessing burst far larger than our systems had previously encountered. Although the database recovered quickly, the brief outage disconnected a large portion of the device fleet at once. The resulting wave of simultaneous reconnection attempts overwhelmed a connection-handling limitation in our real-time gateway service, creating a self-sustaining backlog that prevented new devices from connecting reliably for the remainder of the window. At 19:14 UTC we deployed a fix that bounds request handling in the gateway service; connectivity recovered within minutes and was fully restored by 19:25 UTC. Customer impact * Devices starting sessions during the window experienced delayed first location updates — in the worst cases by several hours — and a portion never delivered location data for orders created during the window. * Some orders completed during the window are missing arrival or on-site time data. * Location data for devices that remained connected was captured normally. What we are doing to prevent recurrence * Fixed the database connection-handling configuration that allowed the burst to exhaust capacity. * Fixed the gateway connection-handling limitation: bounded request handling, timeouts, and load-shedding so mass reconnections can no longer create a sustained backlog. * Capping and pacing the internal reprocessing behavior that generated the burst. * New monitoring and alerting across every layer involved, tuned to detect this failure pattern well before customer impact.

Read the full incident report →

Started Mar 19, 2026, 04:30 AM UTC· resolved Mar 19, 2026, 04:30 AM UTC

1. monitoring Mar 19, 2026, 04:30 AM UTC

   Mar 19, 04:30 UTC Resolved - A platform deployment caused an elevated rate of API errors between 21:08 UTC to 21:47 UTC. New systems were provisioned to mitigate the issues. API degradation is fully resolved. All APIs are operating normally, and we are monitoring closely.

Read the full incident report →

Started Mar 18, 2026, 10:07 PM UTC· resolved Mar 19, 2026, 04:30 AM UTC

1. resolved Mar 18, 2026, 10:07 PM UTC

   A platform deployment caused an elevated rate of API errors between 21:08 UTC to 21:47 UTC. New systems were provisioned to mitigate the issues. API degradation is fully resolved. All APIs are operating normally, and we are monitoring closely.

Read the full incident report →

Started Feb 03, 2026, 11:30 PM UTC· resolved Feb 03, 2026, 11:30 PM UTC

1. monitoring Feb 03, 2026, 11:30 PM UTC

   Feb 3, 23:30 UTC Resolved - A system update caused an elevated rate of API errors between 15:21 and 15:48 PST ( 23:21 to 23:48 UTC ). The update has been mitigated and the issue is fully resolved. All APIs are operating normally, and we are monitoring closely.

Read the full incident report →

Started Sep 10, 2025, 12:08 PM UTC· resolved Sep 10, 2025, 06:15 PM UTC

1. investigating Sep 10, 2025, 12:56 PM UTC

   We are currently investigating this issue.

2. investigating Sep 10, 2025, 01:57 PM UTC

   We are continuing to investigate this issue.

3. identified Sep 10, 2025, 03:07 PM UTC

   We resolved outstanding issues in the platform and are working with AWS to go through the steps of unlocking our AWS account required to resume platform operations.

4. resolved Sep 10, 2025, 06:15 PM UTC

   This incident has been resolved.

5. postmortem Sep 12, 2025, 04:49 PM UTC

   # **Summary** During the [incident](https://status.hypertrack.com/incidents/rslmyh4y3yy3) on Friday, September 5th, attackers gained access to our AWS account using a leaked key that had been inadvertently exposed in a JavaScript build. Within minutes, the attackers created concealed backdoor access resources. AWS and HyperTrack cleanup operations removed them, though a few resources initially went undetected. On September 10, attackers used the remaining backdoor resources to access the AWS account used by the HyperTrack platform and launched large-scale crypto-mining operations across multiple non-default AWS regions. The attack used compromised third-party AWS accounts unrelated to HyperTrack. This triggered AWS automated abuse detection, which temporarily restricted account access at 11:56 UTC. After investigation and remediation, AWS restored full account functionality by 18:10 UTC. Importantly, **no customer data or platform systems were accessed**. The incident was contained to crypto-mining activity. # **Timeline** **September 10, 2025 \(Wednesday\)** * **10:51 UTC** - Attacker accessed account resources from third-party compromised AWS accounts. * **10:53 UTC** - Crypto-mining workloads launched in new regions. * **10:57–11:00 UTC** - Multiple regions disabled again to hide activity. * **11:43–11:44 UTC** - Attacker switched between external AWS accounts from a new proxy IP address. * **11:56 UTC** - AWS issued compute resources abuse alert and started shutdown process. * **12:08 UTC** - Services were temporarily restricted. * **12:10 UTC** - HyperTrack engineers started responding to the incident and engaged with the AWS security team. * **15:07 UTC** - HyperTrack engineers removed all crypto mining resources, including all hidden non-default regions. * **18:10 UTC** - AWS security team confirmed restrictions lifted and account access restored. HyperTrack platform started receiving events and processing API requests. # **Next Steps** We recently detected and contained unauthorized access attempts related to leaked CI/CD credentials. Our investigation confirmed that no customer data was accessed and build pipelines for platform code repositories have been hardened. In response to the two incidents, we have taken immediate actions to strengthen our security posture in the coming weeks. Since the first incident, we implemented a code deployment freeze to address necessary steps to harden platform environments. These steps include the following: * **Improve Incident Containment & Remediation** * Multiple meetings with AWS security team members for account status and configuration review. * Continue to monitor CloudTrail for anomalies with alerting enabled. * **Continue Strengthening Identity & Access Measures** * Apply tighter Service Control Policies to limit usage to approved regions. * **Expanded Security Monitoring** * Review and update GuardDuty, Inspector, Security Hub, and IAM Access Analyzer configurations across all accounts and regions. * Configure additional alerting for high-severity findings and account activity to ensure immediate response. * **Implement Governance & Ongoing Improvements** * Aggregate all security findings centrally for faster triage and remediation. * Review Cognito and network configurations to ensure no hidden backdoors exist. * Evaluate AWS Control Tower and additional protections \(WAF, Firewall Manager\) for consistent org-wide governance. Our focus remains on delivering reliable, outstanding value to our customers. We are hardening our platform, strengthening comprehensive monitoring, and implementing industry best practices to prevent the recurrence of incidents. Thank you for your continued trust and patience.

Read the full incident report →

Started Sep 06, 2025, 12:48 AM UTC· resolved Sep 06, 2025, 07:19 AM UTC

1. investigating Sep 06, 2025, 12:48 AM UTC

   We are currently investigating this issue.

2. investigating Sep 06, 2025, 05:17 AM UTC

   We are working closely with AWS to resolve this issue.

3. resolved Sep 06, 2025, 07:19 AM UTC

   The incident was resolved.

4. postmortem Sep 08, 2025, 08:11 PM UTC

   ### Summary On Friday, September 5 at 23:33 UTC \(16:33 PDT\), one of our AWS accounts was accessed using a set of unauthorized keys. The intrusion was quickly detected, and the compromised keys were revoked within 30 minutes. During this window, the attackers provisioned a large number of expensive compute resources for cryptocurrency mining, which in turn triggered AWS to lock the account. Importantly, there was no unauthorized access to platform resources or customer data. Our investigation determined the root cause: a misconfiguration in our JavaScript build process inadvertently exposed CI/CD environment variables. This led to the leak of a pipeline key, which the attackers exploited. We immediately cleaned up all malicious resources and engaged with AWS in real time to fully restore account functionality. Full traffic was restored at September 6, 7:07 UTC \(12:07 am PDT\) ### Timeline **Sep 06 2025 00:13 UTC:** The outage started at as AWS locked down account‌ **Sep 06 2025 00:18 UTC:** AWS alerts HyperTrack engineering **Sep 06 2025 04:00 UTC:** * HyperTrack engineering completed actions to remove over 3000 EC2 instances and associated resources * HyperTrack engineering reiterates to AWS urgency of reopening full access to the account and turning traffic back on **Sep 06 2025 05:33 UTC:** * AWS confirms case was escalated to Service team to reinstate account. * HyperTrack explains urgency: HyperTrack SDK used in millions of apps, supporting nurses and essential workers **Sep 06 2025 07:07 UTC:** * AWS turned account back on **Sep 06 2025 10:30 UTC:** * HyperTrack engineers continued working with the production resources and handling scale as mobile devices were coming back online after the outage. HyperTrack SDK caches tracking data and thus core pipeline inbound traffic scaled up by a couple orders of magnitude **Sep 06 2025 17:15 UTC:** * HyperTrack engineers continued investigation to understand the source of the leak. The source of the leak was identified to be a misconfiguration in our JavaScript build process.‌ ### Next steps Our team is conducting an in-depth analysis of the incident and is enhancing our security policies and procedures. These updates are designed to reinforce safeguards and ensure the continued safety of customer data.

Read the full incident report →

Started May 23, 2025, 06:25 PM UTC· resolved May 23, 2025, 06:45 PM UTC

1. investigating May 23, 2025, 06:25 PM UTC

   We are currently investigating the issue and working to resolve it.

2. investigating May 23, 2025, 06:25 PM UTC

   We are continuing to investigate this issue. The issue emerged at 17:20 UTC.

3. resolved May 23, 2025, 07:11 PM UTC

   Issues have been resolved at 18:45 UTC. The team is gathering data for the postmortem and action steps to prevent future degradations.

4. postmortem May 27, 2025, 07:15 PM UTC

   # **Postmortem: System-Wide Outage Due to Database Degradation** **Incident Date:** May 23, 2025 **Time to Resolution:** 85 minutes **Status:** Resolved **Severity:** Critical \(P0\) ### **Summary** On May 23, 2025, our platform experienced a widespread outage due to degraded performance in our database infrastructure. Specifically, a set of read replicas were affected during the period. This degradation resulted in elevated error rates and unavailability across multiple APIs, including Orders, Workers, Places, and SDK-related services. The issue was fully resolved within 85 minutes. We understand how critical our services are to your operations and sincerely apologize for the disruption. ### **What Happened** A query pattern in our system targeting a key Orders API table failed to use a necessary index. This led to full table scans that overloaded some of our reader instances. As a result, several core APIs failed or experienced extreme latency. ### **Impact** * Customers experienced timeouts or errors when accessing Orders, Workers, and Places APIs * Monitoring and dashboard functionality was temporarily unavailable ### **What We Did** * Identified the problematic query * Deployed a hotfix to ensure proper index usage * Applied a secondary patch to reduce load when workers were not actively tracking * Restarted degraded infrastructure and monitored stabilization * Performed a full incident review across impacted components ### **Remediation and Next Steps** We are taking the following actions to ensure this does not happen again: * **Automated slow-query detection**: We’re enhancing our review pipeline with weekly audits and real-time alerting. * **Improved infrastructure alarms**: CPU and query performance alarms will provide earlier visibility into degradation. ### **Final Thoughts** We are committed to providing a stable and resilient platform. This incident has highlighted areas we must improve, and we’re taking swift action to reinforce our architecture. Thank you for your trust and patience.

Read the full incident report →

Started Apr 28, 2025, 03:30 AM UTC· resolved Apr 28, 2025, 03:30 AM UTC

1. resolved Apr 28, 2025, 04:00 AM UTC

   Due to underlying database conditions, we are experiencing degraded platform performance and elevated errors. Team is currently investigating and troubleshooting.

Read the full incident report →

Started Apr 09, 2025, 12:22 AM UTC· resolved Apr 09, 2025, 12:50 AM UTC

1. resolved Apr 10, 2025, 02:31 AM UTC

   We experienced degraded performance in one of our databases. This has caused elevated response times and timeouts in parts of our service from 12:22 am to 12:50 am UTC.

Read the full incident report →

Started Apr 01, 2025, 08:00 PM UTC· resolved Apr 02, 2025, 12:00 AM UTC

1. resolved Apr 02, 2025, 06:27 PM UTC

   Less than 5 % of ETA calculations were behind for up to one hour between April 1, 20:00 UTC and to April 2, 00:00 UTC due to a processing issue.

Read the full incident report →

Started Mar 28, 2025, 10:00 PM UTC· resolved Mar 28, 2025, 11:00 PM UTC

1. resolved Mar 28, 2025, 11:05 PM UTC

   Less than 5 % of ETA calculations were behind for up to one hour between 22:00 UTC and to 23:00 UTC due to an underlying infrastructure issue.

Read the full incident report →