HyperTrack experienced a major incident on September 6, 2025 affecting Cloud service and Orders and 1 more component, lasting 6h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 06, 2025, 12:48 AM UTC
We are currently investigating this issue.
- investigating Sep 06, 2025, 05:17 AM UTC
We are working closely with AWS to resolve this issue.
- resolved Sep 06, 2025, 07:19 AM UTC
The incident was resolved.
- postmortem Sep 08, 2025, 08:11 PM UTC
### Summary On Friday, September 5 at 23:33 UTC \(16:33 PDT\), one of our AWS accounts was accessed using a set of unauthorized keys. The intrusion was quickly detected, and the compromised keys were revoked within 30 minutes. During this window, the attackers provisioned a large number of expensive compute resources for cryptocurrency mining, which in turn triggered AWS to lock the account. Importantly, there was no unauthorized access to platform resources or customer data. Our investigation determined the root cause: a misconfiguration in our JavaScript build process inadvertently exposed CI/CD environment variables. This led to the leak of a pipeline key, which the attackers exploited. We immediately cleaned up all malicious resources and engaged with AWS in real time to fully restore account functionality. Full traffic was restored at September 6, 7:07 UTC \(12:07 am PDT\) ### Timeline **Sep 06 2025 00:13 UTC:** The outage started at as AWS locked down account **Sep 06 2025 00:18 UTC:** AWS alerts HyperTrack engineering **Sep 06 2025 04:00 UTC:** * HyperTrack engineering completed actions to remove over 3000 EC2 instances and associated resources * HyperTrack engineering reiterates to AWS urgency of reopening full access to the account and turning traffic back on **Sep 06 2025 05:33 UTC:** * AWS confirms case was escalated to Service team to reinstate account. * HyperTrack explains urgency: HyperTrack SDK used in millions of apps, supporting nurses and essential workers **Sep 06 2025 07:07 UTC:** * AWS turned account back on **Sep 06 2025 10:30 UTC:** * HyperTrack engineers continued working with the production resources and handling scale as mobile devices were coming back online after the outage. HyperTrack SDK caches tracking data and thus core pipeline inbound traffic scaled up by a couple orders of magnitude **Sep 06 2025 17:15 UTC:** * HyperTrack engineers continued investigation to understand the source of the leak. The source of the leak was identified to be a misconfiguration in our JavaScript build process. ### Next steps Our team is conducting an in-depth analysis of the incident and is enhancing our security policies and procedures. These updates are designed to reinforce safeguards and ensure the continued safety of customer data.