Onfleet experienced a minor incident on February 9, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Feb 09, 2023, 03:49 AM UTC
We are currently investigating this issue.
- postmortem Feb 09, 2023, 03:50 AM UTC
At 2023-01-27 9:45 UTC, a failover event occurred due to a transient networking issue for a database cluster related to location storage. This event lasted for around a minute. As a result, several processes related to persisting location data entered a state which prevented them from persisting locations. Due to a lack of effective monitoring, we were not immediately aware of this issue. The root issue was resolved at 2023-01-30 22:11 UTC, soon after we became aware of it. However, as a result of this issue, complete location information for about 20% of tasks during this time period was not collected. We subsequently backfilled derived location information when appropriate, providing distance information for about 75% of these tasks by 2023-01-31 19:07 UTC. We understand how crucial location information is and apologize for any impact this incident may have caused. We have since improved our monitoring so a similar incident would have immediately paged an on-call engineer.