Memfault incident
Data Processing Delays and Errors - Ingress Affected
Memfault experienced a major incident on February 29, 2024 affecting API & Dashboard and Ingress, lasting 10h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 29, 2024, 05:46 PM UTC
Our data processing infrastructure is running behind. No data has been lost and the system should be caught up shortly.
- monitoring Feb 29, 2024, 08:37 PM UTC
A fix has been implemented and we're monitoring the results. We're churning through the backlog of data we've queued up, and expect to be caught up processing in the next few hours. No data has been lost and everything will eventually be processed.
- monitoring Feb 29, 2024, 09:42 PM UTC
We had an issue processing data today. This issue would have led to some device requests receiving error responses between 18:15-19:00 UTC, as well as some errors processing data during the same period. You may see processing errors in the UI for that time period. In addition, we queued a large amount of incoming data for later processing. We're currently churning through the backlog of queued data. We expect to be caught up in ~2 hours.
- monitoring Mar 01, 2024, 04:22 AM UTC
This issue is fully resolved. We are caught up on data processing.
- resolved Mar 01, 2024, 04:23 AM UTC
This incident has been resolved.