FMX experienced a minor incident on October 24, 2024 affecting Web App and API and 1 more component, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 23, 2024, 01:57 PM UTC
On 10/22/2024 at approximately 10:30 PM EDT FMX experienced a server failure that required manual intervention to restore it. On 10/23/2024 at 4:10 AM EDT we had successfully restored the affected databases and are continuing to investigate the extent of data that was not restored between 10 and 11 pm EDT
- resolved Oct 24, 2024, 01:11 PM UTC
Through our investigation we have identified a subset of customers that were potentially affected by this issue. If you are one of the customers that we have identified, our Customer Success team will be reaching out to you via phone or email. What we know at this time is that we experienced multiple hardware failures that required us to roll back a subset of our databases to an earlier restore point. Any data that was added, edited, or deleted between 10:00pm EDT, and 11:00pm EDT on 10/22/2024 for these identified customers has been corrupted and our Customer Success team will be reaching out to you to answer any further questions. We are still in the process of investigating alternative remediation steps and will be conducting an internal postmortem to better understand and prioritize future infrastructure changes and processes to remove the possibility of this ever happening again. We know that any amount of data corruption can be a problem, and we apologize for any inconvenience this issue has caused.