FMX incident

Intermittent timeouts

Minor Resolved View vendor source →

FMX experienced a minor incident on August 4, 2022 affecting Web App and API and 1 more component, lasting 6d. The incident has been resolved; the full update timeline is below.

Started
Aug 04, 2022, 12:52 PM UTC
Resolved
Aug 10, 2022, 01:36 PM UTC
Duration
6d
Detected by Pingoru
Aug 04, 2022, 12:52 PM UTC

Affected components

Web AppAPIEmailReporting Dashboards

Update timeline

  1. investigating Aug 04, 2022, 12:52 PM UTC

    We are investigating an issue that is causing intermittent timeouts for our application. We are in the process of adding additional logging to determine the cause. We'll update the post as we have more information. We apologize for the inconvenience!

  2. investigating Aug 04, 2022, 09:02 PM UTC

    We are continuing to investigate this elusive issue and look for solutions. We implemented a fix for one possible cause around 4 pm EST today. We are monitoring whether that fix was effective and are continuing our investigation for any other possible causes. We'll update the post as we have more information. We apologize for the inconvenience!

  3. investigating Aug 05, 2022, 03:27 PM UTC

    We are continuing to investigate this issue. We are seeing a reduced number of timeouts since our last hotfix, but we need to adjust our fix to account for additional scenarios, and we're working on that now. Please continue to reach out to [email protected] with any issues you encounter. We'll continue to update this post as we have more information. We apologize for the inconvenience!

  4. investigating Aug 05, 2022, 03:58 PM UTC

    We are continuing to investigate this issue. We have released an adjustment to our fix to account for additional scenarios. We are monitoring whether our fix is effective and are continuing our investigation for any other possible causes. Please continue to reach out to [email protected] with any issues you encounter. We'll continue to update this post as we have more information. We apologize for the inconvenience!

  5. investigating Aug 05, 2022, 06:35 PM UTC

    We are continuing to investigate this issue. We are not currently seeing any timeouts related to the issue, but we are continuing to monitor whether our fix is fully effective and are continuing our investigation into any other possible causes. Please continue to reach out to [email protected] with any issues you encounter. We'll continue to update this post as we have more information.

  6. investigating Aug 06, 2022, 02:01 AM UTC

    We are continuing to investigate this issue. We are still not currently seeing any timeouts related to the issue, but we are continuing to monitor whether our fix is fully effective and are continuing our investigation into any other possible causes. Please continue to reach out to [email protected] with any issues you encounter. We'll continue to update this post as we have more information.

  7. resolved Aug 10, 2022, 01:36 PM UTC

    We've determined that the fixes that we introduced last week have solved the acute performance issue impacting our application. We are prioritizing additional application performance improvements to help alleviate any ongoing general performance problems. We apologize for the inconvenience!