Mezmo Outage History

Started Oct 20, 2025, 05:00 PM UTC· resolved Oct 21, 2025, 02:08 AM UTC

1. identified Oct 20, 2025, 05:00 PM UTC

   The latency and error rate issues that AWS is currently experiencing is now affecting the performance of the Mezmo platform. We will do everything we can to mitigate the impact and recover as quickly as possible once AWS has resolved their issues. See here for more detail on what is happening with AWS: https://health.aws.amazon.com/health/status

2. monitoring Oct 20, 2025, 06:49 PM UTC

   AWS is reporting that services on their end are recovering, and we are seeing evidence to support that on our end.

3. monitoring Oct 20, 2025, 06:49 PM UTC

   We are continuing to monitor for any further issues.

4. monitoring Oct 20, 2025, 10:46 PM UTC

   We are continuing to monitor our platform for any further issues during AWS' recovery.

5. resolved Oct 21, 2025, 02:08 AM UTC

   Mezmo is no longer seeing impacts from the AWS events of today. We will continue to monitor our services and respond if we notice any abnormalities.

Read the full incident report →

Started Sep 15, 2025, 08:18 PM UTC· resolved Sep 15, 2025, 09:16 PM UTC

1. identified Sep 15, 2025, 08:18 PM UTC

   We have identified that there are some performance issues related to search in Mezmo Log Analysis. We are currently working to resolve the problem. Your data is still be ingested, indexed, and retained but are slow to display in the live tail because of the issues with Search. Please subscribe to the page to stay updated.

2. monitoring Sep 15, 2025, 08:51 PM UTC

   Search performance should be returning to normal. You may notice some degraded performance.

3. resolved Sep 15, 2025, 09:16 PM UTC

   The issues with Search in Live Tail have been resolved, and Search performance is back to normal.

Read the full incident report →

Started Sep 09, 2025, 08:49 PM UTC· resolved Sep 09, 2025, 11:52 PM UTC

1. identified Sep 09, 2025, 08:49 PM UTC

   Apologies Mezmo customers as we are again experiencing problems with our Search Engine and live tail. We have identified the problem and are actively working to fix it now.

2. identified Sep 09, 2025, 09:04 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Sep 09, 2025, 11:51 PM UTC

   At this time services are restored. Mezmo continues to perform diagnosis to identify and address the root causes or triggers for degradations in service, with engineering staying on high alert for any recurrences. We sincerely apologize for the impacts this incident may have had to your services.

4. resolved Sep 09, 2025, 11:52 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 08, 2025, 07:55 PM UTC· resolved Sep 08, 2025, 09:53 PM UTC

1. investigating Sep 08, 2025, 07:55 PM UTC

   Mezmo has identified an issue with our search engine which is preventing events from being displayed in live tail. We are actively working on the problem and expect to have it resolved shortly. Please subscribe to this page for updates on this issue.

2. monitoring Sep 08, 2025, 09:24 PM UTC

   Impacts from this issue are currently mitigated. Mezmo engineering continues to investigate the underlying causes of this issue and will leave this ticket in a monitoring state for now.

3. resolved Sep 08, 2025, 09:53 PM UTC

   Mezmo has made a backend change that we believe should mitigate this issue in the future. Mezmo will continue to proactively monitor the platform for any signs of recurrence. Please reach out through our normal support channels if you are still experiencing issues at this time or need further information about this incident. We apologize for the impacts this incident may have had to your services.

Read the full incident report →

Started Jun 09, 2025, 10:02 AM UTC· resolved Jun 09, 2025, 10:44 AM UTC

1. investigating Jun 09, 2025, 10:02 AM UTC

   We are experiencing intermittent issues with the Web App

2. identified Jun 09, 2025, 10:04 AM UTC

   The problem has been identified and our infrastructure team is working on the resolution

3. monitoring Jun 09, 2025, 10:35 AM UTC

   A fix has been implemented. The backlog is being processed.

4. resolved Jun 09, 2025, 10:44 AM UTC

   The issue has been resolved. The processing of the backlog is progressing, and all delayed logs will be available shortly

Read the full incident report →

Started Apr 14, 2025, 08:12 PM UTC· resolved Apr 14, 2025, 10:29 PM UTC

1. investigating Apr 14, 2025, 08:12 PM UTC

   The cause is known and is being actively mitigated.

2. resolved Apr 14, 2025, 10:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 11, 2025, 07:17 PM UTC· resolved Apr 11, 2025, 09:03 PM UTC

1. investigating Apr 11, 2025, 07:17 PM UTC

   We are currently investigating this issue.

2. resolved Apr 11, 2025, 09:03 PM UTC

   This incident has been resolved. All services are fully operational.

Read the full incident report →

Started Apr 02, 2025, 08:08 PM UTC· resolved Apr 02, 2025, 10:17 PM UTC

1. investigating Apr 02, 2025, 08:08 PM UTC

   We are currently experiencing intermittent issues with our Pipeline service and are actively investigating the matter. Some customers may experience delays in the Pipeline data.

2. resolved Apr 02, 2025, 10:17 PM UTC

   The Pipeline service has resumed. All services are fully operational.

Read the full incident report →

Started Mar 18, 2025, 01:33 PM UTC· resolved Mar 18, 2025, 02:29 PM UTC

1. investigating Mar 18, 2025, 01:33 PM UTC

   We are currently investigating the issue.

2. identified Mar 18, 2025, 01:45 PM UTC

   The issue has been identified and the fix implemented.

3. monitoring Mar 18, 2025, 01:50 PM UTC

   We are monitoring the performance. There is some lag and once it's been fully processed all the recent logs will be available.

4. resolved Mar 18, 2025, 02:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 12, 2024, 09:51 PM UTC· resolved Dec 12, 2024, 10:15 PM UTC

1. investigating Dec 12, 2024, 09:51 PM UTC

   New logs being submitted are not available for some accounts. We are investigating.

2. monitoring Dec 12, 2024, 10:09 PM UTC

   A fix has been implemented and new logs are now available for the accounts. We are monitoring the results.

3. resolved Dec 12, 2024, 10:15 PM UTC

   New logs are available for all accounts. This incident has been resolved.

4. postmortem Dec 18, 2024, 10:24 PM UTC

   **Dates:** Start Time: Thursday, December 12, 2024 at 19:24 UTC End Time: Thursday, December 12, 2024 at 20:54 UTC Duration: 1 hour and 30 minutes ‌ **What happened:** Log lines submitted to Mezmo for ingestion into Log Analysis were never made available in our WebUI for Searching, Graphing, and Timelines – neither during the incident, nor afterwards. This affected a significant number of accounts. Log lines were still passed through Pipeline and made available at all times within Live Tail. The log data still triggered both Telemetry Pipeline and Log Analysis based alerts, and were still archived in both places. ‌ **Why it happened:** A single pod within our indexing service ran out of disk space. This was due to a sudden increase in the volume of log lines sent to Mezmo by accounts that happened to be assigned to this pod. Our pods are configured to limit how much disk space is available for writing new data; these limits should have prevented any pod from running out of disk space, in any scenario. After the incident, we discovered that the limits had been configured incorrectly, which explains why it was possible for this pod to run out of disk space. Our service is designed to tolerate the loss of a single pod within its indexing service without any widespread impact to customers. Instead, for reasons still under investigation, the impact expanded to many other pods. Our service is also designed to retain submitted log lines, even if the indexing portion of our service cannot process them immediately; these log lines can be indexed later, when the service is functional again. In this incident, however, the log lines were never indexed. The reason for this failure is also under investigation. ‌ **How we fixed it:** We restarted the pod that had run out of disk space. It immediately had enough free disk space to accept new log lines for indexing. All other indexing pods also returned to a normal operational state. ‌ **What we are doing to prevent it from happening again:** We have properly configured our pods to prevent them from running out of disk space. This step alone should prevent any recurrence of the same problem. We have updated our monitoring to send high priority alerts when any indexing pod is in danger of running out of disk space. These alerts were in place before, but set to “low” priority; they did not come to our attention in time to prevent the incident. We will continue to actively investigate why the impact spread to other indexing pods and why log lines were not retained for indexing in the future.

Read the full incident report →

Started Oct 26, 2024, 01:00 AM UTC· resolved Oct 26, 2024, 08:23 PM UTC

1. investigating Oct 26, 2024, 01:00 AM UTC

   Our Pipeline WebUI is not loading pages. We are investigating.

2. investigating Oct 26, 2024, 02:12 AM UTC

   The Pipeline WebUI is still unavailable. Ingress and egress are unaffected. Our engineers are investigating.

3. monitoring Oct 26, 2024, 03:35 AM UTC

   The Pipeline WebUI is working now. We are monitoring.

4. monitoring Oct 26, 2024, 03:35 AM UTC

   We are continuing to monitor for any further issues.

5. monitoring Oct 26, 2024, 05:24 AM UTC

   The Pipeline WebUI is available, but at times pages are slow to load and metrics may be unavailable. We are taking remedial action and continuing to monitor.

6. monitoring Oct 26, 2024, 06:34 PM UTC

   The Pipeline WebUI is available and loading pages normally. We're working to resolve the root cause permanently and continuing to monitor.

7. resolved Oct 26, 2024, 08:23 PM UTC

   The Pipeline UI is now fully functional.

8. postmortem Oct 31, 2024, 08:31 PM UTC

   **Dates:** Start Time: Saturday, October 26, 2024 at 01:00 UTC End Time: Saturday, October 26, 2024 at 20:23 UTC Duration: 19 hours and 23 minutes \(Note that customer impact was limited to the first four hours of the incident.\) ‌ **What happened:** For approximately two hours, pages on the Web UI for Pipeline did not load. Afterwards, pages were able to load, but often only after delays of 15 seconds or longer. After another two hours, the Web UI returned to normal usage. The incident was kept open until all remediation was completed. The Web UI for Log Analysis and the ingress and egress of Pipeline data were unaffected. ‌ **Why it happened:** On Wednesday, October 23, 2024, we deployed a change to our Pipeline service, by which metrics on pipeline usage began to be saved to a postgres database. The same database also stores account configuration information; this information must be accessed to display pages in the Web UI. The deployment on Wednesday changed the performance profile of the database, most notably in the number and frequency of writes. There was no immediate customer impact, but we noted that backups of the database were unable to run successfully because of the increased load. Customer impact only began on Saturday, October 26th when an unrelated user action – running a Profiler from the Web UI – placed even more demands on the postgres database. It was unable to process queries and, according to its design, moved into “read-only” mode to prevent any loss of data. At times the database was entirely inaccessible or very slow to respond. This made our Web UI unable to display pages, as it relies on configuration information stored in the database. ‌ **How we fixed it:** We first enabled two replicas of the database, which allowed the Web UI to load pages again, albeit slowly. We then deployed a new code change that removed all superfluous writes about metric usage to the postgres database. This reduced the number of queries and Web UI usage returned to normal. We kept the incident open while working on further remediation. Finally we took steps to bring the postgres database back to a normal state. This remediation phase was complicated by the fact that a full backup had not completed successfully in the last two days. By the end of the incident, the database was operating normally and a full backup had completed. ‌ **What we are doing to prevent it from happening again:** The problems with the deployment on Wednesday, October 23, 2024 only revealed themselves under the load of our production environment. To better simulate production, we will update our testing environment and processes to use higher data loads. We will closely evaluate the current workload of the postgres database and consider making a separate database to store just metrics, rather than combining metrics and configurations in one location. Separate databases would have prevented Web UI pages from not loading, thus avoiding any customer impact. We will re-evaluate our backup strategy for the postgres database, since this slowed down the remediation phase.

Read the full incident report →