Formant Outage History

Formant is up right now

Formant had 18 outages in the last 2 years totaling 542h 14m of downtime, averaging 0.7 incidents per month.

There were 18 Formant outages since November 5, 2024 totaling 542h 14m of downtime. Each is summarised below. Incident details, duration, and resolution information.

Source: status.formant.io

Major May 11, 2026

Some queries against the Analytics SQL API return incomplete results

Detected by Pingoru
May 11, 2026, 09:32 PM UTC
Resolved
May 12, 2026, 04:37 PM UTC
Duration
19h 4m
Affected: Formant Web ApplicationAdmin APIQuery API
Timeline · 4 updates
  1. investigating May 11, 2026, 09:32 PM UTC

    We are currently investigating this issue.

  2. identified May 12, 2026, 01:27 AM UTC

    The issue has been identified. Working to identify the full fix.

  3. monitoring May 12, 2026, 02:05 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved May 12, 2026, 04:37 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor April 21, 2026

Degraded performance in V1 view modules like bitset

Detected by Pingoru
Apr 21, 2026, 03:12 PM UTC
Resolved
Apr 21, 2026, 06:26 PM UTC
Duration
3h 13m
Affected: Formant Web Application
Timeline · 2 updates
  1. investigating Apr 21, 2026, 03:12 PM UTC

    We are currently seeing an issue where data is not loading in some module types

  2. resolved Apr 21, 2026, 06:26 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor March 26, 2026

We're currently experiencing some slowness in the platform

Detected by Pingoru
Mar 26, 2026, 02:24 PM UTC
Resolved
Mar 26, 2026, 05:54 PM UTC
Duration
3h 29m
Affected: Formant Web ApplicationAdmin API
Timeline · 3 updates
  1. investigating Mar 26, 2026, 02:24 PM UTC

    We are currently investigating this issue.

  2. monitoring Mar 26, 2026, 03:05 PM UTC

    Performance is returning to normal levels

  3. resolved Mar 26, 2026, 05:54 PM UTC

    Incident has been resolved, but we'll continue to monitor the situation.

Read the full incident report →

Major March 25, 2026

Widespread issues reported by various customers

Detected by Pingoru
Mar 25, 2026, 01:17 PM UTC
Resolved
Mar 25, 2026, 02:39 PM UTC
Duration
1h 22m
Affected: Formant Web ApplicationIngestion PipelineAdmin APIQuery APIReal Time Signaling
Timeline · 4 updates
  1. investigating Mar 25, 2026, 01:17 PM UTC

    Various customers are reporting issues with the platform.

  2. investigating Mar 25, 2026, 01:21 PM UTC

    We are continuing to investigate this issue.

  3. investigating Mar 25, 2026, 01:33 PM UTC

    We are continuing to investigate this issue.

  4. resolved Mar 25, 2026, 02:39 PM UTC

    This incident has been resolved. Thank you for your patience.

Read the full incident report →

Minor March 17, 2026

Data Processing Delays - Alerting Worker falling behind - Reporting Tools Affected

Detected by Pingoru
Mar 17, 2026, 04:33 PM UTC
Resolved
Mar 17, 2026, 06:14 PM UTC
Duration
1h 40m
Affected: Formant Web ApplicationIngestion Pipeline
Timeline · 3 updates
  1. investigating Mar 17, 2026, 04:33 PM UTC

    Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost. We are currently investigating the issue.

  2. monitoring Mar 17, 2026, 05:54 PM UTC

    We've reverted to the previous version, upgraded the number of kubernetes pods, and are monitoring as the system is recovering

  3. resolved Mar 17, 2026, 06:14 PM UTC

    Everything is back to normal

Read the full incident report →

Minor February 5, 2026

Some users are getting logged out after 5 minutes of inactivity

Detected by Pingoru
Feb 05, 2026, 03:14 PM UTC
Resolved
Feb 05, 2026, 05:50 PM UTC
Duration
2h 35m
Affected: Formant Web Application
Timeline · 4 updates
  1. investigating Feb 05, 2026, 03:14 PM UTC

    We are currently investigating an issue where some users are getting logged out after 5 minutes of inactivity.

  2. identified Feb 05, 2026, 03:33 PM UTC

    The issue has been identified and a fix is incoming.

  3. monitoring Feb 05, 2026, 05:25 PM UTC

    A fix has been deployed. Testing and monitoring the fix.

  4. resolved Feb 05, 2026, 05:50 PM UTC

    The issue has been fixed and tested.

Read the full incident report →

Notice December 3, 2025

Service degradation – Backend API errors

Detected by Pingoru
Dec 03, 2025, 09:00 PM UTC
Resolved
Dec 03, 2025, 09:00 PM UTC
Duration
–
Timeline · 1 update
  1. resolved Dec 04, 2025, 04:32 PM UTC

    We experienced a period of severely degraded backend performance between approximately 2:45 pm and 3:16 pm US Central Time on December 3, 2025 due to a database query change in a new release. We promptly reverted the change and confirmed that services returned to normal. No data loss occurred.

Read the full incident report →

Critical October 20, 2025

AWS outage caused disruptions to Coherence, Teleop, and other Formant.io services

Detected by Pingoru
Oct 20, 2025, 07:00 AM UTC
Resolved
Oct 20, 2025, 07:00 AM UTC
Duration
–
Timeline · 1 update
  1. resolved Oct 21, 2025, 09:35 PM UTC

    We experienced disruptions to our Coherence, Telelop and other Formant.io services due to the widespread Amazon Web Services (AWS) outage that began around 2:11 AM Central Time on October 20th, 2025. Although most services had returned to normal by 1 PM Central Time, AWS did not provide the all clear until 5 PM Central Time. All Formant services have returned to normal operation.

Read the full incident report →

Minor May 8, 2025

Ingestion lag

Detected by Pingoru
May 08, 2025, 08:04 PM UTC
Resolved
May 08, 2025, 09:05 PM UTC
Duration
1h
Affected: Ingestion Pipeline
Timeline · 2 updates
  1. investigating May 08, 2025, 08:04 PM UTC

    We are currently investigating delayed data ingestion for some customers. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available. Thank you for your patience.

  2. resolved May 08, 2025, 09:34 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor March 26, 2025

Partial Outage Causing Ingestion Lag

Detected by Pingoru
Mar 26, 2025, 10:29 PM UTC
Resolved
Mar 26, 2025, 11:43 PM UTC
Duration
1h 14m
Affected: Ingestion Pipeline
Timeline · 4 updates
  1. investigating Mar 26, 2025, 10:29 PM UTC

    We are currently investigating a partial outage that is resulting in delayed data ingestion for some customers. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available. Thank you for your patience.

  2. identified Mar 26, 2025, 11:23 PM UTC

    The issue has been identified.

  3. monitoring Mar 26, 2025, 11:42 PM UTC

    A fix has been implemented and we're monitoring the performance of ingestion.

  4. resolved Mar 26, 2025, 11:43 PM UTC

    The ingestion lag issue has been resolved.

Read the full incident report →

Major February 27, 2025

Asset ingestion disruption

Detected by Pingoru
Feb 27, 2025, 10:30 PM UTC
Resolved
Feb 27, 2025, 10:30 PM UTC
Duration
–
Timeline · 1 update
  1. resolved Feb 28, 2025, 05:10 PM UTC

    Formant agent version 1.232.0 contained an issue which caused disruption to ingestion of asset data. This was fixed in version 1.232.6. To resolve this issue, update your Formant agent. For more information on asset data, see How telemetry streams work in our documentation center.

Read the full incident report →

Major February 27, 2025

'Failed to import rosgraph' when installing Formant agent

Detected by Pingoru
Feb 27, 2025, 08:52 PM UTC
Resolved
Feb 28, 2025, 10:30 AM UTC
Duration
13h 38m
Affected: Formant Agent
Timeline · 2 updates
  1. identified Feb 27, 2025, 08:52 PM UTC

    We've discovered an issue causing an import failure when installing the Formant agent on a ROS 1 machine in agent version 1.232. To resolve this issue, add `export FORMANT_ROS_VERSION=1` to `/var/lib/formant/.bashrc`.

  2. resolved Feb 28, 2025, 03:52 PM UTC

    This incident has been resolved. To fix this issue, update to the latest version of the agent.

Read the full incident report →

Minor January 16, 2025

Formant agent intermittently fails to connect to ROS

Detected by Pingoru
Jan 16, 2025, 10:39 PM UTC
Resolved
Feb 05, 2025, 07:38 PM UTC
Duration
19d 20h
Affected: Formant Agent
Timeline · 2 updates
  1. monitoring Feb 03, 2025, 10:39 PM UTC

    We've identified an issue that sometimes causes the Formant agent to fail to reconnect to the ROS Master when the ROS Master is restarted. We have identified a fix and are testing it. In the meantime, if you are affected by this issue, you can roll back to agent version 1.212.0 See Install the Formant agent for instructions: https://docs.formant.io/docs/getting-started-install-the-formant-agent#install-specific-agent-version

  2. resolved Feb 05, 2025, 07:38 PM UTC

    This incident has been resolved in agent version 1.226.

Read the full incident report →

Critical January 16, 2025

Formant disruption

Detected by Pingoru
Jan 16, 2025, 12:00 AM UTC
Resolved
Jan 16, 2025, 12:12 AM UTC
Duration
12m
Timeline · 1 update
  1. resolved Jan 16, 2025, 07:10 PM UTC

    The Formant application was unavailable for 12 minutes due to a backend deployment issue. There was no data loss during this time.

Read the full incident report →

Minor January 3, 2025

Data ingestion lag

Detected by Pingoru
Jan 03, 2025, 10:15 PM UTC
Resolved
Jan 03, 2025, 10:38 PM UTC
Duration
23m
Timeline · 1 update
  1. resolved Jan 06, 2025, 04:58 PM UTC

    On January 3, 2025, from 5:15 pm to 5:38pm EST, data ingestion to Formant experienced degraded service with high lag times. Data ingested during that period did not appear until after 5:38 pm EST. No data was lost during this incident.

Read the full incident report →

Minor December 3, 2024

Intermittent scheduled command issue

Detected by Pingoru
Dec 03, 2024, 05:00 AM UTC
Resolved
Dec 03, 2024, 11:00 AM UTC
Duration
6h
Timeline · 1 update
  1. resolved Dec 12, 2024, 04:08 PM UTC

    From December 3, 2024 to December 9, 2024, we saw intermittent issues with scheduled commands. Occasionally, scheduled commands failed to fire. This was fixed on December 9, 2024.

Read the full incident report →

Minor November 13, 2024

Some users experiencing degraded performance in Formant web app and Admin API

Detected by Pingoru
Nov 13, 2024, 06:55 PM UTC
Resolved
Nov 13, 2024, 08:02 PM UTC
Duration
1h 7m
Affected: Formant Web ApplicationAdmin API
Timeline · 3 updates
  1. investigating Nov 13, 2024, 07:01 PM UTC

    We are currently investigating this issue.

  2. monitoring Nov 13, 2024, 07:25 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Nov 13, 2024, 08:02 PM UTC

    This incident has been resolved.

Read the full incident report →

Major November 5, 2024

Some users cannot log in

Detected by Pingoru
Nov 05, 2024, 06:00 PM UTC
Resolved
Nov 06, 2024, 04:12 AM UTC
Duration
10h 12m
Affected: Formant Web Application
Timeline · 7 updates
  1. investigating Nov 05, 2024, 07:22 PM UTC

    We’re currently experiencing intermittent login issues. Our team is actively working to restore access as soon as possible. Thank you for your patience and understanding.

  2. investigating Nov 05, 2024, 08:22 PM UTC

    We are continuing to investigate this issue.

  3. investigating Nov 05, 2024, 08:28 PM UTC

    We are continuing to investigate this issue.

  4. identified Nov 05, 2024, 11:01 PM UTC

    The issue has been identified and a fix is being implemented. Most users are unaffected.

  5. identified Nov 05, 2024, 11:04 PM UTC

    We are continuing to work on a fix for this issue.

  6. resolved Nov 06, 2024, 04:12 AM UTC

    All affected users have been reinstated and can now reset their password to log in.

  7. postmortem Nov 07, 2024, 11:33 PM UTC

    We found an issue that prevented a limited number of users from logging in. Access for affected users has been restored. Affected users will have to reset their passwords. No user data was ever exposed as a result of this issue.

Read the full incident report →