Internal server errors from api.cloud.factbird.com
Timeline · 1 update
- resolved Sep 16, 2026, 12:15 PM UTC
After a deployment, our external API started returning 500 Internal server error on every request. A fix was deployed within 60 minutes
Factbird Outage History
Factbird is up right nowFactbird had 16 outages in the last 2 years totaling 54h 54m of downtime, averaging 0.7 incidents per month.
There were 16 Factbird outages since October 25, 2024 totaling 54h 54m of downtime. Each is summarised below. Incident details, duration, and resolution information.
System down
Affected: User InterfaceAPI ServiceData AnalysisAuthentication Service
Timeline · 4 updates
- investigating Jun 23, 2026, 01:09 PM UTC
We are currently receiving reports of users being unable to access the system. We are investigating ASAP
- identified Jun 23, 2026, 01:17 PM UTC
We have identified the issue and rolled out a fix. We expect services to be up again shortly
- monitoring Jun 23, 2026, 01:39 PM UTC
System should be up again now. We are monitoring
- resolved Jun 23, 2026, 04:43 PM UTC
This incident has been resolved.
Wrong OEE and status on Line cards in Factory Overview
Affected: User InterfaceData Analysis
Timeline · 3 updates
- identified May 28, 2026, 11:52 AM UTC
We are currently seeing issues with OEE calculation and status on Line cards in Factory Overview. Other parts of the system might also be affected. Directory cards and OEE in the OEE tab on lines are not affected. We have identified the root cause and are working on a fix
- monitoring May 28, 2026, 04:58 PM UTC
A fix has been deployed. We are monitoring the results
- resolved May 28, 2026, 04:59 PM UTC
This incident has been resolved.
Increased error rate on API requests
Timeline · 1 update
- resolved Apr 09, 2026, 12:54 PM UTC
We started seeing increase in number of errors on our external API after our latest deployment. This turned out to be because our API gateway was directing traffic to an old target. We updated the routing and that resolved the issue
Several pages reporting "lambda function error: Unhandled"
Affected: User InterfaceAPI ServiceData Analysis
Timeline · 4 updates
- investigating Mar 10, 2026, 12:02 PM UTC
We have received reports that many customers are seeing "lambda function error: Unhandled" errors when trying to access the Faccbird system. We are investigating
- monitoring Mar 10, 2026, 12:33 PM UTC
We identified an issue in connecting to one of our databases. We have made a quick update to circumvent it, and are monitoring error rates
- resolved Mar 10, 2026, 01:26 PM UTC
Services are back to normal
- postmortem Mar 13, 2026, 02:59 PM UTC
On March 10th, around 8AM CET, we started seeing sporadic connectivity failures towards a database that couldn’t be reached, resulting in some compute resources failing to start up and serve user requests. This particular database connection was not at all required and was made optional to address the failing API requests during this incident. The root cause was an optional, and under-provisioned database, that started seeing an unexpected load from an internal service which it failed to recover from on its own requiring manual intervention at the time of the incident.
Stuck at the sign-in page
Affected: Authentication Service
Timeline · 5 updates
- investigating Jan 28, 2026, 04:10 PM UTC
We are investigating an issue where some users are being sent back to the sign-in page repeatedly as they are marked unauthorized for accessing parts of the application.
- investigating Jan 28, 2026, 04:10 PM UTC
We are continuing to investigate this issue.
- monitoring Jan 28, 2026, 05:46 PM UTC
Our service provider, AWS, has reported latency issues in eu-west-1 affecting Factbird services. For details, please see https://health.aws.amazon.com/health/status
- resolved Jan 28, 2026, 09:46 PM UTC
The underlying issue in Dublin, Ireland at AWS has been resolved.
- postmortem Jan 29, 2026, 08:56 AM UTC
On January 28th, around 4PM CET, AWS has a degradation of their services in eu-west-1 \(Dublin, Ireland\), affecting Factbird services relying on infrastructure of Amazon Cognito, AWS DynamoDB, AWS ECS \(Elastic Container Service\). End-users would lose their access to the application and be directed back to the sign-in page repeatedly. By redirecting the traffic to an unaffected service, users would remain signed in while other API requests would fail to return data to the application. The incident lasted around 4 hours. AWS has yet to release their own postmortem of the incident. The investigation by AWS can be found at [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status)
Legacy Factbirds offline
Timeline · 1 update
- resolved Nov 11, 2025, 10:16 AM UTC
All Legacy Factbird were offline for around 45 minutes. Other devices were not affected
Administration tabs non-functioning
Affected: User Interface
Timeline · 2 updates
- identified Aug 18, 2025, 07:29 AM UTC
We are aware of a problem where the tabs on the administration page are non-functional. A solution has been identified, and we are working on resolving the issue.
- resolved Aug 18, 2025, 11:36 AM UTC
This incident has been resolved.
Reported 500 errors due to failing authorization
Timeline · 1 update
- resolved Mar 27, 2025, 04:38 PM UTC
We received reports on 500 errors being shown caused by failed attempts at authorizing the user session. Our authorization service has been reverted to its previous version to continue its operation.
Reduced scope of asset access
Affected: User InterfaceAPI Service
Timeline · 2 updates
- investigating Mar 13, 2025, 09:07 AM UTC
We've received reports of a sudden loss of access to assets for users who previously had access but can no longer reach them. We're investigating the reason behind the additional restrictions. Super users are not affected.
- resolved Mar 13, 2025, 10:50 AM UTC
The recent increase in access restrictions have been lifted, allowing affected users to view the assets they could access previously.
Degraded application experience resulting in internal server errors
Affected: User InterfaceAPI Service
Timeline · 4 updates
- investigating Mar 12, 2025, 01:20 PM UTC
We are actively investigating reported 504 errors due to service timeouts.
- identified Mar 12, 2025, 01:51 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 12, 2025, 04:07 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 12, 2025, 10:31 PM UTC
This incident has been resolved.
Degraded application experience resulting in 504 errors
Affected: User InterfaceAPI Service
Timeline · 1 update
- resolved Mar 12, 2025, 10:39 AM UTC
This morning (CET), our backend encountered service timeouts due to a specific service/application usage pattern, resulting in a ripple effect that impacted a large number of Factbird users. The issue stemmed from a change implemented on March 11th, intended to enhance overall performance. For now, we have disabled the change and are actively assessing solutions to mitigate the ripple effect it caused.
Sporadic application error code 413
Timeline · 2 updates
- resolved Feb 14, 2025, 10:51 AM UTC
Some users experience API error 413 when navigating Factbird. The issue is isolated to users that have access to a large quantity of sensors and lines.
- postmortem Feb 14, 2025, 10:51 AM UTC
The root cause of the incident was an internal authentication payload exceeding a limit, which caused the API layer to error. # Actions taken * Deploy a change to the authentication layer that reduces the authentication payload size
Missing data on legacy Factbird Digital and Factbird Analog devices
Affected: Data ConnectivityFactbird IoT Devices
Timeline · 2 updates
- monitoring Jan 26, 2025, 08:52 AM UTC
On Saturday January 25th at 06:00 PM UTC, all legacy Factbird Digital and Factbird Analog devices went offline. Other devices, such as Factbird Duo and Factbird View, were not affected. The cause of the issue was an outage in a 3rd party service that we rely on. Restarting the service resolved the issue at around 08:33 AM UTC, Sunday January 26th, and data is now coming in again. However, you'll see missing data in the period where the devices were offline. We are still monitoring the legacy devices to ensure everything is working again. We are also investigating if it is possible to restore the missing data from our internal logs.
- resolved Jan 26, 2025, 07:34 PM UTC
This incident has been resolved.
Infinite loop during sign-in
Affected: Authentication Service
Timeline · 4 updates
Data delay of Factbird Legacy devices caused by a scheduled queue cluster upgrade
Timeline · 1 update
- resolved Oct 25, 2024, 07:47 AM UTC
Our data forwarding service consuming Legacy Factbird Device data from our queue provider, CloudAMQP, failed to re-establish its connection after a scheduled cluster upgrade of CloudAMQP. The connection re-establishment required manual intervention and the queue has been fully consumed and we're now operational again. No data was lost in the process.