Flatfile Outage History

Started Jul 04, 2025, 01:18 AM UTC· resolved Jul 04, 2025, 01:52 AM UTC

1. investigating Jul 04, 2025, 01:18 AM UTC

   We are currently investigating an issue where our AU regional server is not loading for some customers. We are working to get a fix out as quickly as possible.

2. identified Jul 04, 2025, 01:39 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 04, 2025, 01:44 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 04, 2025, 01:52 AM UTC

   This incident has been resolved.

5. postmortem Jul 07, 2025, 03:23 PM UTC

   # Incident Overview **Date and Time of Incident:** July 3rd starting at about 7:15pm MT to about 7:45pm MT **Nature of Incident:** Ephemeral Database CPU Pinning on Australia Regional Environment **Services Affected:** All Workbooks and Sheets endpoints ## Details of the Incident At approximately 7:15pm MT, we received a report from a customer on the Australia region that a database timeout error was being returned by the API. Upon investigation it was determined that the ephemeral database was experiencing degraded performance due to CPU load. We also investigated the API service in the Australia regional ECS cluster and rolled over the API deployment as a precautionary measure. ## Impact Assessment All Australia Regional platform users were unable to load workbooks or sheets during the incident. This resulted in degraded service for most customer workflows during the incident, including completely blocking some workflows due to API response failures. The incident was fully resolved about 30 minutes after initial report. ## Root Cause The root cause of this was CPU load on the ephemeral database instance. Upon investigation we discovered that the instance was a small size unable to handle the amount of load on it. We suspect also that background load due to postgres auto-vacuum might have been occurring at this time, exacerbating the problem. ## Resolution The ephemeral datastore instance was scaled up to a much larger instance size. This was sufficient to relieve the load and accommodate a much larger load from application users. We have confirmed that this resolved the problem. Going forward we will audit our regional deployments and determine if ephemeral datastores are at an appropriate scale to the volume of traffic on the region. ## Security and Data Integrity There was no loss of customer data and no security breach. We have reviewed application and database logs related to the incident and concluded that the database capacity problem was the root cause and that no other parts of the system experienced degradation.

Read the full incident report →

Started Jun 30, 2025, 02:09 PM UTC· resolved Jun 30, 2025, 02:58 PM UTC

1. investigating Jun 30, 2025, 02:09 PM UTC

   We are currently investigating reports of users unable to fetch historical events and load listeners

2. identified Jun 30, 2025, 02:11 PM UTC

   Our events provider is experiencing some instability, causing the issues to the spaces ui being unable to get historical events. We are monitoring and will update once this is back up again.

3. monitoring Jun 30, 2025, 02:23 PM UTC

   Our events provider has rolled out a fix, we are currenntly actively monitoring the platform.

4. monitoring Jun 30, 2025, 02:27 PM UTC

   We are continuing to monitor for any further issues.

5. monitoring Jun 30, 2025, 02:37 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Jun 30, 2025, 02:58 PM UTC

   The incident has been resolved. At approximately 13:11 UTC, our events provider service began experiencing elevated latencies and server errors in the Europe and North US region.

Read the full incident report →

Started Jun 11, 2025, 10:27 PM UTC· resolved Jun 11, 2025, 11:58 PM UTC

1. identified Jun 11, 2025, 10:27 PM UTC

   We are seeing an issue where excel files are not being extracted. We have identified a cause and are working on a fix. CSV files are still working.

2. monitoring Jun 11, 2025, 10:43 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Jun 11, 2025, 10:43 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Jun 11, 2025, 11:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 11, 2025, 05:00 PM UTC· resolved Jun 11, 2025, 10:17 PM UTC

1. investigating Jun 11, 2025, 05:00 PM UTC

   We are currently investigating this issue.

2. identified Jun 11, 2025, 05:24 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Jun 11, 2025, 05:42 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Jun 11, 2025, 09:20 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Jun 11, 2025, 10:17 PM UTC

   This incident has been resolved.

6. postmortem Jun 12, 2025, 02:36 PM UTC

   # Incident Overview A series of compounding events on the US region Platform led to network throttling of a network file system which severely degraded file extractions on the Platform. ## Details of the Incident At approximately 10 AM PDT on June 11, Flatfile experienced several conflating events that resulted in network bandwidth on our network file systems being overwhelmed and throttled by our cloud service provider. The event resulted from a spike in usage on the Platform that led to autoscaling events which dramatically increased read operations against the network file stores, eventually leading to exhaustion of throughput. This event led to a severe degradation in file extraction performance on the Platform for all customers. The initial response to the incident involved increasing capacity which compounded the problem. At approximately 14:20 PDT, the engineering team provisioned a large increase in throughput which ended the throttling and allowed file extraction jobs to proceed as expected. ## Impact Assessment The incident severely degraded all file extractions and led to increased latency on jobs as the queue became backlogged. ## Root Cause The throttling of throughput on the network file system was the cause of the incident; this was exacerbated by automated scaling operations that attempted to increase queue capacity. ## Resolution Flatfile engineering provisioned increased throughput on the file system to resolve the incident at 14:20 PDT. ## Security and Data Integrity Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Read the full incident report →

Started Jun 10, 2025, 08:52 PM UTC· resolved Jun 10, 2025, 09:01 PM UTC

1. investigating Jun 10, 2025, 08:52 PM UTC

   We are investigating an issue where some users are seeing bad gateway errors.

2. monitoring Jun 10, 2025, 08:57 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 10, 2025, 09:01 PM UTC

   This incident has been resolved.

4. postmortem Jun 10, 2025, 09:28 PM UTC

   # Incident Overview **Date and Time of Incident:** June 10, 2025 from approximately 1:50pm to 2:05pm PDT **Nature of Incident:** Requests routing to an unavailable frontend pod **Services Affected:** Flatfile Platform Dashboard and Spaces ## Details of the Incident At approximately 1:50pm PDT on June 10, the Flatfile Platform Dashboard reported intermittent 503 status codes. This was due to an infrastructure change made earlier in the day. As a consequence 1 in every 4 requests made to the Platform dashboard returned a HTTP 503 status code. The engineering team were alerted to the incident as soon as the 503’s started appearing and manually scaled up the infrastructure resources. ## Impact Assessment The incident was intermittent and affected 1 in 4 requests made by users of the Platform frontend applications for approximately 15 minutes. The API was unaffected. ## Root Cause The root cause was due to an infrastructure scaling configuration made in the early of the morning. ## Resolution Flatfile engineering manually provisioned extra capacity in order for the resources to get allocated appropriately. Following this, further improvements will be made with how we serve traffic to end users. ## Security and Data Integrity Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Read the full incident report →

Started Jun 04, 2025, 11:30 AM UTC· resolved Jun 04, 2025, 05:46 PM UTC

1. investigating Jun 04, 2025, 11:30 AM UTC

   We are currently investigating reports of failures of file uploads in our UK platform region

2. identified Jun 04, 2025, 11:35 AM UTC

   The issue has been identified and a fix has been rolled out

3. monitoring Jun 04, 2025, 11:39 AM UTC

   The fix has been fully rolled out, we are currently monitoring the platform.

4. resolved Jun 04, 2025, 05:46 PM UTC

   This incident has been resolved.

5. postmortem Jun 13, 2025, 01:30 PM UTC

   # Incident Overview **Date and Time of Incident:** June 4, 2025 from approximately 4:57 AM to 5:31 AM PDT **Nature of Incident:** Requests routing to an unavailable worker pod **Services Affected:** Flatfile Platform UK Regional Environment ## Details of the Incident At approximately 4:57 AM PDT \(11:57 AM BST\) on June 4th, Flatfile UK Regional environment infrastructure experienced intermittent failures on file uploads due to the backend worker fleet restarting. This was due to a misconfiguration on the backend worker infrastructure which created a hard dependency on an internal observability component used by FlatFile for monitoring. The internal observability component had experienced some downtime which led to backend worker fleet restarts. The infrastructure team were alerted to the incident and issued a patch to remove the hard dependency on the worker fleet. ## Impact Assessment The incident impacted only half of the total worker fleet due as the infrastructure safe guards kicked and rolled out extra workers in order to meet minimum capacity threshold of 50%. The UK regional environment experienced disruption for a total of 33 minutes. The API was unaffected. ## Root Cause The root cause was due to a misconfiguration on the infrastructure worker observability configuration made in a previous release 3 weeks ago. ## Resolution Flatfile infrastructure team fixed the miconfiguration on the worker infrastructure components by removing the hard depedency on Flatfile’s internal observability stack. ## Security and Data Integrity Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Read the full incident report →

Started Jun 03, 2025, 10:36 PM UTC· resolved Jun 03, 2025, 10:53 PM UTC

1. investigating Jun 03, 2025, 10:36 PM UTC

   We are currently investigating an issue causing users to not be able to access platform.flatfile.com.

2. monitoring Jun 03, 2025, 10:40 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 03, 2025, 10:53 PM UTC

   This incident has been resolved.

4. postmortem Jun 03, 2025, 11:17 PM UTC

   # Incident Overview **Nature of Incident:** Incorrect Application Frontend Container Deployed to Platform Frontend **Services Affected:** Flatfile Platform Dashboard and Spaces ## Details of the Incident At approximately 4:26pm MDT on June 3, a tag collision caused an image for an unreleased product to be deployed in place of the Platform frontend container. This led to frontend assets for an incorrect frontend application to be served in place of the Flatfile dashboard. 8 minutes later the engineering team was alerted to the incident and manually deployed the correct image to resolve the issue. ## Impact Assessment The incident affected all users of the Platform frontend applications for approximately 20 minutes. The API was unaffected. ## Root Cause The root cause was a collision in tagging on images within the Flatfile image registry. ## Resolution Flatfile engineering manually deployed a known good image to the frontend service to immediately restore service. Following this, the image registry was patched to prevent future incidents. ## Security and Data Integrity Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Read the full incident report →

Started May 30, 2025, 06:45 PM UTC· resolved May 30, 2025, 07:46 PM UTC

1. investigating May 30, 2025, 06:45 PM UTC

   We are currently investigating this issue.

2. identified May 30, 2025, 06:47 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 30, 2025, 07:18 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring May 30, 2025, 07:20 PM UTC

   We are continuing to monitor for any further issues.

5. identified May 30, 2025, 07:46 PM UTC

   We are continuing to work on a fix for this issue.

6. resolved May 30, 2025, 07:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 28, 2025, 02:45 PM UTC· resolved May 28, 2025, 03:45 PM UTC

1. investigating May 28, 2025, 02:45 PM UTC

   We are currently investigating this issue.

2. identified May 28, 2025, 02:45 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 28, 2025, 02:45 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 28, 2025, 03:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 23, 2025, 10:23 PM UTC· resolved May 24, 2025, 12:35 AM UTC

1. investigating May 23, 2025, 10:23 PM UTC

   We are currently investigating this issue.

2. monitoring May 23, 2025, 10:33 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 24, 2025, 12:35 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 23, 2025, 09:30 PM UTC· resolved May 23, 2025, 09:37 PM UTC

1. investigating May 23, 2025, 09:30 PM UTC

   We are currently investigating an issue that is causing some file uploads to fail.

2. identified May 23, 2025, 09:35 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 23, 2025, 09:37 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 23, 2025, 05:05 PM UTC· resolved May 23, 2025, 05:27 PM UTC

1. investigating May 23, 2025, 05:05 PM UTC

   Our identify provider is experiencing some instability, causing the login to the dashboard to be inaccessible at the moment. We are monitoring and will update once this is back up again.

2. identified May 23, 2025, 05:07 PM UTC

   We're watching our service provider for updates.

3. identified May 23, 2025, 05:10 PM UTC

   We are continuing to work on a fix for this issue.

4. identified May 23, 2025, 05:11 PM UTC

   We're watching our service provider for updates.

5. resolved May 23, 2025, 05:27 PM UTC

   Logins to the US, EU, and UK environments are operational again! All dashboards should now be accessible.

Read the full incident report →

Started May 20, 2025, 05:10 AM UTC· resolved May 20, 2025, 05:10 AM UTC

1. resolved May 20, 2025, 05:10 AM UTC

   A frontend application configuration variable change was deployed prior to deploying the code change. The configuration has been patched and the code change deployed.

Read the full incident report →

Started May 07, 2025, 02:12 PM UTC· resolved May 07, 2025, 03:14 PM UTC

1. investigating May 07, 2025, 02:12 PM UTC

   We are currently investigating customer reports of 504 errors on the platform

2. identified May 07, 2025, 02:15 PM UTC

   We have identified the issue and are rolling out a fix

3. monitoring May 07, 2025, 02:29 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 07, 2025, 03:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 01, 2025, 03:41 PM UTC· resolved May 01, 2025, 06:52 PM UTC

1. investigating May 01, 2025, 03:41 PM UTC

   We are seeing errors creating spaces and are investigating.

2. identified May 01, 2025, 03:43 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified May 01, 2025, 03:59 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring May 01, 2025, 04:08 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 01, 2025, 06:52 PM UTC

   Earlier today we deployed a feature that relied on a new environment variable. Despite updating the relevant files with the correct variable, one failed to deploy properly resulting in the variable not being found after deployment, and causing the errors that led to spaces not loading. This has been rolled back.

Read the full incident report →

Started Apr 29, 2025, 09:11 AM UTC· resolved Apr 29, 2025, 09:22 AM UTC

1. identified Apr 29, 2025, 09:11 AM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Apr 29, 2025, 09:21 AM UTC

   An update to our networking has been deployed and we are monitoring the solution

3. resolved Apr 29, 2025, 09:22 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 24, 2025, 05:00 PM UTC· resolved Apr 24, 2025, 05:00 PM UTC

1. resolved Apr 24, 2025, 07:08 PM UTC

   A recent update introduced a change to how sheet counts are processed. As part of this update, we began including a timestamp (countsComputedAt) to indicate when the counts were calculated. Shortly after being deployed to production, several endpoints (such as get workbook, update records, patch job) started returning 400 errors with the message recordCount.updated_at.getTime is not a function.

2. postmortem Apr 24, 2025, 07:09 PM UTC

   # **Introduction** A recent update introduced a change to how sheet counts are processed. As part of this update, we began including a timestamp \(countsComputedAt\) to indicate when the counts were calculated. Shortly after being deployed to production, several endpoints \(such as get workbook, update records, patch job\) started returning 400 errors with the message recordCount.updated\_at.getTime is not a function. # **Incident Details** **Date Reported**: April 24, 2025 **What Went Wrong:** Data from workbooks is cached for performance. When this cached data is retrieved, it goes through a process that converts date objects into plain text \(specifically, ISO8601-formatted strings\). This conversion removes the special capabilities of a date object—such as calculating the time—causing our system to break when it tried to use those capabilities. This error did not appear in our testing or development environments because caching is disabled there, so the date information remained intact and behaved as expected. **Why It Wasn’t Caught Earlier:** **Development and test environments differ from production**: Since caching is turned off during development and testing, we didn’t see the issue where the date value was converted into a plain string. **Incomplete type checking**: The part of our system responsible for managing cached data didn’t reflect that the cached date values had changed from their original form, so the issue wasn’t flagged by our automated checks. **Next Steps:** We’re updating our development environment to more closely mirror production so that we can catch this type of issue earlier in the future. Additionally, we’re improving our type handling and tests around cached data to ensure these transformations are properly accounted for going forward. **Resolution & Fix** 1. Production was rolled back while the issue was investigated. 2. The workbook presenter logic was updated to handle the different types resulting from caching behavior. 3. The local and test environments were updated to perform type conversion consistently with production, even when caching is disabled.

Read the full incident report →

Started Apr 17, 2025, 11:14 PM UTC· resolved Apr 17, 2025, 11:23 PM UTC

1. investigating Apr 17, 2025, 11:14 PM UTC

   We are investigating an issue where some users are unable to load spaces from the dashboard.

2. identified Apr 17, 2025, 11:15 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Apr 17, 2025, 11:23 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 02, 2025, 05:53 PM UTC· resolved Apr 02, 2025, 07:08 PM UTC

1. investigating Apr 02, 2025, 05:53 PM UTC

   We are investigating an issue where some users are seeing intermittent 503 errors

2. monitoring Apr 02, 2025, 06:07 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 02, 2025, 07:08 PM UTC

   This incident has been resolved.

4. postmortem Apr 24, 2025, 07:11 PM UTC

   # **Introduction** On Apr 2, 2025 a service degradation caused intermittent requests for static assets to fail; these included requests for HTML, JS, CSS and other assets resulting in failed delivery of frontend applications for several short bursts of time. # **Incident Details** * **Date Reported**: April 2, 2025 * **Issue Summary**: Delivery of frontend application assets degraded # **Impact Assessment** The incident resulted in degraded delivery of static assets used in the frontend applications, manifesting in the following: 1. Intermittent errors loading spaces 2. Missing assets in applications 3. NGINX error pages being viewed instead of Spaces The incident did not affect usage of the API and browser clients which had cached the static asset files. # **Root Cause** Our cloud hosting provider terminated several EC2 instances in our Kubernetes fleet over several hours the morning of April 2. The NGINX proxy that delivers static assets was forced to recreate on another node, resulting in several seconds of failed requests for assets. This occurred several times in succession. # **Resolution & Fix** 1. **Immediate Remediation** * Flatifle infrastructure engineers scaled NGINX resources across the fleet to avoid downtime during disruptions 2. **Recovery Strategy** * We implemented new routing and retry strategy combined with affinity rules to prevent scheduling on ephemeral resources # **Follow-Up Actions** * **Monitoring Enhancement**: While monitoring for this type of issue exists and alerts triggered correctly, enhancements could be made to escalate alerts and prompt faster response times.

Read the full incident report →

Started Mar 26, 2025, 05:31 PM UTC· resolved Mar 26, 2025, 06:09 PM UTC

1. investigating Mar 26, 2025, 05:31 PM UTC

   We are seeing a 503 Service Temporarily Unavailable error when trying to reach our dashboard. We are currently investigating this.

2. investigating Mar 26, 2025, 05:40 PM UTC

   We are continuing to investigate this issue.

3. monitoring Mar 26, 2025, 05:45 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 26, 2025, 06:09 PM UTC

   While making an enhancement update on the way we do releases, we encountered a drop in network traffic to our services due to a misconfiguration. We identified and reverted back the change to resolve the issue.

Read the full incident report →

Started Mar 25, 2025, 04:18 PM UTC· resolved Mar 25, 2025, 06:27 PM UTC

1. investigating Mar 25, 2025, 04:18 PM UTC

   We are seeing some intermittent 502 errors when loading Portals and performing some actions inside spaces. Our team is investigating this currently.

2. monitoring Mar 25, 2025, 05:53 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 25, 2025, 06:27 PM UTC

   Flatfile experienced an issue maintaining database connections via a proxy which led to service degradations whereby some connections from the app to the database unexpectedly hung up, returning a 502 error to the client. The connection issue with the proxy was resolved at 11:47am MDT.

Read the full incident report →