Siit Outage History

Started Oct 08, 2026, 07:31 AM UTC· resolved Oct 08, 2026, 09:01 AM UTC

1. investigating Oct 08, 2026, 07:31 AM UTC

   We are receiving multiple reports and alerts of issues affecting our app

2. investigating Oct 08, 2026, 07:42 AM UTC

   We have pushed a fix and expect recovery to happen during the next 10-40 minutes

3. monitoring Oct 08, 2026, 07:54 AM UTC

   Our fix appears to be working - our process are still catching their queue so you could still expect some latency in the delivery of pilled up notifications. We’ll come up with a detailed reports once we confirm that things settled down for good

4. monitoring Oct 08, 2026, 08:50 AM UTC

   We are still having impactful latency until we reach full recovery

5. monitoring Oct 08, 2026, 09:01 AM UTC

   All our monitoring data now appears back to normal, we will continue monitoring and provide a post mortem shortly.

6. resolved Oct 08, 2026, 09:43 AM UTC

   The issue is now fixed, the postmortem of the incident is available below. -- Summary On the morning of October 8, Siit stopped processing most of our background tasks between about 07:45 and 09:40 CEST. During that time, notifications, emails, Slack and Teams messages, workflows and integration syncs were delayed. Some interactive actions, such as clicking a button or talking to the AI Agent in a Slack or Teams message, couldn't be completed and may have needed a retry. While we caught up on the delayed work that had piled up, the app was slower than usual until about 11:00 CEST. All delayed work was queued safely and processed once service was restored. No data was lost. What happened A routine configuration cleanup after a recent release left our background processing unable to restart during scaling phases. The effect built up gradually overnight and became visible in the morning when activity picked up. How we resolved it We found the cause a few minutes after the investigation started and restored processing, handling the most time-sensitive tasks first. Catching up on the backlog put heavy load on our database, so we added database capacity and spread out the remaining work. Everything was back to normal by 11:00 CEST. Where we failed We had no alert for this specific failure, and the alerts we did have relied on the affected system itself As a result, the problem went unnoticed until it impacted you. What we're changing • We are revamping our Alerting, so such cases are caught right away. • Stricter safeguards and automated checks on production configuration changes. • We’ve upgraded our database capacity • Improving out-of-hours alerting and escalation We're sorry for the disruption to your teams. If an action from that morning still looks incomplete, retrying it should work. If not, our support team is happy to help.

Read the full incident report →

Started Sep 16, 2026, 10:13 PM UTC· resolved Sep 17, 2026, 08:51 AM UTC

1. investigating Sep 16, 2026, 10:13 PM UTC

   We are investigating slower than usual response time and are investigating the reason

2. monitoring Sep 16, 2026, 11:20 PM UTC

   We have identified the issue, applied a fix and we are seeing a noticeable decrease in response time. We'll keep monitoring for now.

3. identified Sep 17, 2026, 07:25 AM UTC

   The load and response time have been steadily increasing this morning and we're on it

4. monitoring Sep 17, 2026, 07:42 AM UTC

   We applied some configuration changes on our database cluster and are monitoring for recovery

5. monitoring Sep 17, 2026, 08:39 AM UTC

   Our jobs are pilling up, we are still applying some changes to fix the underlying issue and we expect to come back to normal in a few minutes. You might notice errors with our agents in the meantime until our queues has fully catched up.

6. monitoring Sep 17, 2026, 08:51 AM UTC

   Our infra seems to be back on tracks, we are monitoring for results and will provide more information once we confirm after a while.

7. resolved Sep 17, 2026, 09:33 AM UTC

   Summary Between the evening of 16 September and the morning of 17 September, Siit was intermittently slow and some requests failed. The admin dashboard was most affected. Two short interruptions of a few minutes each occurred on 17 September while our primary database failed over and recovered. The platform has been stable since 10:50 CEST on 17 September. Impact • 16 September, 20:20 to 00:15 CEST (18:20 to 22:15 UTC): intermittent slowness in the admin dashboard, with occasional requests timing out. • 17 September, 08:30 to 10:50 CEST (06:30 to 08:50 UTC): sustained slowness. Requests from the admin dashboard, the employee portal and the Slack and Teams integrations could take many seconds or fail with an error. • 17 September, 09:19 to 09:31 CEST and 10:36 to 10:47 CEST: two brief interruptions of all write operations while the primary database failed over and returned. • 17 September, 10:18 to 10:35 CEST: some Slack/Teams interactions and integration webhooks were not processed on first attempt. They were retried automatically. No data was lost or corrupted. Requests that failed returned an error to the user and were not partially applied. Automations and integrations that could not run were queued and completed once the platform recovered. What happened A recent change introduced real-time refresh of the admin dashboard, so that changes made by other admins or by automations appeared without a manual reload. Under real-world conditions, especially during high traffic / compute periods, this feature generated far more database read traffic than our testing had shown. How we resolved it - We rolled back the real-time refresh feature. The admin dashboard has returned to refreshing on navigation and auto/manual pulling for now. - Restored the primary database and increased read replicas capacity. - Reduced the number of open database connections from our application servers. What we are doing next - Redesigning the real-time refresh feature to batch updates on the server before it is re-enabled - Tightening the safeguards on our database autoscaling and improving our overall architecture to ensure more resilience in the future We are sorry for the disruption. If you noticed an impact we have not described here, please contact [email protected]: mailto:[email protected].

Read the full incident report →

Started Sep 10, 2026, 02:32 PM UTC· resolved Sep 10, 2026, 02:43 PM UTC

1. investigating Sep 10, 2026, 02:32 PM UTC

   We are experiencing an issue with our infra and our team is on it

2. monitoring Sep 10, 2026, 02:43 PM UTC

   We have deployed a fix and are monitoring

3. resolved Sep 10, 2026, 03:21 PM UTC

   This incident is now resolved. It was caused by an infrastructure configuration change that had unintended side effects. The change has been rolled back and we've added safeguards so it can't happen again. Apologies for the disruption and thank you for your patience.

Read the full incident report →

Started Sep 01, 2026, 01:42 PM UTC· resolved Sep 02, 2026, 08:41 AM UTC

1. identified Sep 01, 2026, 01:42 PM UTC

   We are seing an unusualy elevated error rate response from one of our LLM provider (OpenAI), our own AI agents are impacted

2. resolved Sep 02, 2026, 08:41 AM UTC

   We haven't received such errors for the past 12h on OpenAI's LLM

Read the full incident report →

Started Sep 01, 2026, 11:58 AM UTC· resolved Sep 01, 2026, 12:29 PM UTC

1. investigating Sep 01, 2026, 11:58 AM UTC

   We are investigating an increase in latency

2. monitoring Sep 01, 2026, 12:29 PM UTC

   Infrastructure is back to "normal" state - we are still working on improving the delay to resorb such spikes even more. Timeline (CEST): 13:51 - issues affecting our read cluster started pilling up 13.56 - Autoscalers started kicking in 14:01 - Autoscalers fully resorbed the issue

3. resolved Sep 01, 2026, 06:12 PM UTC

   The issue didn't reproduce - we implemented a few counter mesure for future occurences.

Read the full incident report →

Started Aug 27, 2026, 07:50 AM UTC· resolved Aug 27, 2026, 08:30 AM UTC

1. investigating Aug 27, 2026, 07:50 AM UTC

   We are investigating why our job processing are being much slower than usual

2. identified Aug 27, 2026, 08:00 AM UTC

   we have identified the issue and are working on resolving it asap

3. monitoring Aug 27, 2026, 08:30 AM UTC

   Our metrics are back to normal, we are still implementing more safeguards in the meantime while we keep monitoring

4. resolved Aug 27, 2026, 11:05 AM UTC

   We have resolved the incident and improved some of our infra to lower the risk of it happening again with those conditions.

Read the full incident report →

Started Jul 29, 2026, 01:49 PM UTC· resolved Jul 29, 2026, 02:39 PM UTC

1. identified Jul 29, 2026, 01:49 PM UTC

   We are investigating an issue of old Zendesk/Intercom/Jira messages from existing Siit threads being re-imported following a fix we recently pushed. This issue only affects messages with attachments, we are working on a fix and we'll keep you update.

2. identified Jul 29, 2026, 02:39 PM UTC

   After thorough investigation the blast radius has been identified to only a handful of customers, you should receive a message shortly after the fix is pushed explaining the impact. A fix is being pushed shortly to avoid those conditions occurring again - we'll also run a cleanup of the few affected requests (~80). Important: no data was lost or exposed. The re-imported messages came from your own tickets and were only ever visible to people who already had access to the requests in question. Unfortunately, Slack notifications that were already sent cannot be recalled.

3. monitoring Jul 29, 2026, 03:09 PM UTC

   The fix is currently being pushed

4. resolved Jul 29, 2026, 03:41 PM UTC

   Post mortem What happened: We received a report of old integration comments surfacing on stale requests at 13:39 UTC today, after investigation, and until the fix was shippe (between 2026-07-29 13:30:08 UTC to 2026-07-29 15:12:06 UTC) a handful of customers were impacted. Why it happened: following a sync-configuration update on our ticketing integrations (Jira, Intercom…), our synchronization provider re-delivered historical comments from your tickets. Some of these old comments were re-imported as new messages on the affected requests, specifically on customers that had previously removed / re-plugged their ticketing integration. What we did: A fix has been shipped live shortly after: historical comments can no longer be re-imported as new messages, and the import pipeline is now protected against this kind of re-delivery. We've also run a cleanup of the duplicated data. We'll contact the affected customer shortly with final number on affected imported messages. We're sorry for the inconvenience.

Read the full incident report →

Started Jul 21, 2026, 09:00 AM UTC· resolved Jul 21, 2026, 09:45 AM UTC

1. investigating Jul 21, 2026, 09:00 AM UTC

   We are currently investigating higher than usual response times

2. identified Jul 21, 2026, 09:13 AM UTC

   Our main database is under heavy load and is taking higher than usual time to process queries, auto-scaling is working to compensate but we're still investigating what caused the issue on the first place.

3. identified Jul 21, 2026, 09:45 AM UTC

   we are still working on fixes to prevent the burst from happening again but our DB metrics are back to normal

4. monitoring Jul 21, 2026, 03:53 PM UTC

   A fix has been deployed and we'll keep monitoring for the next few days

5. resolved Jul 22, 2026, 07:52 AM UTC

   Incident is resolve, we'll keep monitoring tightly for response times

Read the full incident report →

Started May 27, 2026, 03:32 PM UTC· resolved May 27, 2026, 10:26 PM UTC

1. identified May 27, 2026, 03:32 PM UTC

   We see a high number of errors due to an issue is currently affecting OpenAI, impacting customers running on its models (cf https://status.openai.com/incidents/01KSMYEEWP0BZYANSMAXWJ3K59)

2. monitoring May 27, 2026, 04:23 PM UTC

   The provider applied corrective actions and we are noticing a big decrease in the error response

3. resolved May 27, 2026, 10:26 PM UTC

   We haven't seen any error popping for a while and the provider's incident is closed as well.

Read the full incident report →

Started Apr 08, 2026, 01:08 PM UTC· resolved Apr 08, 2026, 02:18 PM UTC

1. investigating Apr 08, 2026, 01:08 PM UTC

   We are seeing an increased error rate on our Agent service and are currently investigating

2. monitoring Apr 08, 2026, 01:15 PM UTC

   A fix has been implemented and we're monitoring for results

3. resolved Apr 08, 2026, 02:18 PM UTC

   The fix has been successfully implemented and our AI service response rate is back to normal

Read the full incident report →

Started Mar 03, 2026, 09:44 AM UTC· resolved Mar 03, 2026, 09:54 AM UTC

1. investigating Mar 03, 2026, 09:44 AM UTC

   We are investigating an issue affecting our application as we see an increase in our error rate and timeouts happening

2. identified Mar 03, 2026, 09:50 AM UTC

   Our jobs are pilling up du to an issue with our infrastructure, we are implementing counter measures

3. monitoring Mar 03, 2026, 09:54 AM UTC

   The issue affecting our Aurora cluster is fixed and jobs haved catched up. We are still investigating why our cluster behaved in such a way to not have it happen again.

4. resolved Mar 04, 2026, 08:50 AM UTC

   We have implemented some safeguard to avoid our yesterday's cluster issues. We're closing this incident & keeping an eye on the metrics.

Read the full incident report →

Started Feb 27, 2026, 09:09 AM UTC· resolved Feb 27, 2026, 01:46 PM UTC

1. identified Feb 27, 2026, 09:09 AM UTC

   We are noticing an increase error rate on our application and are on it

2. monitoring Feb 27, 2026, 09:43 AM UTC

   A fix has been deployed and we're monitoring for results before confirming, but things do appear to be better

3. monitoring Feb 27, 2026, 09:50 AM UTC

   Still monitoring

4. identified Feb 27, 2026, 01:04 PM UTC

   We are noticing an increase response time which correlates with higher than usual load and are implementing counter measures

5. monitoring Feb 27, 2026, 01:34 PM UTC

   A fix has been deployed and we're monitoring for results

6. monitoring Feb 27, 2026, 01:46 PM UTC

   Our metrics are showing improvements after deployment of the fix, we'll continue monitoring

7. resolved Feb 27, 2026, 02:22 PM UTC

   Throughput and latency are back to normal, all systems are now 100% ok. Sorry about this inconvenience.

Read the full incident report →

Started Feb 25, 2026, 12:55 PM UTC· resolved Feb 25, 2026, 01:02 PM UTC

1. identified Feb 25, 2026, 12:55 PM UTC

   An issue is affecting existing SAML config and clients might be unable to log in to the admin Portal, a fix is being pushed.

2. monitoring Feb 25, 2026, 01:02 PM UTC

   Fix has been deployed and SAML login should be back to normal

3. resolved Feb 25, 2026, 01:25 PM UTC

   The issue is confirmed to be fixed

Read the full incident report →

Started Feb 17, 2026, 02:03 PM UTC· resolved Feb 23, 2026, 10:29 AM UTC

1. identified Feb 17, 2026, 02:03 PM UTC

   We have identified an issue on our analytics, having a backlog of data pilling up for a few days for some clients. We are currently running a backfill of the data, it might take a bit of time but we expect everything to be back to normal in circa 24h~.

2. resolved Feb 23, 2026, 10:29 AM UTC

   Data has fully catched-up by the end of last week

Read the full incident report →

Started Feb 11, 2026, 02:05 PM UTC· resolved Feb 11, 2026, 04:39 PM UTC

1. investigating Feb 11, 2026, 02:05 PM UTC

   We are noticing an increase in our response times and are currently investigating that issue.

2. identified Feb 11, 2026, 02:41 PM UTC

   We had a temporary huge spike in usage, we are working on a fix to ensure such spikes are absorbed more easily.

3. resolved Feb 11, 2026, 04:39 PM UTC

   Our infrastructure has been in a normal state for the past 90min, we'll keep a eye on it but for now we consider that incident resolved.

Read the full incident report →

Started Nov 18, 2025, 12:28 PM UTC· resolved Nov 18, 2025, 06:34 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Oct 20, 2025, 09:00 AM UTC· resolved Oct 21, 2025, 08:23 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →