Is Siit down?

Status: Investigating We are receiving multiple reports and alerts of issues affecting our app Affected components Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance) AI Agents (Partial outage) Customer Portal (Degraded performance)

Status: Investigating We have pushed a fix and expect recovery to happen during the next 10-40 minutes Affected components Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance) AI Agents (Partial outage) Customer Portal (Degraded performance)

Status: Monitoring Our fix appears to be working - our process are still catching their queue so you could still expect some latency in the delivery of pilled up notifications. We’ll come up with a detailed reports once we confirm that things settled down for good Affected components Public API & MCP (Operational) AI Agents (Degraded performance) Customer Portal (Degraded performance) Admin Dashboard (Operational)

Status: Monitoring We are still having impactful latency until we reach full recovery Affected components Customer Portal (Degraded performance) Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance) AI Agents (Degraded performance)

Status: Monitoring All our monitoring data now appears back to normal, we will continue monitoring and provide a post mortem shortly. Affected components AI Agents (Operational) Customer Portal (Operational) Admin Dashboard (Operational) Public API & MCP (Operational)

Status: Resolved The issue is now fixed, the postmortem of the incident is available below. -- Summary On the morning of October 8, Siit stopped processing most of our background tasks between about 07:45 and 09:40 CEST. During that time, notifications, emails, Slack and Teams messages, workflows and integration syncs were delayed. Some interactive actions, such as clicking a button or talking to the AI Agent in a Slack or Teams message, couldn't be completed and may have needed a retry. While we caught up on the delayed work that had piled up, the app was slower than usual until about 11:00 CEST. All delayed work was queued safely and processed once service was restored. No data was lost. What happened A routine configuration cleanup after a recent release left our background processing unable to restart during scaling phases. The effect built up gradually overnight and became visible in the morning when activity picked up. How we resolved it We found the cause a few minutes after the investigation started and restored processing, handling the most time-sensitive tasks first. Catching up on the backlog put heavy load on our database, so we added database capacity and spread out the remaining work. Everything was back to normal by 11:00 CEST. Where we failed We had no alert for this specific failure, and the alerts we did have relied on the affected system itself As a result, the problem went unnoticed until it impacted you. What we're changing • We are revamping our Alerting, so such cases are caught right away. • Stricter safeguards and automated checks on production configuration changes. • We’ve upgraded our database capacity • Improving out-of-hours alerting and escalation We're sorry for the disruption to your teams. If an action from that morning still looks incomplete, retrying it should work. If not, our support team is happy to help. Affected components Public API & MCP (Operational) AI Agents (Operational) Customer Portal (Operational) Admin Dashboard (Operational)

Status: Investigating We are investigating slower than usual response time and are investigating the reason Affected components AI Agents (Degraded performance) Customer Portal (Degraded performance) Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance)

Status: Monitoring We have identified the issue, applied a fix and we are seeing a noticeable decrease in response time. We'll keep monitoring for now. Affected components Public API & MCP (Operational) AI Agents (Operational) Customer Portal (Operational) Admin Dashboard (Operational)

Status: Identified The load and response time have been steadily increasing this morning and we're on it Affected components Public API & MCP (Degraded performance) AI Agents (Degraded performance) Customer Portal (Degraded performance) Admin Dashboard (Degraded performance)

Status: Monitoring We applied some configuration changes on our database cluster and are monitoring for recovery Affected components Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance) AI Agents (Degraded performance) Customer Portal (Degraded performance)

Status: Monitoring Our jobs are pilling up, we are still applying some changes to fix the underlying issue and we expect to come back to normal in a few minutes. You might notice errors with our agents in the meantime until our queues has fully catched up. Affected components Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance) AI Agents (Partial outage) Customer Portal (Degraded performance)

Status: Monitoring Our infra seems to be back on tracks, we are monitoring for results and will provide more information once we confirm after a while. Affected components Customer Portal (Operational) Admin Dashboard (Operational) Public API & MCP (Operational) AI Agents (Operational)

Status: Resolved Summary Between the evening of 16 September and the morning of 17 September, Siit was intermittently slow and some requests failed. The admin dashboard was most affected. Two short interruptions of a few minutes each occurred on 17 September while our primary database failed over and recovered. The platform has been stable since 10:50 CEST on 17 September. Impact • 16 September, 20:20 to 00:15 CEST (18:20 to 22:15 UTC): intermittent slowness in the admin dashboard, with occasional requests timing out. • 17 September, 08:30 to 10:50 CEST (06:30 to 08:50 UTC): sustained slowness. Requests from the admin dashboard, the employee portal and the Slack and Teams integrations could take many seconds or fail with an error. • 17 September, 09:19 to 09:31 CEST and 10:36 to 10:47 CEST: two brief interruptions of all write operations while the primary database failed over and returned. • 17 September, 10:18 to 10:35 CEST: some Slack/Teams interactions and integration webhooks were not processed on first attempt. They were retried automatically. No data was lost or corrupted. Requests that failed returned an error to the user and were not partially applied. Automations and integrations that could not run were queued and completed once the platform recovered. What happened A recent change introduced real-time refresh of the admin dashboard, so that changes made by other admins or by automations appeared without a manual reload. Under real-world conditions, especially during high traffic / compute periods, this feature generated far more database read traffic than our testing had shown. How we resolved it - We rolled back the real-time refresh feature. The admin dashboard has returned to refreshing on navigation and auto/manual pulling for now. - Restored the primary database and increased read replicas capacity. - Reduced the number of open database connections from our application servers. What we are doing next - Redesigning the real-time refresh feature to batch updates on the server before it is re-enabled - Tightening the safeguards on our database autoscaling and improving our overall architecture to ensure more resilience in the future We are sorry for the disruption. If you noticed an impact we have not described here, please contact [email protected]: mailto:[email protected]. Affected components Public API & MCP (Operational) AI Agents (Operational) Customer Portal (Operational) Admin Dashboard (Operational)

Status: Investigating We are experiencing an issue with our infra and our team is on it Affected components Admin Dashboard (Partial outage) Public API & MCP (Partial outage) AI Agents (Partial outage) Customer Portal (Partial outage)

Status: Monitoring We have deployed a fix and are monitoring Affected components AI Agents (Operational) Customer Portal (Operational) Admin Dashboard (Operational) Public API & MCP (Operational)

Status: Resolved This incident is now resolved. It was caused by an infrastructure configuration change that had unintended side effects. The change has been rolled back and we've added safeguards so it can't happen again. Apologies for the disruption and thank you for your patience. Affected components Admin Dashboard (Operational) Public API & MCP (Operational) AI Agents (Operational) Customer Portal (Operational)

Status: Identified We are seing an unusualy elevated error rate response from one of our LLM provider (OpenAI), our own AI agents are impacted Affected components AI Agents (Degraded performance)

Status: Resolved We haven't received such errors for the past 12h on OpenAI's LLM Affected components AI Agents (Operational)

Status: Investigating We are investigating an increase in latency Affected components Customer Portal (Degraded performance) Admin Dashboard (Degraded performance) Public API & MCP (Degraded performance) AI Agents (Degraded performance)

Status: Monitoring Infrastructure is back to "normal" state - we are still working on improving the delay to resorb such spikes even more. Timeline (CEST): 13:51 - issues affecting our read cluster started pilling up 13.56 - Autoscalers started kicking in 14:01 - Autoscalers fully resorbed the issue Affected components Admin Dashboard (Operational) Public API & MCP (Operational) AI Agents (Operational) Customer Portal (Operational)

Status: Resolved The issue didn't reproduce - we implemented a few counter mesure for future occurences. Affected components Admin Dashboard (Operational) Public API & MCP (Operational) AI Agents (Operational) Customer Portal (Operational)