Olark Outage History

Started Apr 07, 2025, 08:51 PM UTC· resolved Apr 07, 2025, 10:20 PM UTC

1. investigating Apr 07, 2025, 08:51 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Apr 07, 2025, 08:53 PM UTC

   We're investigating reports of delayed message delivery and will update here as soon as we know more.

3. monitoring Apr 07, 2025, 09:07 PM UTC

   A fix has been implemented and we're back to normal message delivery times. Agents can resume using the service as normal. We'll continue to monitor closely and post updates here.

4. resolved Apr 07, 2025, 10:20 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 19, 2025, 08:41 PM UTC· resolved Mar 19, 2025, 10:49 PM UTC

1. investigating Mar 19, 2025, 08:41 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Mar 19, 2025, 08:44 PM UTC

   We're investigating issues with message delivery. We'll update here as soon as we know more.

3. identified Mar 19, 2025, 08:58 PM UTC

   We've identified the cause of the issue with message delivery and working to implement a fix.

4. monitoring Mar 19, 2025, 09:16 PM UTC

   A fix has been implemented and systems have returned to normal. Agents are free to log in and use the service. We'll continue to monitor closely.

5. resolved Mar 19, 2025, 10:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 27, 2025, 12:37 AM UTC· resolved Feb 27, 2025, 02:46 AM UTC

1. investigating Feb 27, 2025, 12:37 AM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. identified Feb 27, 2025, 01:06 AM UTC

   We've identified an issue with message delivery and are working to implement a fix. We'll update here as soon as we know more.

3. monitoring Feb 27, 2025, 01:45 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 27, 2025, 02:46 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 27, 2025, 12:12 AM UTC· resolved Feb 27, 2025, 01:46 AM UTC

1. identified Feb 27, 2025, 12:12 AM UTC

   We're experiencing degraded performance from an AI provider that may be affecting the performance of Olark bots. We'll continue to monitor and provide updates as we have them.

2. identified Feb 27, 2025, 01:46 AM UTC

   We are continuing to work on a fix for this issue.

3. resolved Feb 27, 2025, 01:46 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 25, 2025, 10:27 PM UTC· resolved Feb 25, 2025, 11:32 PM UTC

1. investigating Feb 25, 2025, 10:27 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Feb 25, 2025, 10:31 PM UTC

   We're investigating app.olark.com not loading properly. We'll update here as soon as we know more.

3. identified Feb 25, 2025, 10:32 PM UTC

   We've identified the cause of app.olark.com not loading and are implementing a fix.

4. monitoring Feb 25, 2025, 10:38 PM UTC

   A fix has been implemented and app.olark.com is loading properly. Agents can resume chatting as normal. We'll continue to monitor closely.

5. resolved Feb 25, 2025, 11:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 24, 2025, 10:29 PM UTC· resolved Feb 25, 2025, 01:30 AM UTC

1. investigating Feb 24, 2025, 10:29 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Feb 24, 2025, 10:36 PM UTC

   We're investigating issues with chat routing. We'll update here as we know more.

3. identified Feb 24, 2025, 10:49 PM UTC

   We've identified the source of the chat routing issues and are working to implement a resolution.

4. monitoring Feb 24, 2025, 11:03 PM UTC

   A fix has been implemented and chats are being routed properly. We'll continue to monitor closely.

5. resolved Feb 25, 2025, 01:30 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 29, 2025, 03:06 PM UTC· resolved Jan 29, 2025, 06:21 PM UTC

1. investigating Jan 29, 2025, 03:06 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. identified Jan 29, 2025, 03:19 PM UTC

   We detected some issues with agent presence and have taken steps to resolve. We'll update here as we know more.

3. monitoring Jan 29, 2025, 05:11 PM UTC

   A fix has been implemented and we're monitoring the results.

4. resolved Jan 29, 2025, 06:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 19, 2025, 04:01 PM UTC· resolved Jan 19, 2025, 07:27 PM UTC

1. investigating Jan 19, 2025, 04:01 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Jan 19, 2025, 04:42 PM UTC

   We're investigating reports of issues with agent presence. We'll update here as we know more.

3. identified Jan 19, 2025, 04:59 PM UTC

   We've identified the source of the issue and are working to correct it.

4. monitoring Jan 19, 2025, 05:19 PM UTC

   We've resolved the agent presence issue and will continue to monitor closely. Agents should be able to resume chat as normal.

5. resolved Jan 19, 2025, 07:27 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 19, 2025, 03:12 AM UTC· resolved Jan 19, 2025, 03:55 AM UTC

1. investigating Jan 19, 2025, 03:12 AM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. resolved Jan 19, 2025, 03:55 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 25, 2024, 04:23 PM UTC· resolved Nov 25, 2024, 07:43 PM UTC

1. investigating Nov 25, 2024, 04:23 PM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Nov 25, 2024, 04:31 PM UTC

   We're investigating issues with agent being set to away unexpectedly and receiving messages when set to away. We'll update here as soon as we know more.

3. investigating Nov 25, 2024, 05:31 PM UTC

   We're continuing to investigate issues with agent presence and message delivery when agents are set to away. We'll update here as soon as we have more information.

4. monitoring Nov 25, 2024, 05:57 PM UTC

   We've implemented a fix for the issue with agent presence and chats being assigned when agents were set to away. Systems have recovered and agents should be free to log in and use the service as normal. We'll continue to monitor closely and update here if there are changes.

5. resolved Nov 25, 2024, 07:43 PM UTC

   This incident has been resolved. We're working on a full investigation of the issue and causes and will publish a post-incident writeup here once complete.

6. postmortem Nov 26, 2024, 09:40 PM UTC

   On Sunday November 24 from approximately 19:12 PST until 19:38 PST, and again on Monday November 25th from 08:23 PST until 09:57 PST, Olark experienced a partial outage that affected some agents’ online/offline status. During these windows, some agents’ status was set to offline involuntarily, and some agents received incoming chats while their status was set to away. ‌ **What happened:** Our team was initially alerted by our internal monitoring systems of an issue with one of our event queuing nodes on Sunday around 19:00 PST. Shortly thereafter, customers began reporting issues with agent status. Initial investigation resulted in steps being taken by our team to restart the queuing node, and systems began to recover quickly. By 19:38 PST systems had fully returned to normal operation. ‌ On Monday November 25, around 08:20 PST, we received new reports of agent status problems for some agents, and we began additional investigation. Initially this issue looked like a state mismatch due to caching in internal systems and we took steps to resolve that issue. However internal tests showed that the presence issue persisted for some agents after those resolutions. Additional steps were taken to once again restart the queuing node that caused the issues the previous day, resolving the status issue and returning systems to normal operation by 09:57 PST. ‌ **Why this happened:** Our event queuing system is integral to the proper functioning of Olark. From our initial investigation, a single node in our event queuing cluster lost network connectivity, and required a hard reboot on Sunday night. Although this queuing system is designed for redundancy, this node did not disconnect cleanly from the cluster. When the node came back up after the first restart, all systems were verified to be working correctly in our testing and monitoring. Once it became clear Monday morning that there was still some remaining cluster state to resolve, we performed a managed restart of the node, which did additional work to force traffic to be redistributed properly to the other nodes. We concluded that the original restart left some cluster state unresolved, which was the cause of the issues on Monday morning. ‌ **How are we preventing this in the future:** Our goal is to detect this kind of cluster state issue for immediate correction in the future. We already have significant monitoring at the infrastructure level, queue level, and application level. However, as a result of this incident, we are immediately this week: \(1\) adding new end-to-end regression tests for this specific application failure, and \(2\) asking our on-call engineers and support team to be extra vigilant over this Thanksgiving holiday week and increase frequency of manual human supervision of our systems. We have already increased the alert level for existing automated monitors, and we have also added new diagnostic tools for our on-call engineers to detect this specific issue. ‌ Beyond those immediate monitoring and correction steps, we aim to prevent these issues from happening in the first place. If one node exits our event queuing cluster, remaining nodes should "heal" the cluster and immediately pick up the work without customer impact. Since healing did not occur correctly in this case, the highest priority for our engineering team is isolating a solution to ensure all disconnects and reconnects are clean. We have made this work a top priority for our engineering team to ensure we continue to have stability for this core component of our infrastructure.

Read the full incident report →

Started Nov 25, 2024, 03:12 AM UTC· resolved Nov 25, 2024, 04:30 AM UTC

1. investigating Nov 25, 2024, 03:12 AM UTC

   We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

2. investigating Nov 25, 2024, 03:25 AM UTC

   We're investigating reports of issues with message delivery. We'll update here as soon as we know more.

3. monitoring Nov 25, 2024, 03:38 AM UTC

   We've implemented a fix for the issue we were seeing with message delivery and systems have returned to normal. We're continuing to monitor closely.

4. resolved Nov 25, 2024, 04:30 AM UTC

   This incident has been resolved.

Read the full incident report →