Chili Piper Outage History

Started Sep 15, 2026, 05:27 PM UTC· resolved Sep 16, 2026, 05:20 PM UTC

1. identified Sep 15, 2026, 05:27 PM UTC

   We are aware of an issue with customers trying to connect to Gong in the Command Center. This should affect customers trying to connect, whether it’s their first time connecting to Gong or not. Customers who are already connected should not see any impact. Based on our findings, the resolution will require a joint effort with Gong. Gong's Support was already engaged with urgency and informed us that their Engineering Team is actively working on it. We will keep this page updated with the resolution’s progress.

2. monitoring Sep 16, 2026, 02:36 PM UTC

   We were informed that Gong has already implemented a fix, meaning customers should be able to connect to Gong successfully in Chili Piper's Command Center. We will now monitor the results after the fix's implementation. Please email [email protected] if you still see any errors.

3. resolved Sep 16, 2026, 05:20 PM UTC

   We received positive feedback from affected customers after Gong's team applied the fix, so we consider this resolved. Still, please email [email protected] if you notice anything unexpected.

Read the full incident report →

Started Jun 22, 2026, 02:27 PM UTC· resolved Jun 22, 2026, 02:55 PM UTC

1. investigating Jun 22, 2026, 02:27 PM UTC

   We are receiving multiple reports of HubSpot-based customers unable to load objects in their instances, which also affects their rule evaluation and routing. Our teams are investigating the source, as we're getting timeout errors from Hubspot API connections. This will cause issues routing, booking or properly updating Hubspot records. We are investigating a potential solution, or temporary workaround, that can be provided until we identify the root cause.

2. investigating Jun 22, 2026, 02:44 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 22, 2026, 02:46 PM UTC

   HubSpot has now confirmed it was partially unavailable in North America from 09:34 AM to 10:25 AM (UTC -04:00). They investigating the cause and will update this page when they have more information. We are starting to see the issues resolved in Chili Piper now. We're also actively monitoring further updates from HubSpot.

4. resolved Jun 22, 2026, 02:55 PM UTC

   HubSpot has now confirmed a fix to the issue. We have also confirmed restoration of services in Chili Piper.

Read the full incident report →

Started May 15, 2026, 09:00 PM UTC· resolved May 15, 2026, 09:00 PM UTC

1. resolved May 18, 2026, 07:14 PM UTC

   We are aware of potential delays while routing records through Distro in one of our environments. As a result, you may notice that Distro is not identifying and, consequently, routing new records. Our Engineering Team is already aware and actively working on a resolution as their top priority. This is not impacting other products such as Concierge, Chat, Handoff, and their respective routing, nor ChiliCal and its browser extension. In case of any further questions, please reach out to our Customer Love Team at [email protected]

Read the full incident report →

Started Apr 15, 2026, 01:56 PM UTC· resolved Apr 15, 2026, 04:18 PM UTC

1. investigating Apr 15, 2026, 01:56 PM UTC

   As of approximately 9:00 AM Eastern Time, our team is investigating reports of issues with rule processing across products. his is causing routing failures in Handoff, Concierge, and potentially other services that rely on rule and team assets. We are actively working to identify the root cause and will provide updates as more information becomes available.

2. resolved Apr 15, 2026, 04:18 PM UTC

   This has now been resolved after our Engineering Team rolled back one of our services. Rules, Teams and routings are expected to be processed normally now across all products that rely on these assets. We have also already identified what caused it, so any new deployments done for this service should include a fix.

Read the full incident report →

Started Apr 08, 2026, 07:29 PM UTC· resolved Apr 08, 2026, 08:38 PM UTC

1. investigating Apr 08, 2026, 07:29 PM UTC

   We see some customers have connectivity issues with Salesforce CRM. In the app, it will display "In Trouble" status when it is not connected. All CRM functionality, such as Contact/Account routing, email lookups, and rules logic may not work correctly while this is the status. We are internally working to reestablish the connection for affected customers, but in the meantime, if you see "In Trouble" or "Disconnected" status in the Admin Center's Integrations page, please try reconnecting to Salesforce from the appropriate integration user's Salesforce account, which should restore the connection. More updates to be shared soon.

2. identified Apr 08, 2026, 08:01 PM UTC

   We have deployed a fix for the Salesforce connectivity issues. We will also apply a retroactive update to proactively reestablish the Salesforce connection for affected tenants. The ETA for completion is approximately 30 minutes. In the meantime, if you see your Salesforce connection status currently set to "In Trouble," you can also have the integration user reconnect to Salesforce in the admin center, which will resolve the problem immediately.

3. resolved Apr 08, 2026, 08:38 PM UTC

   The retroactive update has been completed, and all accounts should properly be in a connected status at this time. We do have a retry mechanism that will automatically re-attempt CRM Actions in Salesforce if they had been previously unsuccessful in most cases.

Read the full incident report →

Started Apr 07, 2026, 07:38 PM UTC· resolved Apr 07, 2026, 07:38 PM UTC

1. resolved Apr 07, 2026, 07:38 PM UTC

   Heads up: Chili Piper has officially deprecated our Legacy service from the HubSpot Marketplace. This legacy app was retired last year and is no longer in use. There is no impact to any functionality or connectivity on the current Fire / Demand Conversion Platform. However, the removal may have triggered some automated emails that lack context. You may have received messages such as: Chili Piper Scheduling deleted by provider Log in with HubSpot deleted by provider Chili Piper - Legacy deleted by provider These notifications can be safely ignored with no further action required. Note: You may have received these emails even if you're currently on Salesforce, if your account was ever connected to HubSpot in the past. This does not mean you are currently connected to HubSpot.

Read the full incident report →

Started Mar 09, 2026, 06:27 PM UTC· resolved Mar 09, 2026, 07:17 PM UTC

1. investigating Mar 09, 2026, 06:27 PM UTC

   We're investigating reported issues that the Chat Inbox feature of our admin center is not loading for customers. This does not impact customer-facing chat appearance, however it will impact end-user ability to reply. As such, we are treating this as a partial outage to the Chat service. AI-based chats and pre-built chat journeys that require no inbox interaction will not be impacted by this incident. Additionally, this doesn't impact the ability to manage or build a Chat journey.

2. resolved Mar 09, 2026, 07:17 PM UTC

   This incident has been resolved. Internal action was taken to ensure that this specific failure point in the chat service cannot again in the future.

Read the full incident report →

Started Feb 19, 2026, 03:11 PM UTC· resolved Feb 19, 2026, 03:45 PM UTC

1. investigating Feb 19, 2026, 03:11 PM UTC

   Our team started investigating an issue where Workspaces and licenses are not loading for Admins. The issue appears to be related to our payment processor/billing service, whose own API is erroring. As we access this API for workspace-related data, such as licenses and even product access within the app, this causes the pages to become unresponsive. This only affects the internal Admin Center, Workspaces access, and product-related menus within the app for Admins. It does not impact products or functionality, including booking, routing, and assignment. We will keep this incident updated as soon as we have more updates.

2. resolved Feb 19, 2026, 03:45 PM UTC

   We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have ongoing efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such scenarios.

Read the full incident report →

Started Dec 17, 2025, 06:22 PM UTC· resolved Dec 17, 2025, 08:09 PM UTC

1. investigating Dec 17, 2025, 06:22 PM UTC

   We are currently investigating an issue that is preventing email notifications from being sent. This means Chili Piper users will likely not receive notifications, such as meeting confirmations and record assignments. This is not impacting Concierge, Concierge routing, the booking/scheduling platform, or ChiliCal's extension. Also, prospects are expected to receive email notifications and calendar events, as the issue is isolated to Chili Piper users' notifications. We will send more updates as soon as our investigation moves forward. If any questions, please reach out to [email protected]

2. resolved Dec 17, 2025, 08:09 PM UTC

   We are now considering this issue resolved, as we're seeing better results. The issue lasted from 17:30 to 19:20 UTC and impacted email notifications such as meeting confirmations emails, record assignment emails, and Reminders for Chili Piper users. The internal service responsible for such notifications had some configuration issues, which were already mitigated. Our teams are already working on lessons learned and how we can avoid this from happening in the future. The issue has not affected the ability to book meetings across products, including our extension, assigning records via Distro, and prospects from receiving meeting invites and calendar event confirmations. If you have any further questions, please reach out to [email protected]

Read the full incident report →

Started Dec 12, 2025, 10:39 PM UTC· resolved Dec 12, 2025, 11:23 PM UTC

1. investigating Dec 12, 2025, 10:39 PM UTC

   We are currently investigating reports of increased load times affecting legacy links and redirects. The impact we are seeing is on both links and web form snippets. This impacts both standard booking links (meeting types, queues), and Concierge router links. Load times vary from 10-30 seconds on impacted calendar links. Once links load, it appears they do not natively redirect to Fire platform as they are supposed to. As this is impacting only our legacy platform, a potential fix would be to fully migrate services and links to their redirected Fire platform counterparts as they are not impacted. No Fire services appear to be impacted at this time.

2. identified Dec 12, 2025, 10:58 PM UTC

   It appears the issue may stem from Cross-Origin Resource Sharing (CORS) adjustments that were made. As a result, the legacy Canary platform links are blocked from accessing various resources needed to properly load booking link pages. Our team is working to restore access to these resources now. This does not impact Fire platform or Fire services including Fire scheduling links, and it appears to be limited to a specific "Canary" environment within our legacy platform.

3. resolved Dec 12, 2025, 11:23 PM UTC

   We've completed the necessary configuration changes and access to resources has been restored for impacted legacy customers. A thorough review of this configuration and what occurred will be completed to ensure it remains stable.

Read the full incident report →

Started Nov 11, 2025, 10:01 PM UTC· resolved Nov 11, 2025, 10:46 PM UTC

1. investigating Nov 11, 2025, 10:01 PM UTC

   Our team has received reports and has found evidence that in many cases the Google calendar service is not creating Events on the calendar. This appears to be a wide-spread issue, but does not appear to impact all meetings booked. We are investigating the root cause of this issue now and will provide recommendations once we discover more about the circumstances of this issue.

2. resolved Nov 11, 2025, 10:46 PM UTC

   Our team identified that the issue was related to a deployment of our integration services at approximately 15:32 EST (20:32 UTC) for an incident span of approximately 2 hours. We rolled back the deployment and this resolved the issue, and we saw error rates drop to zero. It is likely that any meetings booked during this time may have been impacted where no Google calendar event was created in its place. We strongly recommend manually creating these events if they're identified in the meantime rather than waiting on a replay action (if this is possible from our end) as this will ensure that all parties have the most up-to-date calendar appearance, and ensures prospects receive meeting invites. Apologies for the disruption this may have caused.

3. postmortem Nov 12, 2025, 02:42 PM UTC

   We discovered this morning that Outlook invites and events were also impacted by the same service failure, extending the scope to both Google and Outlook customers. We are working on recovering all failed events, but don’t have a specific timeline for this action. Our hope is to get this finalized today. It’s likely that, while we are waiting on recovery, other meetings may get booked into those open calendar slots. This would result in overbooked meetings once the recovery finalizes, but this is the only way we can accommodate both parties who booked meetings during that time slot. One of the remaining events would ultimately need to be rescheduled manually. We are investigating our processes that led to the release of this bug and will improve to ensure more thorough testing is performed before release. Apologies for the inconvenience this has caused.

Read the full incident report →

Started Nov 07, 2025, 01:56 PM UTC· resolved Nov 07, 2025, 03:50 PM UTC

1. investigating Nov 07, 2025, 01:56 PM UTC

   Our team started investigating an issue where Workspaces and licenses are not loading for Admins. As a side effect, some products may appear inaccessible within the app. This only affects the internal Admin Center and product-related menus within the app. It does not impact products or functionality, including booking or routing. We will keep this page updated once we have more updates.

2. investigating Nov 07, 2025, 01:57 PM UTC

   We are continuing to investigate this issue.

3. identified Nov 07, 2025, 02:54 PM UTC

   The issue appears to be related to our payment processor, whose own API is erroring. As we access this API for workspace-related data, such as licenses and even product access within the app, this causes the pages to become unresponsive. We have already escalated this to their team with high priority, who have already acknowledged the issue. We will send more updates as available.

4. resolved Nov 07, 2025, 03:50 PM UTC

   We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have already started coordinating efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such occurrences.

Read the full incident report →

Started Nov 06, 2025, 05:59 PM UTC· resolved Nov 06, 2025, 07:58 PM UTC

1. monitoring Nov 06, 2025, 05:59 PM UTC

   The service responsible for processing Salesforce actions is currently experiencing degraded performance, which can cause delays on the following related Salesforce actions: Create Event, Create or Update Record, Update Owner, Add to Campaign, and Update Field. You may see your Salesforce actions as "Pending" or "In progress". The booking/scheduling process and apps are not affected, meaning meetings should be booked normally. Our teams are currently working to mitigate delays, and we will keep you informed once we have more updates.

2. resolved Nov 06, 2025, 07:58 PM UTC

   Our teams reported that the consumption should be back to normal. You should notice previously delayed Salesforce actions being marked as Successful, and new ones should be processed within our standards. Over the past few days, the system reached a critical load level that exposed a performance bottleneck in our Salesforce data processing. After thorough analysis, we identified and resolved the issue, restoring normal operation and improving overall throughput. We are continuously improving our systems and infrastructure to process customer data more efficiently and reliably. As data volumes increase, we regularly adjust configurations and upgrade underlying tools and libraries to maintain stable performance. If you still notice any unexpected behavior, please report it to [email protected] for further analysis.

Read the full incident report →

Started Oct 29, 2025, 06:46 PM UTC· resolved Oct 29, 2025, 09:15 PM UTC

1. monitoring Oct 29, 2025, 06:46 PM UTC

   Microsoft services, including calendars, Azure, and hosted office apps are all experiencing degradation due to a global Microsoft issue. As a result, this may impact the ability to access or load some Chili Piper pages and services as well. All legacy customers will likely experience some slowness or page loading issues. Most Outlook/Office-based customers will see some performance issues as well. We are monitoring from our end, however for status updates please check official Microsoft service channels such as : https://azure.status.microsoft/en-us/status https://status.cloud.microsoft/m365

2. resolved Oct 29, 2025, 09:15 PM UTC

   We are seeing services recover, so are marking this "Resolved". For more in-depth information about the incident and its current status please check the following status updates provided by Microsoft: https://azure.status.microsoft/en-us/status https://status.cloud.microsoft/m365

Read the full incident report →

Started Oct 01, 2025, 08:22 PM UTC· resolved Oct 02, 2025, 01:20 PM UTC

1. investigating Oct 01, 2025, 08:22 PM UTC

   We have been alerted to potential issues routing records through Distro. In most cases these do recover and route after a delay, but in many instances the routes don't appear to be processed at all. As such, we are treating this as a critical incident within the Distro service. This is not impacting Concierge, Concierge routing, or the booking / scheduling platform. Our team will continue to monitor this and work with our engineers to get to the root cause as soon as possible.

2. monitoring Oct 01, 2025, 11:12 PM UTC

   Our team applied improvements to our Distro service, and we’re seeing better results. Still, records that should’ve been routed via Distro while the service had degraded performance may take some time to be processed. It was also identified that this affected only one of our environments, so we don’t expect all Distro users to be affected by this. We will keep monitoring the results and will soon provide more updates and a root cause. If you have any questions, please email [email protected]

3. resolved Oct 02, 2025, 01:20 PM UTC

   Distro is back to an operational state. The issue appears to have been caused by an influx of triggered events which exceeded the maximum load possible. This caused the event processing time to grow exponentially for all customers. We are correcting for this in several ways including improving current services to accommodate the increased load, and also improving the strategy for how events are processed. While some of these improvements are ongoing, the incident is considered resolved.

Read the full incident report →

Started Sep 10, 2025, 02:41 PM UTC· resolved Sep 10, 2025, 02:56 PM UTC

1. investigating Sep 10, 2025, 02:41 PM UTC

   Our team has begun investigating an issue where any admin attempting to access workspaces or admin center internally will have the app crash and show "Something went wrong. Please, reload the page". While reloading the page does seem to work occasionally, the page will eventually crash again and has other performance issues loading. This only impacts the internal admin center and is not impacting products or functionality including booking or routing.

2. resolved Sep 10, 2025, 02:56 PM UTC

   The issue appears to have been related to our payment processor, whose own API had an outage. As we access this API for various admin center and workspace related data such as licenses and even product access, this caused the pages to become unresponsive and crash. We have observed the related API self-healing, and access to workspace and admin center have both been restored, however as this service is outside of our ecosystem we are monitoring now to confirm that everything is back to being stable. We are going to examine how we can better handle (gracefully) third-party billing and payment outages such as this in order to prevent interruption to workflows, while also maintaining product license access in the future.

Read the full incident report →

Started Aug 19, 2025, 09:49 PM UTC· resolved Aug 19, 2025, 10:36 PM UTC

1. investigating Aug 19, 2025, 09:49 PM UTC

   We have had reports that Hubspot is no longer functional for all customers due to a new scope being requested on the authorization. Our teams are investigating the source, but the issue has caused all Hubspot API connections to drop. This will cause issues routing with Concierge, or properly updating Hubspot records. We are investigating a potential solution, or temporary workaround, that can be provided until we identify the root cause.

2. resolved Aug 19, 2025, 10:36 PM UTC

   A fix was deployed and all connections to Hubspot are working again. We have confirmed that customers are able to route and update Hubspot records properly. It was noted that not all Hubspot CRM customers became impacted by this issue immediately. We began receiving reports as early as 3pm EDT(7pm UTC) and so you may see issues in routing for at least this amount of time. During that time, customers may not have been able to route / book a call. If you find that your Chili Piper account is still not properly connected to Hubspot, or if you continue to experience issues with routing and are a Hubspot customer, please try browsing to your integration settings within the Chili Piper admin panel and reconnecting Hubspot. The root cause appears to be related to a permission scope that Hubspot used to provide "as requested" now requires explicit permissions for it to work. Since we didn't authenticate this permission in the past, all authorizations were failing. We have deployed a fix which essentially removes the need for this scope. Our team will explore if this scope is going to be needed in the future and will take appropriate measures if this is the case. We will also continue monitoring to ensure that all instances are back up and running properly with this provided hotfix. Please feel free to reach out to [email protected] for further information.

Read the full incident report →

Started Aug 19, 2025, 02:30 PM UTC· resolved Aug 19, 2025, 02:30 PM UTC

1. resolved Aug 19, 2025, 02:39 PM UTC

   We are in the process of moving to a new billing / payment processor this morning. As a result, you may have temporarily lost the ability to login for a few seconds up to a few minutes, however this should not impact any workflows, routing, or bookings. If you notice the inability to login, please wait 5-10 minutes and try again. If you continue to experience issues, feel free to reach out to [email protected] so that our Customer Love team can take a closer look for you! Apologies in advance for the inconvenience this may have caused.

Read the full incident report →

Started Jul 27, 2025, 04:00 PM UTC· resolved Jul 28, 2025, 07:02 PM UTC

1. resolved Jul 28, 2025, 04:43 PM UTC

   Chili Piper experienced a full platform outage related to services dependent on Kafka. These services were unable to send or receive data during this time which essentially caused all products to go offline and lose access. The outage was detected immediately by our internal alerting system and our on-call Site Reliability Engineering team began working to correct the issue. We raised the escalation with the Confluent Cloud team through all available channels. Approximately 1 hour and 50 minutes after the incident began, Confluent confirmed that their internal teams were actively engaged in resolving the issue with additional escalation within Confluent prioritized at this time. To mitigate the issue, we created a new subscription through their system in order to restore access. Service was fully restored at 19:12 UTC (3:02pm EDT). No existing data was lost, but no new data was processed during this time which means no routes or meetings could be booked through Chili Piper.

2. postmortem Jul 28, 2025, 04:44 PM UTC

   ## **Action Taken** * Escalated to Confluent and engaged with their engineering team throughout the incident. * Internally began preparation of a fallback Kafka cluster independent of Confluent, to be used if needed. * After recovery, performed thorough verification of platform functionality and monitored stability. * Initiated and coordinated an investigation with Confluent. * Requested and received a detailed postmortem from Confluent outlining root cause and contributing factors. ## **Follow-up Steps** * Evaluate feasibility and timeline for decoupling from Confluent to ensure greater control and resilience. * Set up monitoring and alerting specifically for Kafka cluster reachability to detect future issues earlier. * Review and improve vendor migration validation processes. * Review Confluent root cause analysis and postmortem

Read the full incident report →

Started Jul 21, 2025, 04:27 PM UTC· resolved Jul 21, 2025, 04:27 PM UTC

1. resolved Jul 21, 2025, 04:27 PM UTC

   Our team received reports of degraded performance defined as, and ranging from, slowness loading Concierge routes; pages generally loading slowly; or cases where purchased licenses weren't being immediately updated. We identified the most significant root cause as an issue with a third-party library used in our backoffice infrastructure which was causing our services to become "poisoned". This means that when the service faced a critical state, it did not terminate cleanly and instead kept running while sick -- resulting in slowness. Our fix has been two-fold to begin with: 1. We have adjusted the library to a state where the issue does not exist. 2. We will monitor services for similar poisoned status, and if experienced, these services will be restarted so that they can recover properly. We have already deployed fixes for this and have had confirmation that there is a marked improvement in load times. We will continue to monitor this moving forward, making adjustments as needed.

Read the full incident report →

Started Jun 12, 2025, 06:52 PM UTC· resolved Jun 12, 2025, 10:45 PM UTC

1. identified Jun 12, 2025, 06:52 PM UTC

   Google IAM service is currently experiencing issues resulting in pages that display Google calendars to either load slowly, or not at all. We are awaiting a fix from Google. All customers across all platforms are impacted.

2. identified Jun 12, 2025, 06:55 PM UTC

   For further updates, please refer to Google's status page for their identity provider status: https://status.cloud.google.com/

3. identified Jun 12, 2025, 09:21 PM UTC

   We are receiving many questions around events that are actively getting booked not appearing in Google invites, and what this means for customers. Hopefully we can highlight some important points here. - Invites are failing to get created. This impacts both prospects/customers and your assignees/bookers - The meeting within the CRM and Chili Piper may exist, but the calendar invite will not - Right now, we don't have any way to replay these events - We will see missing invites from the time the outage started until it resolves on Google's side We are working with engineers to see if it may be possible to replay these events, however as mentioned it is not currently possible. If this changes, we will provide an update here with details. Until that time comes, we are recommending a few different options: a) book the events to Google manually for each new booking b) rely on a CRM sync to create / sync the missing google events c) once the outage is over, cancel the event and re-book to the same date/time with all guests Unfortunately, until the outage is over and Google services are restored, we will likely continue to see the issue continue.

4. resolved Jun 12, 2025, 10:45 PM UTC

   Google has marked most services as recovered, and we are seeing recovery from our services as well. Please reference Google's status page at https://status.cloud.google.com/ in case you continue to experience issues with invite creation or logins as some services might still be recovering slowly. The latest status of these issues can be found there.

Read the full incident report →

Started Apr 30, 2025, 09:45 AM UTC· resolved Apr 30, 2025, 10:10 PM UTC

1. investigating Apr 30, 2025, 10:32 AM UTC

   Some users are experiencing issues when accessing Chili Cal scheduling links. In certain cases, the links fail to load and instead display a "Something Went Wrong" error screen. This behavior is not affecting all links, but the issue appears to occur intermittently across multiple users. Our team is actively investigating the root cause of the issue and working to restore normal functionality as soon as possible. Updates will be provided as more information becomes available.

2. monitoring Apr 30, 2025, 01:08 PM UTC

   The underlying issue appears to have been caused by caching which naturally took time to clear. Our team has forced the instances to clear and we should see service recovered now. Local cache may still be a factor in displaying these pages, so please try clearing browser cache if you see the issue remain. Our team is monitoring from this point forward to ensure it's fully resolved.

3. resolved Apr 30, 2025, 10:10 PM UTC

   This incident has been resolved.

4. postmortem May 01, 2025, 03:37 PM UTC

   ### **Incident Overview** Some users experienced intermittent failures when accessing ChiliCal scheduling links while others began experiencing no access to any booking links within their organization. Affected users were presented with a **“Something Went Wrong”** error screen. The issue did not affect all links, but occurred unpredictably across different users, affecting scheduling and handoff workflows. After resolution, we found that some links were impacted further by caching of services which delayed the rollout of the fix to all users and made the impact’s duration longer than expected. ### **Timeline** * **09:00 - 09:30 UTC** – Incident reported. Users began experiencing intermittent issues with ChiliCal scheduling links. * **09:45 UTC** – Engineering team began investigation and confirmed issue reproducibility. * **10:00 UTC** – Initial fixes for the root cause were implemented as a hotfix * **10:30 UTC** – Suspected caching-related issue identified as extending the issue’s behavior * **12:45 UTC** – Team forced cache clear across relevant instances to expedite recovery. * **13:08 UTC** – Systems showed recovery. Monitoring phase initiated. * **Post-Incident** – Recommended browser cache clear for users still seeing the issue. Ongoing monitoring confirmed full resolution. ### **Root Cause** On April 28th we found that our JS framework \(NextJS\) [did not provide deterministic application builds](https://github.com/vercel/next.js/issues/63201). We deployed [a fix ](https://github.com/vercel/next.js/discussions/59493#discussioncomment-7824870)for this as an unrelated incident. While fixes were implemented to resolve the library’s update, this incident was also impacted by stale or inconsistent **application-level caching** that impacted the rendering of certain ChiliCal scheduling links. The cache did not invalidate properly after recent backend changes, resulting in outdated or broken state served to some users. ### **Resolution** The team manually forced a cache clear on affected services. This action immediately restored functionality for most users. Those still affected were likely experiencing residual issues from **local browser cache**, which was resolved with manual clearing. ### **Preventative Actions** #### Completed: * Manual cache purge on affected infrastructure. * Different technique used to handle the stop-gap measure we had in place previously which is not reliant upon framework-specific functions. * Communication to Customer Love and affected stakeholders with recommended local steps. #### Planned: * Implement automated cache invalidation tied to relevant backend deployments. * Improve detection and alerting for cache-related anomalies \(e.g., error rate spikes\). * Migration from our current JS library / framework to an alternative with broader support and flexibility. ### **Outcome / Next Steps** * Caching mechanisms need tighter integration with deployment workflows to avoid serving stale content. * Partial or intermittent failures are harder to detect and require better instrumentation. * Proactive cache management and user-friendly fallback experiences are key to resilience. * A new and more robust JS framework is needed to avoid this scenario in the first place. The ChiliCal incident on April 29, 2025, was resolved with no lasting impact, but highlighted important areas for improvement in cache handling and monitoring. Work is underway to strengthen these areas and reduce the likelihood of recurrence.

Read the full incident report →

Started Apr 17, 2025, 06:00 PM UTC· resolved Apr 17, 2025, 06:00 PM UTC

1. resolved Apr 17, 2025, 07:19 PM UTC

   Our engineers identified an issue that caused our system to stop tracking changes made to calendar invites by customers in Outlook. Fixing the bug, and triggering a re-sync of the services caused all of these previous updates to trigger finally, even if they were severely outdated or days old. Although this would have caused a flood of notifications, this would be a one-time side effect of fixing the underlying bug, and is actually indicating that those users who received the notifications are now in a good state. This could have impacted all forms of notifications including reschedules, cancelations, invitations, reminders, and so forth. It only impacted customers connected with Outlook. The issue is already resolved to prevent it from continuing, but old notifications may still be making their way through the system. Please expect some notifications may show up as a result of the above.

Read the full incident report →

Started Apr 16, 2025, 07:50 PM UTC· resolved Apr 16, 2025, 09:46 PM UTC

1. identified Apr 16, 2025, 07:50 PM UTC

   Zoom is experiencing an outage across multiple services on the zoom.us domain, which will affect new meetings booked via Chili Piper and prevent the generation of Zoom links. We are actively monitoring Zoom's status page https://status.zoom.us/incidents/pw9r9vnq5rvk to learn when their services will return.

2. resolved Apr 16, 2025, 09:46 PM UTC

   Zoom has successfully restored its services, and they are now fully operational once again. During this outage, users' Zoom integrations may have been affected, resulting in the status indicators "In Trouble" or "Disconnected. " If this situation pertains to a user, they must reconnect their Zoom account at https://fire.chilipiper.com/app/integrations. Super Admins can access the list of affected users through the following links and are advised to notify them to reconnect: - https://fire.chilipiper.com/fire/admin/users/active?zoom=InTrouble - https://fire.chilipiper.com/fire/admin/users/active?zoom=Disconnected For meetings scheduled during the Zoom outage, users may need to manually add a Zoom meeting to their calendar invitations. Additionally, the meeting reminders will not include the Zoom links manually added to the calendar event.

Read the full incident report →

Started Apr 10, 2025, 07:10 PM UTC· resolved Apr 11, 2025, 04:58 PM UTC

1. investigating Apr 10, 2025, 07:10 PM UTC

   In our Fire platform we have a mechanism that will retry an CRM action if something fails, or if a Lead doesn't exist yet, and so forth. We have heard that there are long delays in executing these updates which include updates to ownership as well as other fields and creation logic. This is a top-priority issue, and our engineers are investigating the cause as we speak. As reported, this only impacts actions that required a "retry", and is only within our Demand Conversion Platform products (Fire).

2. investigating Apr 10, 2025, 07:56 PM UTC

   We have identified a larger scope for the incident, in that all CRM Actions can be impacted by delays or failures. This can include Lead / Contact creations prior to any Event creation. As a side-effect from some of the issues, Event records have been duplicated in some instances as well. This is following an "at least once" method where it will attempt to create the event in the case that there may have been an issue, to ensure at least one event is created. This is not an intended side-effect, and we are investigating how to mitigate this aspect of the issue as well.

3. monitoring Apr 11, 2025, 03:22 PM UTC

   We've deployed a fix that will help improve the CRM action delays / failures. However, we will need to monitor traffic for at least today in order to determine if the fix ultimately did the trick. We are actively monitoring the situation, and the impact of this fix, and will act quickly if further action needs to be taken.

4. resolved Apr 11, 2025, 04:58 PM UTC

   Our tracking of the delays indicates a near zero-latency after our fix was deployed, so we are considering this incident resolved. If you continue to experience any delays in retries or issues where ownership / lead updates are not performed, please reach out to [email protected] for further assistance.

Read the full incident report →