Chili Piper Outage History

We are aware of potential delays while routing records through Distro in one of our environments. As a result, you may notice that Distro is not identifying and, consequently, routing new records. Our Engineering Team is already aware and actively working on a resolution as their top priority. This is not impacting other products such as Concierge, Chat, Handoff, and their respective routing, nor ChiliCal and its browser extension. In case of any further questions, please reach out to our Customer Love Team at [email protected]

This has now been resolved after our Engineering Team rolled back one of our services. Rules, Teams and routings are expected to be processed normally now across all products that rely on these assets. We have also already identified what caused it, so any new deployments done for this service should include a fix.

The retroactive update has been completed, and all accounts should properly be in a connected status at this time. We do have a retry mechanism that will automatically re-attempt CRM Actions in Salesforce if they had been previously unsuccessful in most cases.

Heads up: Chili Piper has officially deprecated our Legacy service from the HubSpot Marketplace. This legacy app was retired last year and is no longer in use. There is no impact to any functionality or connectivity on the current Fire / Demand Conversion Platform. However, the removal may have triggered some automated emails that lack context. You may have received messages such as: Chili Piper Scheduling deleted by provider Log in with HubSpot deleted by provider Chili Piper - Legacy deleted by provider These notifications can be safely ignored with no further action required. Note: You may have received these emails even if you're currently on Salesforce, if your account was ever connected to HubSpot in the past. This does not mean you are currently connected to HubSpot.

This incident has been resolved. Internal action was taken to ensure that this specific failure point in the chat service cannot again in the future.

We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have ongoing efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such scenarios.

We are now considering this issue resolved, as we're seeing better results. The issue lasted from 17:30 to 19:20 UTC and impacted email notifications such as meeting confirmations emails, record assignment emails, and Reminders for Chili Piper users. The internal service responsible for such notifications had some configuration issues, which were already mitigated. Our teams are already working on lessons learned and how we can avoid this from happening in the future. The issue has not affected the ability to book meetings across products, including our extension, assigning records via Distro, and prospects from receiving meeting invites and calendar event confirmations. If you have any further questions, please reach out to [email protected]

We've completed the necessary configuration changes and access to resources has been restored for impacted legacy customers. A thorough review of this configuration and what occurred will be completed to ensure it remains stable.

Our team identified that the issue was related to a deployment of our integration services at approximately 15:32 EST (20:32 UTC) for an incident span of approximately 2 hours. We rolled back the deployment and this resolved the issue, and we saw error rates drop to zero. It is likely that any meetings booked during this time may have been impacted where no Google calendar event was created in its place. We strongly recommend manually creating these events if they're identified in the meantime rather than waiting on a replay action (if this is possible from our end) as this will ensure that all parties have the most up-to-date calendar appearance, and ensures prospects receive meeting invites. Apologies for the disruption this may have caused.

We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have already started coordinating efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such occurrences.

Our teams reported that the consumption should be back to normal. You should notice previously delayed Salesforce actions being marked as Successful, and new ones should be processed within our standards. Over the past few days, the system reached a critical load level that exposed a performance bottleneck in our Salesforce data processing. After thorough analysis, we identified and resolved the issue, restoring normal operation and improving overall throughput. We are continuously improving our systems and infrastructure to process customer data more efficiently and reliably. As data volumes increase, we regularly adjust configurations and upgrade underlying tools and libraries to maintain stable performance. If you still notice any unexpected behavior, please report it to [email protected] for further analysis.

We are seeing services recover, so are marking this "Resolved". For more in-depth information about the incident and its current status please check the following status updates provided by Microsoft: https://azure.status.microsoft/en-us/status https://status.cloud.microsoft/m365

Distro is back to an operational state. The issue appears to have been caused by an influx of triggered events which exceeded the maximum load possible. This caused the event processing time to grow exponentially for all customers. We are correcting for this in several ways including improving current services to accommodate the increased load, and also improving the strategy for how events are processed. While some of these improvements are ongoing, the incident is considered resolved.

The issue appears to have been related to our payment processor, whose own API had an outage. As we access this API for various admin center and workspace related data such as licenses and even product access, this caused the pages to become unresponsive and crash. We have observed the related API self-healing, and access to workspace and admin center have both been restored, however as this service is outside of our ecosystem we are monitoring now to confirm that everything is back to being stable. We are going to examine how we can better handle (gracefully) third-party billing and payment outages such as this in order to prevent interruption to workflows, while also maintaining product license access in the future.

We are in the process of moving to a new billing / payment processor this morning. As a result, you may have temporarily lost the ability to login for a few seconds up to a few minutes, however this should not impact any workflows, routing, or bookings. If you notice the inability to login, please wait 5-10 minutes and try again. If you continue to experience issues, feel free to reach out to [email protected] so that our Customer Love team can take a closer look for you! Apologies in advance for the inconvenience this may have caused.

Chili Piper experienced a full platform outage related to services dependent on Kafka. These services were unable to send or receive data during this time which essentially caused all products to go offline and lose access. The outage was detected immediately by our internal alerting system and our on-call Site Reliability Engineering team began working to correct the issue. We raised the escalation with the Confluent Cloud team through all available channels. Approximately 1 hour and 50 minutes after the incident began, Confluent confirmed that their internal teams were actively engaged in resolving the issue with additional escalation within Confluent prioritized at this time. To mitigate the issue, we created a new subscription through their system in order to restore access. Service was fully restored at 19:12 UTC (3:02pm EDT). No existing data was lost, but no new data was processed during this time which means no routes or meetings could be booked through Chili Piper.

Our team received reports of degraded performance defined as, and ranging from, slowness loading Concierge routes; pages generally loading slowly; or cases where purchased licenses weren't being immediately updated. We identified the most significant root cause as an issue with a third-party library used in our backoffice infrastructure which was causing our services to become "poisoned". This means that when the service faced a critical state, it did not terminate cleanly and instead kept running while sick -- resulting in slowness. Our fix has been two-fold to begin with: 1. We have adjusted the library to a state where the issue does not exist. 2. We will monitor services for similar poisoned status, and if experienced, these services will be restarted so that they can recover properly. We have already deployed fixes for this and have had confirmation that there is a marked improvement in load times. We will continue to monitor this moving forward, making adjustments as needed.

Our engineers identified an issue that caused our system to stop tracking changes made to calendar invites by customers in Outlook. Fixing the bug, and triggering a re-sync of the services caused all of these previous updates to trigger finally, even if they were severely outdated or days old. Although this would have caused a flood of notifications, this would be a one-time side effect of fixing the underlying bug, and is actually indicating that those users who received the notifications are now in a good state. This could have impacted all forms of notifications including reschedules, cancelations, invitations, reminders, and so forth. It only impacted customers connected with Outlook. The issue is already resolved to prevent it from continuing, but old notifications may still be making their way through the system. Please expect some notifications may show up as a result of the above.

Zoom has successfully restored its services, and they are now fully operational once again. During this outage, users' Zoom integrations may have been affected, resulting in the status indicators "In Trouble" or "Disconnected. " If this situation pertains to a user, they must reconnect their Zoom account at https://fire.chilipiper.com/app/integrations. Super Admins can access the list of affected users through the following links and are advised to notify them to reconnect: - https://fire.chilipiper.com/fire/admin/users/active?zoom=InTrouble - https://fire.chilipiper.com/fire/admin/users/active?zoom=Disconnected For meetings scheduled during the Zoom outage, users may need to manually add a Zoom meeting to their calendar invitations. Additionally, the meeting reminders will not include the Zoom links manually added to the calendar event.

Our tracking of the delays indicates a near zero-latency after our fix was deployed, so we are considering this incident resolved. If you continue to experience any delays in retries or issues where ownership / lead updates are not performed, please reach out to [email protected] for further assistance.

We identified a third-party script was causing the admin center page to hang. We have blocked this script, and pages now load as expected. No further impact is expected as a result of blocking this script.

Our team was made aware of a wide-spread issue involving the ChiliCal scheduler for Handoff not loading routers, 404 pages being experienced, errors indicating a connection cannot be established, and perpetual loading of Concierge routes. This issue was a result of unplanned database maintenance which triggered the temporary outage. This does not appear to have impacted our legacy / classic services at this time. We are seeing all services up and running after a brief approximately 20 minute outage.