Chili Piper Outage History

We received positive feedback from affected customers after Gong's team applied the fix, so we consider this resolved. Still, please email [email protected] if you notice anything unexpected.

HubSpot has now confirmed a fix to the issue. We have also confirmed restoration of services in Chili Piper.

We are aware of potential delays while routing records through Distro in one of our environments. As a result, you may notice that Distro is not identifying and, consequently, routing new records. Our Engineering Team is already aware and actively working on a resolution as their top priority. This is not impacting other products such as Concierge, Chat, Handoff, and their respective routing, nor ChiliCal and its browser extension. In case of any further questions, please reach out to our Customer Love Team at [email protected]

This has now been resolved after our Engineering Team rolled back one of our services. Rules, Teams and routings are expected to be processed normally now across all products that rely on these assets. We have also already identified what caused it, so any new deployments done for this service should include a fix.

The retroactive update has been completed, and all accounts should properly be in a connected status at this time. We do have a retry mechanism that will automatically re-attempt CRM Actions in Salesforce if they had been previously unsuccessful in most cases.

Heads up: Chili Piper has officially deprecated our Legacy service from the HubSpot Marketplace. This legacy app was retired last year and is no longer in use. There is no impact to any functionality or connectivity on the current Fire / Demand Conversion Platform. However, the removal may have triggered some automated emails that lack context. You may have received messages such as: Chili Piper Scheduling deleted by provider Log in with HubSpot deleted by provider Chili Piper - Legacy deleted by provider These notifications can be safely ignored with no further action required. Note: You may have received these emails even if you're currently on Salesforce, if your account was ever connected to HubSpot in the past. This does not mean you are currently connected to HubSpot.

This incident has been resolved. Internal action was taken to ensure that this specific failure point in the chat service cannot again in the future.

We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have ongoing efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such scenarios.

We are now considering this issue resolved, as we're seeing better results. The issue lasted from 17:30 to 19:20 UTC and impacted email notifications such as meeting confirmations emails, record assignment emails, and Reminders for Chili Piper users. The internal service responsible for such notifications had some configuration issues, which were already mitigated. Our teams are already working on lessons learned and how we can avoid this from happening in the future. The issue has not affected the ability to book meetings across products, including our extension, assigning records via Distro, and prospects from receiving meeting invites and calendar event confirmations. If you have any further questions, please reach out to [email protected]

We've completed the necessary configuration changes and access to resources has been restored for impacted legacy customers. A thorough review of this configuration and what occurred will be completed to ensure it remains stable.

Our team identified that the issue was related to a deployment of our integration services at approximately 15:32 EST (20:32 UTC) for an incident span of approximately 2 hours. We rolled back the deployment and this resolved the issue, and we saw error rates drop to zero. It is likely that any meetings booked during this time may have been impacted where no Google calendar event was created in its place. We strongly recommend manually creating these events if they're identified in the meantime rather than waiting on a replay action (if this is possible from our end) as this will ensure that all parties have the most up-to-date calendar appearance, and ensures prospects receive meeting invites. Apologies for the disruption this may have caused.

We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have already started coordinating efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such occurrences.

Our teams reported that the consumption should be back to normal. You should notice previously delayed Salesforce actions being marked as Successful, and new ones should be processed within our standards. Over the past few days, the system reached a critical load level that exposed a performance bottleneck in our Salesforce data processing. After thorough analysis, we identified and resolved the issue, restoring normal operation and improving overall throughput. We are continuously improving our systems and infrastructure to process customer data more efficiently and reliably. As data volumes increase, we regularly adjust configurations and upgrade underlying tools and libraries to maintain stable performance. If you still notice any unexpected behavior, please report it to [email protected] for further analysis.

We are seeing services recover, so are marking this "Resolved". For more in-depth information about the incident and its current status please check the following status updates provided by Microsoft: https://azure.status.microsoft/en-us/status https://status.cloud.microsoft/m365