Kustomer Outage History

Started Oct 20, 2025, 01:36 PM UTC· resolved Oct 20, 2025, 10:54 PM UTC

1. investigating Oct 20, 2025, 01:36 PM UTC

   Kustomer has identified the disruption impacting platform services as a result of the significant widespread outage at Amazon Web Services (AWS). Further details and real-time updates are also available in https://health.aws.amazon.com/health/status Our teams are actively working to mitigate the impact and restore services as quickly as possible, while maintaining close coordination with AWS as they work to resolve the underlying issue. We’ll continue to post updates here as systems recover and more information becomes available. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any queries.

2. investigating Oct 20, 2025, 01:40 PM UTC

   We are continuing to investigate this issue.

3. investigating Oct 20, 2025, 03:28 PM UTC

   Kustomer is aware of an event affecting causing service disruption and latency due to the AWS incident. Our team is currently working to restore full service and are seeing some improvements to latency. We’ll continue to post updates here as systems recover and more information becomes available. Please reach out to Kustomer Support at [email protected] for any queries.

4. investigating Oct 20, 2025, 04:16 PM UTC

   Kustomer is aware of an event affecting causing service disruption, latency, and potential login issues due to the AWS incident. Our team is currently working to restore full service and are seeing some improvements to latency. We’ll continue to post updates here as systems recover and more information becomes available. Please reach out to Kustomer Support at [email protected] for any queries.

5. investigating Oct 20, 2025, 05:55 PM UTC

   Kustomer is aware of an event affecting causing service disruption, latency, and potential login issues due to the AWS incident. Our team is continuing to work to restore full service. We’ll continue to post updates here as systems recover and more information becomes available. Please reach out to Kustomer Support at [email protected] for any queries.

6. identified Oct 20, 2025, 08:40 PM UTC

   Kustomer is aware of an event causing service disruption and latency due to the AWS incident. Our team is seeing significant improvements across the platform. Login functionality has been restored, and services are steadily returning to normal capacity. We’ll continue to monitor performance closely to ensure full stability We’ll continue to post updates here as systems recover and more information becomes available. Please reach out to Kustomer Support at [email protected] for any queries.

7. monitoring Oct 20, 2025, 09:11 PM UTC

   We’re continuing to monitor the platform. Performance and services continue to improve, and overall stability is trending positively. We’re also redriving affected events to ensure delivery and data consistency. Please reach out to Kustomer Support at [email protected] for any questions or concerns. Another update will be provided within 30 minutes.

8. monitoring Oct 20, 2025, 09:58 PM UTC

   We are continuing to monitor the platform as performance and services steadily improve. The redrive of affected events is nearly complete, ensuring full delivery and data integrity. For any questions or concerns, please contact Kustomer Support at [email protected]. Another update will be provided within 30 minutes.

9. monitoring Oct 20, 2025, 10:41 PM UTC

   We continue to monitor the platform as performance and services steadily improve. The redrive of affected events has been completed, ensuring full delivery and data integrity. For any questions or concerns, please contact Kustomer Support at [email protected]. Another update will be provided within 30 minutes.

10. resolved Oct 20, 2025, 10:54 PM UTC

    After careful monitoring, our team has confirmed that the platform and all services have returned to normal operation. All affected areas are fully restored, and performance remains stable. We will continue to monitor to ensure ongoing reliability. For any questions or concerns, please contact Kustomer Support at [email protected].

11. postmortem Oct 29, 2025, 05:18 PM UTC

    # **Summary** On October 20, 2025, Kustomer experienced a significant service disruption affecting all customers on our platform. The incident was triggered by a major AWS service disruption in the us-east-1 region that impacted ~140 AWS services, including core infrastructure components our platform depends on. The incident occurred in two distinct phases: * Phase 1 \(3:00 AM - 6:30 AM EDT\): Initial AWS service disruption causing widespread connectivity issues and elevated error rates across our platform. Services began recovering as AWS addressed the underlying DNS resolution issues. * Phase 2 \(9:20 AM - 5:00 PM EDT\): Secondary wave of issues, including a critical authentication problem that prevented users from logging into the platform from 11:47 AM to 4:30 PM EDT. This simultaneously overlaps with the disruption to AWS EC2 service on October 20 2:48am EDT to 4:50pm EDT. This authentication issue was caused by our disaster recovery automation during our attempts to establish failover capabilities while the AWS EC2 disruption was ongoing. Full platform recovery was achieved by 5:00 PM EDT on October 20, with final cleanup operations completed by 7:00 PM EDT. A remaining issue with AIC and AIR observability infrastructure was fully resolved by 11:45 AM EDT on October 21. **Root Cause** **Primary Cause: AWS Regional Service Disruption** The incident originated from a widespread AWS service disruption in the us-east-1 region. According to AWS, the root cause was a DNS resolution failure for internal service endpoints, specifically affecting DynamoDB regional endpoints. This DNS issue cascaded into AWS's EC2 internal launch system, creating widespread connectivity problems across the region. **Secondary Cause: Disaster Recovery Automation Issue** During our disaster recovery response, an infrastructure configuration issue emerged that prevented proper authentication for approximately 4.5 hours. When disaster recovery preparations were initiated at 11:25 AM EDT, our infrastructure automation discovered that primary and secondary region configurations shared the same management context. As the automation began provisioning resources in the secondary region, it simultaneously triggered scaling changes in the primary region's authentication service, reducing it to a minimal operational state. Under normal circumstances, an automated deployment would have immediately corrected this configuration. However, this deployment failed due to the ongoing AWS service disruptions, leaving the authentication service's load balancer in an inconsistent state—pointing to a configuration with insufficient capacity while healthy instances remained unreachable. This was an edge case that required the specific combination of: \(1\) initiating disaster recovery operations, \(2\) the infrastructure management coupling between regions that were partially operational \(us-east-1 was significantly degraded but remained operational\), and \(3\) simultaneous AWS service failures in us-east-1 preventing the standard recovery mechanism from completing. Had us-east-1 been completely offline, there would not have been extra consideration required for operations in that region. # **Timeline** _All times in Eastern Daylight Time_ ### **Initial Service Disruption Phase** **Oct 20, 2025** **3:11 AM** - Multiple alerts triggered indicating elevated error rates across platform services. Kustomer’s Statuspage was accessible but oncall responders could not authenticate successfully to provide updates. **3:18 AM** - Incident response initiated; engineers begin investigation **4:25 AM** - Confirmed correlation with AWS service disruptions in us-east-1 region **5:01 AM** - AWS identifies root cause of DNS resolution failures **5:27 AM - 6:03 AM** - AWS reports significant recovery; Kustomer platform functionality observed to be restored. Access to Kustomer’s Statuspage was restored at 5:37am. **6:30 AM** - Initial recovery verified; platform communications channels tested successfully. Kustomer platform is verified fully functional ### **Secondary Service Disruption Phase** **8:55 AM** - New wave of elevated error rates detected across multiple services **9:42 AM** - AWS announces mitigation in progress with EC2 throttling. Additional compute is significantly limited and affects autoscaling operations. **10:11 AM** - Severity escalated; separate incident channel created **10:40 AM** - Decision made to initiate disaster recovery preparations in us-east-2 region as contingency **11:02 AM** - Engineers begin provisioning infrastructure in secondary region **11:46 AM -** Due to authentication issues, Kustomer Technical Support began responding to client inquiries directly through email to continue correspondence **11:47 AM** - **3:48 PM** Multiple customer reports of authentication failures; disaster recovery efforts are taking place during this time. At 3:48pm, the root cause for authentication errors: load balancer configuration prematurely routing traffic to secondary region **4:03 PM** - AWS reports ongoing recovery across most services **4:15 PM** - Load balancer configuration corrected **4:30 PM** - Customer authentication restored; users able to access platform **5:00 PM** - API traffic normalized; error rates return to baseline **5:48 PM** - AWS announces full recovery of regional services in us-east-1 **6:01 PM - 6:35 PM** - Queues redriven to recover delayed operations **7:00 PM** - Secondary region infrastructure fully prepared for future failover needs ### **Extended Recovery** **8:40 PM** - Observability logs for AIC and AIR feature remain impacted due to Opensearch Serverless recovery mitigations implemented by AWS **Oct 21, 2025** **11:45 AM** - Full recovery of Opensearch Serverless which restores AIC and AIR observability ‌ **Lessons/Improvements** ### **What Went Well** **Disaster recovery readiness**: Our recent disaster recovery exercise was most recently conducted in July 2025. The updated documentation proved valuable during the incident. The team was able to reference established runbooks and procedures, even as we encountered unexpected challenges. **Cross-team coordination**: Engineers across multiple teams collaborated effectively to simultaneously address recovery in the primary region while preparing failover capabilities in the secondary region. **Iterative improvement**: The team identified and implemented fixes to disaster recovery automation in real-time, improving our processes even during the incident. **Customer communication**: Our status page experienced authentication issues during the incident, limiting our ability to communicate with customers. Additionally, Kustomer’s own access to the platform was impacted from this service disruption. Our tech support and customer success teams were able to keep customers updated by reaching out via the email channel. ### **Areas for Improvement** **Earlier detection of infrastructure anomalies**: The authentication service load balancer inconsistency was observed earlier in the incident but wasn't immediately prioritized for investigation amid numerous other AWS-related issues. We're implementing additional health checks and automated detection to ensure redundant and proactive alerting for this critical functionality as well as updating runbooks to reinforce the importance of investigating the minor anomalies during major incidents. **Status page reliability**: The ability to keep customers updated to the latest status about the Kustomer platform is critical to our operations and maintaining trust. We are exploring options to improve the reliability of this tool. ### **Planned Improvements** **Immediate Actions \(Completed or In Progress\):** * Exploring redundant customer communication channels beyond our primary status page * Separated infrastructure management for secondary region to enable independent deployment and faster disaster recovery * Adding safeguards to disaster recovery automation and race conditions to prevent unintended traffic shifts and validate target health before routing changes **Strategic Initiatives:** * Conducting more frequent disaster recovery exercises to validate our failover procedures under realistic conditions * Developing faster, more reliable automation scripts and orchestration tools for disaster recovery operations to reduce manual intervention and accelerate failover execution * Establishing disaster recovery readiness as a mandatory requirement in our engineering standards, ensuring all new features and services are designed with multi-region capabilities and tested regularly for failover scenarios ### **Commitment to Reliability** This incident reinforced our commitment to building a resilient platform. While we cannot prevent cloud provider service disruptions, we can minimize their impact through better disaster recovery capabilities, more robust automation safeguards, and regular testing of our failover procedures. We recognize that our customers depend on Kustomer for critical business operations, and we take that responsibility seriously. The improvements outlined above represent concrete steps toward faster recovery times and reduced impact from future infrastructure disruptions. We appreciate the patience of our customers during this incident and remain committed to continuous improvement of our platform's reliability and resilience.

Read the full incident report →

Started Oct 20, 2025, 09:48 AM UTC· resolved Oct 20, 2025, 10:29 AM UTC

1. identified Oct 20, 2025, 09:48 AM UTC

   Kustomer has identified the disruption impacting platform services as a result of the significant widespread outage at Amazon Web Services (AWS). Further details and real-time updates are also available in https://health.aws.amazon.com/health/status Our teams are actively working to mitigate the impact and restore services as quickly as possible, while maintaining close coordination with AWS as they work to resolve the underlying issue. We’ll continue to post updates here as systems recover and more information becomes available. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any queries.

2. monitoring Oct 20, 2025, 10:18 AM UTC

   Kustomer platform services should now be operational, though some Business Rules items may remain delayed while our teams work to retrieve and process any queued events. AWS has reported significant signs of recovery across affected services and continues to work toward full resolution. We continue to closely monitor platform performance as AWS progresses with their own recovery efforts and keep sharing updates here as systems stabilize further. Please expect additional updates within the next 30 minutes and if you have any questions or continue to notice delays, please reach out to Kustomer Support at [email protected].

3. resolved Oct 20, 2025, 10:29 AM UTC

   After careful monitoring, our team has determined that Kustomer platform services are now fully operational and the queued Business Rules items have been successfully processed. Thank you for your patience and understanding during this event. If you experience any residual issues or have questions, please reach out to Kustomer Support at [email protected].

4. postmortem Oct 29, 2025, 05:15 PM UTC

   # **Summary** On October 20, 2025, Kustomer experienced a significant service disruption affecting all customers on our platform. The incident was triggered by a major AWS service disruption in the us-east-1 region that impacted ~140 AWS services, including core infrastructure components our platform depends on. The incident occurred in two distinct phases: * Phase 1 \(3:00 AM - 6:30 AM EDT\): Initial AWS service disruption causing widespread connectivity issues and elevated error rates across our platform. Services began recovering as AWS addressed the underlying DNS resolution issues. * Phase 2 \(9:20 AM - 5:00 PM EDT\): Secondary wave of issues, including a critical authentication problem that prevented users from logging into the platform from 11:47 AM to 4:30 PM EDT. This simultaneously overlaps with the disruption to AWS EC2 service on October 20 2:48am EDT to 4:50pm EDT. This authentication issue was caused by our disaster recovery automation during our attempts to establish failover capabilities while the AWS EC2 disruption was ongoing. Full platform recovery was achieved by 5:00 PM EDT on October 20, with final cleanup operations completed by 7:00 PM EDT. A remaining issue with AIC and AIR observability infrastructure was fully resolved by 11:45 AM EDT on October 21. **Root Cause** **Primary Cause: AWS Regional Service Disruption** The incident originated from a widespread AWS service disruption in the us-east-1 region. According to AWS, the root cause was a DNS resolution failure for internal service endpoints, specifically affecting DynamoDB regional endpoints. This DNS issue cascaded into AWS's EC2 internal launch system, creating widespread connectivity problems across the region. **Secondary Cause: Disaster Recovery Automation Issue** During our disaster recovery response, an infrastructure configuration issue emerged that prevented proper authentication for approximately 4.5 hours. When disaster recovery preparations were initiated at 11:25 AM EDT, our infrastructure automation discovered that primary and secondary region configurations shared the same management context. As the automation began provisioning resources in the secondary region, it simultaneously triggered scaling changes in the primary region's authentication service, reducing it to a minimal operational state. Under normal circumstances, an automated deployment would have immediately corrected this configuration. However, this deployment failed due to the ongoing AWS service disruptions, leaving the authentication service's load balancer in an inconsistent state—pointing to a configuration with insufficient capacity while healthy instances remained unreachable. This was an edge case that required the specific combination of: \(1\) initiating disaster recovery operations, \(2\) the infrastructure management coupling between regions that were partially operational \(us-east-1 was significantly degraded but remained operational\), and \(3\) simultaneous AWS service failures in us-east-1 preventing the standard recovery mechanism from completing. Had us-east-1 been completely offline, there would not have been extra consideration required for operations in that region. # **Timeline** _All times in Eastern Daylight Time_ ### **Initial Service Disruption Phase** **Oct 20, 2025** **3:11 AM** - Multiple alerts triggered indicating elevated error rates across platform services. Kustomer’s Statuspage was accessible but oncall responders could not authenticate successfully to provide updates. **3:18 AM** - Incident response initiated; engineers begin investigation **4:25 AM** - Confirmed correlation with AWS service disruptions in us-east-1 region **5:01 AM** - AWS identifies root cause of DNS resolution failures **5:27 AM - 6:03 AM** - AWS reports significant recovery; Kustomer platform functionality observed to be restored. Access to Kustomer’s Statuspage was restored at 5:37am. **6:30 AM** - Initial recovery verified; platform communications channels tested successfully. Kustomer platform is verified fully functional ### **Secondary Service Disruption Phase** **8:55 AM** - New wave of elevated error rates detected across multiple services **9:42 AM** - AWS announces mitigation in progress with EC2 throttling. Additional compute is significantly limited and affects autoscaling operations. **10:11 AM** - Severity escalated; separate incident channel created **10:40 AM** - Decision made to initiate disaster recovery preparations in us-east-2 region as contingency **11:02 AM** - Engineers begin provisioning infrastructure in secondary region **11:46 AM -** Due to authentication issues, Kustomer Technical Support began responding to client inquiries directly through email to continue correspondence **11:47 AM** - **3:48 PM** Multiple customer reports of authentication failures; disaster recovery efforts are taking place during this time. At 3:48pm, the root cause for authentication errors: load balancer configuration prematurely routing traffic to secondary region **4:03 PM** - AWS reports ongoing recovery across most services **4:15 PM** - Load balancer configuration corrected **4:30 PM** - Customer authentication restored; users able to access platform **5:00 PM** - API traffic normalized; error rates return to baseline **5:48 PM** - AWS announces full recovery of regional services in us-east-1 **6:01 PM - 6:35 PM** - Queues redriven to recover delayed operations **7:00 PM** - Secondary region infrastructure fully prepared for future failover needs ### **Extended Recovery** **8:40 PM** - Observability logs for AIC and AIR feature remain impacted due to Opensearch Serverless recovery mitigations implemented by AWS **Oct 21, 2025** **11:45 AM** - Full recovery of Opensearch Serverless which restores AIC and AIR observability ‌ **Lessons/Improvements** ### **What Went Well** **Disaster recovery readiness**: Our recent disaster recovery exercise was most recently conducted in July 2025. The updated documentation proved valuable during the incident. The team was able to reference established runbooks and procedures, even as we encountered unexpected challenges. **Cross-team coordination**: Engineers across multiple teams collaborated effectively to simultaneously address recovery in the primary region while preparing failover capabilities in the secondary region. **Iterative improvement**: The team identified and implemented fixes to disaster recovery automation in real-time, improving our processes even during the incident. **Customer communication**: Our status page experienced authentication issues during the incident, limiting our ability to communicate with customers. Additionally, Kustomer’s own access to the platform was impacted from this service disruption. Our tech support and customer success teams were able to keep customers updated by reaching out via the email channel. ### **Areas for Improvement** **Earlier detection of infrastructure anomalies**: The authentication service load balancer inconsistency was observed earlier in the incident but wasn't immediately prioritized for investigation amid numerous other AWS-related issues. We're implementing additional health checks and automated detection to ensure redundant and proactive alerting for this critical functionality as well as updating runbooks to reinforce the importance of investigating the minor anomalies during major incidents. **Status page reliability**: The ability to keep customers updated to the latest status about the Kustomer platform is critical to our operations and maintaining trust. We are exploring options to improve the reliability of this tool. ### **Planned Improvements** **Immediate Actions \(Completed or In Progress\):** * Exploring redundant customer communication channels beyond our primary status page * Separated infrastructure management for secondary region to enable independent deployment and faster disaster recovery * Adding safeguards to disaster recovery automation and race conditions to prevent unintended traffic shifts and validate target health before routing changes **Strategic Initiatives:** * Conducting more frequent disaster recovery exercises to validate our failover procedures under realistic conditions * Developing faster, more reliable automation scripts and orchestration tools for disaster recovery operations to reduce manual intervention and accelerate failover execution * Establishing disaster recovery readiness as a mandatory requirement in our engineering standards, ensuring all new features and services are designed with multi-region capabilities and tested regularly for failover scenarios ### **Commitment to Reliability** This incident reinforced our commitment to building a resilient platform. While we cannot prevent cloud provider service disruptions, we can minimize their impact through better disaster recovery capabilities, more robust automation safeguards, and regular testing of our failover procedures. We recognize that our customers depend on Kustomer for critical business operations, and we take that responsibility seriously. The improvements outlined above represent concrete steps toward faster recovery times and reduced impact from future infrastructure disruptions. We appreciate the patience of our customers during this incident and remain committed to continuous improvement of our platform's reliability and resilience.

Read the full incident report →

Started Oct 16, 2025, 02:42 PM UTC· resolved Oct 16, 2025, 06:19 PM UTC

1. investigating Oct 16, 2025, 02:42 PM UTC

   Kustomer is currently aware of an event affecting inbound messaging on Facebook Messenger that may cause inbound Facebook Messenger messages not to appear in instances in our Prod1 environment. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. identified Oct 16, 2025, 03:10 PM UTC

   Kustomer has identified the underlying cause of the issue that was preventing inbound Facebook Messenger messages from being created in instances in our Prod1 environment. Our team is implementing a resolution. We’ll share another update within 30 minutes. For any additional questions or updates, please contact Support at [email protected].

3. monitoring Oct 16, 2025, 03:19 PM UTC

   Kustomer has implemented an update to address an event affecting Facebook Messages that caused them to not ingest. Facebook messages received from 11:30pm ET, October 15th onward will begin to ingest over the next hour. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. monitoring Oct 16, 2025, 03:58 PM UTC

   Kustomer has implemented an update to address an event affecting Facebook Messages that caused them to not ingest. Facebook messages received from 11:30pm ET, October 15th onward will begin to ingest over the next hour. Our team is continuing to monitor the update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. monitoring Oct 16, 2025, 04:27 PM UTC

   Kustomer has implemented an update to address an event affecting Facebook Messages that caused them to not ingest. Facebook messages received from 11:30pm ET, October 15th onward will begin to ingest over the next hour. We are continuing to monitor the ingestion of the impacted Facebook messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. monitoring Oct 16, 2025, 05:10 PM UTC

   Kustomer has implemented an update to address an event affecting Facebook Messages that caused them to not ingest. Facebook messages received from 11:30pm ET, October 15th onward will begin to ingest over the next hour. We are continuing to monitor the ingestion of the impacted Facebook messages. We expect all messages to be ingested in about an hour. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. monitoring Oct 16, 2025, 05:42 PM UTC

   Kustomer has implemented an update to address an event affecting Facebook Messages that caused them to not ingest. Facebook messages received from 11:30pm ET, October 15th onward will continue to ingest. We are continuing to monitor the ingestion of the impacted Facebook messages. We expect all messages to be ingested in about 30 minutes. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

8. resolved Oct 16, 2025, 06:19 PM UTC

   Kustomer has resolved an event affecting Facebook Messages that caused messages to fail to ingest. To resolve this issue, our team has released an update. After careful monitoring, our team has determined that all affected areas are now fully restored, and all impacted messages have been ingested. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

Read the full incident report →

Started Oct 08, 2025, 02:09 PM UTC· resolved Oct 08, 2025, 03:16 PM UTC

1. investigating Oct 08, 2025, 02:09 PM UTC

   Kustomer is aware of an event affecting workflows and routing that may cause platform delays. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. monitoring Oct 08, 2025, 02:31 PM UTC

   Kustomer has implemented an update to address an event affecting workflows, business rules, and routing in Prod 2 that caused platform delays. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. resolved Oct 08, 2025, 03:16 PM UTC

   Kustomer has resolved an event affecting Prod 2 that may have caused issues with routing and workflows. To resolve this issue, our team has released an update. After careful monitoring, our team has determined that our systems are now fully restored, and has completed a redrive. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. postmortem Oct 16, 2025, 04:40 PM UTC

   # **Summary** On October 8, 2025, Kustomer experienced a period of degraded performance in our prod2 EU environment that affected business rules, workflow execution and routing responsiveness. The elevated load stemmed from an internal feedback loop in workflow processing, which caused increased memory usage and instability in a downstream automation component. Systems serving automation traffic became rate-limited as they approached memory thresholds, resulting in delays across related services. The issue was mitigated by reducing processing throughput and rebalancing service traffic, allowing impacted systems to stabilize. Full recovery was confirmed later that day. # **Impact** * **Duration:** ~2.5 hours of degraded performance * **Scope:** Production region prod-2 \(EU\) * **Customer Impact:** * Minor latency observed across multiple customer organizations # **Next Steps** While existing safeguards are already in place, we are implementing additional safety measures to better detect anomalous workflow patterns and further enhance the stability of our automation systems under high-load scenarios. We’re also investing in expanded observability to improve early detection of similar issues in the future.

Read the full incident report →

Started Sep 13, 2025, 01:11 AM UTC· resolved Sep 13, 2025, 11:04 PM UTC

1. investigating Sep 13, 2025, 01:11 AM UTC

   Kustomer is aware of an event affecting Facebook and Instagram Authentication that may cause Pages to fail to display in their selection drop downs in the FB/IG admin UIs while authenticated. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] or via chat if you have additional questions or concerns.

2. investigating Sep 13, 2025, 01:42 AM UTC

   Kustomer is aware of an event affecting Facebook and Instagram Authentication that may cause Pages to fail to display in their selection drop downs in the FB/IG admin UIs while authenticated. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] or via chat if you have additional questions or concerns.

3. investigating Sep 13, 2025, 02:33 AM UTC

   Kustomer is aware of an event affecting Facebook and Instagram Authentication that may cause Pages to fail to display in their selection drop downs in the FB/IG admin UIs while authenticated. Our team is actively investigating the root cause of this issue and working toward a fix. We’ll share another update within 30 minutes. If you have any questions, please contact Kustomer Support at [email protected] or through chat.

4. investigating Sep 13, 2025, 03:32 AM UTC

   Kustomer is aware of an event affecting Facebook and Instagram Authentication that may cause Pages to fail to display in their selection drop-downs in the FB/IG admin UIs while authenticated. Our team is actively investigating the root cause and working toward a fix. We’ll provide updates as more information becomes available. If you have any questions, please contact Kustomer Support at [email protected] or through chat.

5. investigating Sep 13, 2025, 03:50 PM UTC

   Kustomer is aware of an event affecting Facebook and Instagram Authentication that may cause Pages to fail to display in their selection drop-downs in the FB/IG admin UIs while authenticated. Our team is actively investigating potential root causes. Work is ongoing and we will continue to share updates as progress is made. If you have any questions, please contact Kustomer Support at [email protected] or through chat.

6. resolved Sep 13, 2025, 11:04 PM UTC

   Kustomer has resolved an event affecting Facebook and Instagram Authentication that caused Pages to fail to display in selection drop-downs within the FB/IG admin UIs while authenticated. The issue can be addressed through a manual workaround that adds the missing reducers for Facebook. After careful monitoring, our team has confirmed that all affected areas are now fully restored when using this manual workaround. Please reach out to Kustomer Support at [email protected] or via chat if you have additional questions or concerns.

7. postmortem Sep 29, 2025, 06:07 PM UTC

   # **Summary** Between **September 10, 2025, and September 15, 2025**, customers were unable to log in or reauthenticate via **Facebook, Instagram, and WhatsApp** social channels. Other login methods \(such as email and password\) were unaffected. The root cause was a configuration update to our web security headers that inadvertently blocked required authentication flows. This prevented the login page from loading inside the hidden browser frames these providers use during OAuth. All services were restored by September 15, 2025, following a configuration correction. # **Root Cause** On **September 10, 2025**, a security update was deployed that added restrictive browser directives \(X-Frame-Options: DENY and frame-ancestors 'none'\) across our login endpoints. While these headers improved protection against clickjacking, they also prevented OAuth flows from loading our login page within an , which is required for Facebook, Instagram, and WhatsApp authentication. As a result, customers who needed to reauthenticate tokens for these channels were blocked until the issue was identified and corrected. # **Timeline** * **Sep 10, 2025, 10:51 AM \(EST\):** Security header update deployed. * **Sep 12, 2025, 10:09 AM \(EST\):** First customer reports issues adding Facebook/Instagram pages. * **Sep 12, 2025, 3:04 PM \(EST\):** Incident escalated to Priority 1 as multiple customers were impacted. * **Sep 13–14, 2025:** Teams investigated potential fixes and implemented manual customer workarounds. * **Sep 15, 2025, 2:48 PM \(EST\):** Configuration update deployed to allow necessary domains while removing the conflicting X-Frame-Options. Authentication fully restored. # **Lessons/Improvements** * **Monitoring & Escalation:** The issue could have been escalated more quickly; clearer guidance is being put in place to ensure similar incidents are prioritized earlier. * **Testing Gaps:** Our local development and staging environments did not apply the same header rules as production, which made reproducing the issue difficult. We are updating our environments to align with production behavior. * **Documentation:** Notes on header restrictions were not well-documented across teams. We are improving visibility of known constraints and best practices.

Read the full incident report →

Started Sep 05, 2025, 10:31 PM UTC· resolved Sep 06, 2025, 03:05 AM UTC

1. identified Sep 05, 2025, 10:31 PM UTC

   Kustomer is aware of an event affecting AI Agent messages via WhatsApp and SMS that may cause outbound messages to be delayed. Our team has identified the cause and is currently working toward implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. monitoring Sep 05, 2025, 10:58 PM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Sep 05, 2025, 11:28 PM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. monitoring Sep 06, 2025, 12:09 AM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is currently working to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. monitoring Sep 06, 2025, 01:00 AM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is continuing to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. monitoring Sep 06, 2025, 01:40 AM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is continuing to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. monitoring Sep 06, 2025, 02:13 AM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

8. monitoring Sep 06, 2025, 02:46 AM UTC

   Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is continuing to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

9. resolved Sep 06, 2025, 03:05 AM UTC

   Kustomer has verified the successful redrive of all messages on the WhatsApp channel.

10. postmortem Sep 24, 2025, 01:56 PM UTC

    # **Summary** On September 5, 2025, some outbound AI responses messages were delayed or failed to be sent. This primarily affected the WhatsApp channel. A code change introduced late on September 4, 2025, caused certain messages to be rejected by our processing system. This led to messages being stuck in a pending state until engineering teams identified and fixed the issue. # **Root Cause** Engineering identified the change that introduced the issue and implemented fixes to ensure messages could be properly processed. Once the fixes were deployed, all affected messages were retried and successfully delivered. # **Timeline** **Sep 4, 2025** * 10:00 pm EST: A system update was deployed. This unintentionally caused some AI and WhatsApp messages to enter a stuck state. **Sep 5, 2025** * Morning EST: Alerts indicated delayed message delivery. Engineers began investigating the issue. * 5:16 pm EST: Fixes were deployed to restore message delivery. Pending outbound messages began processing successfully. * 10:56 pm EST: All pending outbound messages were verified as sent. All systems fully operational. # **Lessons/Improvements** * \[IN PROGRESS\] We are enhancing monitoring and alerting to detect message delivery issues faster, so our teams can respond more quickly in the future. * \[IN PROGRESS\] We are expanding our testing approach to better simulate real-world message flows across all channels. This will help us catch downstream issues earlier in the development process before they reach production.

Read the full incident report →

Started Aug 29, 2025, 05:46 PM UTC· resolved Aug 29, 2025, 05:54 PM UTC

1. investigating Aug 29, 2025, 05:46 PM UTC

   Kustomer is aware of an event affecting access to the that may cause access issue to the platform returning a 403. Affecting PROD 1 & 2 Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support Email for any further questions or updates.

2. resolved Aug 29, 2025, 05:54 PM UTC

   Kustomer has resolved an event affecting access to the platform affecting all PRODs that caused permission errors when using the platform. To resolve this issue, our team has rolled back a code update around securities. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Chat or Email if you have additional questions or concerns.

3. postmortem Sep 07, 2025, 04:50 PM UTC

   # **Summary** On August 29, 2025, a security policy update unintentionally restricted access to web assets, resulting in HTTP 403 errors for the site for 26 minutes. The API was unaffected. During this window, some users were unable to log in; users already logged in were generally unaffected. The update was then reverted and service was restored. # **Root Cause** A security policy update inadvertently restricted access required for public web content. Reverting the update restored normal behavior. # **Timeline \(Eastern\)** **1:27 pm** – Security policies updated in production. **1:28 pm** – Monitoring detected elevated 403 errors. **1:34 pm** – Confirmed impact limited to web; API unaffected. **1:53 pm** – Update reverted; recovery observed. # **Lessons/Improvements** While there are safeguards and automated checks for access-related changes, this incident exposed a narrow gap; to cover such edge cases, the following actions are being taken: * **\[IN-PROGRESS\]** Broaden existing change-review rules for updates that could affect platform access. * **\[IN-PROGRESS\]** Extending our automated policy validations and guardrails to detect and block these edge cases before rollout.

Read the full incident report →

Started Aug 14, 2025, 05:32 PM UTC· resolved Aug 14, 2025, 06:57 PM UTC

1. monitoring Aug 14, 2025, 05:32 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting all pods that may prevent Facebook DMs from being ingested into the platform. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. monitoring Aug 14, 2025, 05:33 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting all pods that may prevent Facebook DMs from being ingested into the platform. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. resolved Aug 14, 2025, 06:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 06, 2025, 02:14 PM UTC· resolved Aug 06, 2025, 02:54 PM UTC

1. investigating Aug 06, 2025, 02:14 PM UTC

   Kustomer is aware of an event affecting Chat that may cause outbound chat messages to not deliver. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. identified Aug 06, 2025, 02:29 PM UTC

   Kustomer has identified an event in Chat that may cause Chat messages to fail to deliver Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

3. monitoring Aug 06, 2025, 02:30 PM UTC

   Kustomer has implemented an update to address an event affecting Chat in prod 1 that caused Chat messages to fail to deliver. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. resolved Aug 06, 2025, 02:54 PM UTC

   Kustomer has resolved an event affecting Chat in Prod 1 that caused Chat messages to not deliver. To resolve this issue, our team has released an updated. All chats that did not send have been processed. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support [email protected] if you have additional questions or concerns.

5. postmortem Sep 07, 2025, 04:57 PM UTC

   ## **Summary** On August 6, 2025, customers experienced an incident where chat messages failed to deliver in one of our production environments. The issue was resolved by identifying and using a previously stable image. We have initiated follow-up actions to enhance our image retention policies and improve logging to prevent recurrence. ## **Root Cause** A production issue impacted our chat messaging service, causing messages to not deliver. This was due to an unavailable container image, which prevented the service from recovering after an internal issue. This led to a backlog of messages in the queue and widespread chat delivery failures. The problem was resolved by reverting to a previous, stable image. ## **Timeline** **August 6, 2025** * **8/6 9:51 am EST**: Alerts were triggered indicating a high message count in our chat queue. * **8/6 10:01 am EST**: Attempts to redeploy the service failed due to a missing container image. * **8/6 10:11 am EST**: A status page was created to inform customers. * **8/6 10:34 am EST**: The service began recovering after being reconfigured to use a previous, stable image. * **8/6 10:51 am EST**: The status page was updated to resolved. * **8/6 10:58 am EST**: The incident was marked as resolved. ## **Lessons/Improvements** * **Image Visibility**: We are working to improve visibility and alerting around container image availability to prevent similar issues, as the container image deletion due to lifecycle policy was unexpected and not surfaced clearly. * **Rapid Detection & Triage**: The issue was quickly detected through our monitoring systems, allowing for immediate investigation. We can further enhance this by ensuring alerts are correctly prioritized. * **Clear Cross-Team Collaboration**: Our teams coordinated swiftly to escalate, communicate, and resolve the incident efficiently.

Read the full incident report →

Started Jul 14, 2025, 02:59 PM UTC· resolved Jul 14, 2025, 03:51 PM UTC

1. investigating Jul 14, 2025, 02:59 PM UTC

   Kustomer is aware of an event affecting routing, potential buffering issues, and the ability to update team status. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

2. monitoring Jul 14, 2025, 03:27 PM UTC

   Kustomer has implemented an update to address an event affecting PROD 1 that caused issues with loading behavior. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Jul 14, 2025, 03:36 PM UTC

   Kustomer has implemented an update to address an event affecting PROD 1 that caused issues with loading behavior. Our team is continuing to monitor this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. resolved Jul 14, 2025, 03:51 PM UTC

   Kustomer has resolved an event affecting platform loading issues . To resolve this issue, our team has rolled back changes to the platform. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. postmortem Sep 07, 2025, 04:49 PM UTC

   ## Summary Between July 13th and July 15th, 2025, customer support conversations were not being automatically routed to agents for approximately 12 organizations. This resulted in conversations remaining in queue indefinitely, preventing customer service teams from receiving new tickets. The issue was resolved through manual intervention by Kustomer engineering on July 14th at 6:30 PM EST. ## Root Cause A database cleanup script inadvertently deleted critical routing records \(work-item revisions\) that were still actively referenced by queued work-items. This created an unrecoverable state where the routing system was unable to process these work-items, effectively blocking affected queues and halting the distribution of subsequent conversations to agents. ## Timeline **Jul 13, 2025** * **Evening** - Database cleanup script runs and deletes active routing records, conversations begin backing up in queues **Jul 14, 2025** * **10:35 AM EST** - Kustomer engineering alerted to routing issues affecting multiple customer organizations * **10:41 AM EST** - Initial investigation begins, recent code deployments rolled back as precautionary measure * **11:33 AM EST** - Issue isolated to specific conversations stuck at the front of routing queues * **2:04 PM EST** - Root cause identified: missing database records preventing queue processing * **5:10 PM EST** - Manual database repair process validated and initiated * **6:30 PM EST** - All affected queues restored, normal routing resumed * **6:45 PM EST** - Monitoring confirms full resolution **Jul 15, 2025** * **9:44 AM EST** - Final confirmation that no additional conversations were impacted ## Lessons/Improvements * **Enhanced Database Script Safety** - We have implemented additional validation checks in our database cleanup processes to prevent deletion of records that are actively referenced by live systems. **Status**: Complete * **Graceful Error Handling** - The routing system will now gracefully handle missing database references instead of failing silently, ensuring conversations continue to route even if supporting records are corrupted. **Status**: Not Started * **Queue Health Monitoring** - We are implementing proactive monitoring to detect queue blockages and alert engineering teams before customer impact occurs. **Status**: Not Started

Read the full incident report →

Started Jun 24, 2025, 08:23 PM UTC· resolved Jun 25, 2025, 01:56 AM UTC

1. identified Jun 24, 2025, 08:23 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting, AI Agents, AI Summaries, Customer Assist, and Agent Assist that may cause errors within the platform. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. Please follow OpenAI's Statuspage at [https://status.openai.com/] for more details. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. identified Jun 24, 2025, 11:16 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting, AI Agents, AI Summaries, Customer Assist, and Agent Assist that may cause errors within the platform. Our team is continuing to monitor the incident, and working with the vendor where possible to resolve the issue. Please follow OpenAI's Statuspage at [https://status.openai.com/] for more details. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. resolved Jun 25, 2025, 01:56 AM UTC

   Kustomer has confirmed that error rates from OpenAI have recovered. If you have any questions please reach out to Kustomer support at [email protected]

Read the full incident report →

Started Jun 06, 2025, 05:41 PM UTC· resolved Jun 06, 2025, 09:01 PM UTC

1. investigating Jun 06, 2025, 05:41 PM UTC

   Kustomer is aware of an event affecting Native Whatsapp that may cause outbound messages to delay sending Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. investigating Jun 06, 2025, 05:42 PM UTC

   Kustomer is aware of an event affecting Native Whatsapp that may cause outbound messages to delay sending Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

3. investigating Jun 06, 2025, 06:05 PM UTC

   Kustomer is aware of an event affecting Native Whatsapp that may cause outbound messages to delay sending Our team is continuing to work to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

4. investigating Jun 06, 2025, 06:34 PM UTC

   Kustomer is aware of an event affecting Native Whatsapp that may cause outbound messages to delay sending Our team is still working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

5. identified Jun 06, 2025, 07:04 PM UTC

   Kustomer has identified an event affecting Whatsapp (PROD1) that may cause delayed sending Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. identified Jun 06, 2025, 07:32 PM UTC

   Kustomer has identified an event affecting Whatsapp (PROD1) that may cause delayed sending Our team is continuing to work on implementing a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. identified Jun 06, 2025, 08:01 PM UTC

   Kustomer has identified an event affecting Whatsapp (PROD1) that may cause delayed sending Our team continues to work on implementing a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

8. monitoring Jun 06, 2025, 08:10 PM UTC

   Kustomer has implemented an update to address an event affecting Native Whatsapp that may have caused delays with delivery of outbound messages. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

9. resolved Jun 06, 2025, 09:01 PM UTC

   Kustomer has resolved an event affecting Native WhatsApp Channel in PROD 1 that caused outbound messages to delay in sending. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Chat or Email if you have additional questions or concerns.

10. postmortem Jun 13, 2025, 07:37 PM UTC

    ## **Summary** On Monday, June 6th, drafts for Whatsapp experienced a significant delay in sending. This led to an incident where messages were not delivered before attached media items expired. ## **Root Cause** Our WhatsApp service experienced an overload due to a significant surge in WhatsApp messages. The service was scaling, but it couldn't keep pace with the sudden demand, resulting in elevated latency and timeouts. This, in turn, initiated retries within our service, intensifying the message load and, occasionally, generating duplicate messages. In some cases, the service timed out, but the draft creation was still successful - which caused the same messages to be retried and led to duplicate messages. Consequently, both the Drafts service and WhatsApp service on prod1 experienced considerable spikes in memory and CPU usage. In addition, WhatsApp was returning errors about media items in some of the messages. This was due to the increased latency - the media item in some messages had expired before the message could be sent. Which also caused some additional retries and exacerbated the issue. ## **Timeline** **Jun 6, 2025** **1:33 PM EST** Incident created. **1:41 PM EST** Began investigating recent releases in WhatsApp and other related services. **1:53 PM EST** Discovered spikes in WhatsApp service, not code change related. **4:03 PM EST** Deployed scaling changes to WhatsApp service, spikes settled down. **4:07 PM EST** Created a change to reduce the rate limit in Drafts service for WhatsApp. **7:59 PM EST** Deployed rate limit change; traffic returned to healthy levels. ## **Lessons/Improvements** * **Duplicate Drafts Investigation** - Understand why duplicate WhatsApp drafts occurred during the incident. * Status: Done * **Scaling Enhancements** - Increased scaling for WhatsApp service to better handle message bursts. * Status: Done * **Adjusted Rate Limit** - Decreased WhatsApp rate limit from 400/minute to 300/minute in Drafts service. * Status: Done * **Media Expiration** - Investigate expiration on media items and determine if it can be extended beyond that. * Status: In Progress

Read the full incident report →

Started May 23, 2025, 08:09 PM UTC· resolved May 24, 2025, 12:15 PM UTC

1. investigating May 23, 2025, 08:09 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting Twilio SMS that may cause Twilio SMS outbound messages to not deliver for Verizon customers and may impact verification for AI Agents. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. For more information, please refer to https://status.twilio.com/. Please expect further updates within the next 3 hours & reach out to our Support team with any additional questions. You can reach us via email at [email protected]

2. monitoring May 23, 2025, 08:35 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting Twilio SMS that may cause Twilio SMS outbound messages to not deliver for Verizon customers and may impact verification for AI Agents. Our team continues to monitor the incident, and is working with the vendor where possible to resolve the issue. For more information, please refer to https://status.twilio.com/. Please expect further updates within the next 3 hours & reach out to our Support team with any additional questions. You can reach us via email at [email protected]

3. monitoring May 23, 2025, 11:32 PM UTC

   Kustomer is continuing to monitor an event reported by one of our third party vendors affecting Twilio SMS that may cause Twilio SMS outbound messages to not deliver for Verizon customers and may impact verification for AI Agents. We are working with the vendor where possible to resolve the issue. For more information, please refer to https://status.twilio.com/. Please expect further updates within the next 3 hours & reach out to our Support team with any additional questions. You can reach us via email at [email protected]

4. monitoring May 24, 2025, 02:25 AM UTC

   Kustomer is continuing to monitor an issue reported by our third-party vendor, Twilio, which is impacting SMS delivery. This may result in outbound Twilio SMS messages not reaching Verizon customers and could also affect verification processes for AI Agents. We are working with the vendor, where possible, to help resolve the issue. For more information, please visit: https://status.twilio.com. We’ll provide another update within the next 3 hours. If you have any additional questions, feel free to contact our Support team at [email protected].

5. monitoring May 24, 2025, 05:41 AM UTC

   Kustomer is monitoring an ongoing issue reported by our third-party provider, Twilio, that is affecting SMS message delivery. This issue may prevent outbound Twilio SMS messages from reaching Verizon customers and could impact verification flows for AI Agents. We're working with Twilio where possible to support resolution efforts. For the latest updates from Twilio, please refer to: https://status.twilio.com. If you have any further questions, you can contact our Support team via email at [email protected].

6. resolved May 24, 2025, 12:15 PM UTC

   Kustomer is no longer experiencing the third-party incident affecting Twilio SMS delivery, which may have prevented outbound messages from reaching Verizon customers and impacted AI Agent verification processes. Twilio has resolved the issue, and we will continue to monitor to ensure stability across all systems. Our services are currently operating as expected. If you have any questions or need further assistance, please don’t hesitate to contact our Support team at [email protected].

Read the full incident report →

Started May 14, 2025, 05:35 PM UTC· resolved May 14, 2025, 06:36 PM UTC

1. investigating May 14, 2025, 05:35 PM UTC

   Kustomer is aware of an event affecting Chat widgets that may cause Chat messages to not send. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via email at [email protected] for any further questions or updates.

2. investigating May 14, 2025, 05:40 PM UTC

   Kustomer is aware of an event affecting Chat widgets that may cause Chat messages to not send. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via email at [email protected] for any further questions or updates.

3. monitoring May 14, 2025, 06:00 PM UTC

   Kustomer has implemented an update to address an event affecting Chat Widgets that caused messages to not be sent in the Chat. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email '[email protected]' if you have additional questions or concerns.

4. resolved May 14, 2025, 06:36 PM UTC

   Kustomer has resolved an event affecting Chat messaging that caused Chats to not be allowed to send in the widget. To resolve this issue, our team has rolled back an update to the platform. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Email '[email protected]' if you have additional questions or concerns.

5. postmortem May 23, 2025, 06:35 PM UTC

   ## **Summary** On May 14, 2025, a change was deployed to the chat widget at 10:56 AM EDT to improve the logout experience. This update aimed to fix an issue where the widget would incorrectly return to a "New Chat" screen after a user logged out \(specifically, calling \`Kustomer.logout\(\)\`\) from a different page, such as a knowledge base article. However, this change inadvertently caused the chat widget to lose access to essential functions needed for sending and receiving messages after a logout and subsequent new chat initiation. This resulted in messages disappearing and the chat window becoming unresponsive. ## **Root Cause** The deployed change, while intended to improve the logout flow, inadvertently caused the application to lose references to critical business logic functions stored within the application state. The application state was incorrectly assumed to only hold data, not these essential functions, leading to their loss during the state restoration process after logout. ## **Timeline** **May 14, 2025** * **10:56 AM EST** Chat widget update deployed. * **1:24 PM EST** Incident declared after reports of issues. * **1:51 PM EST** Error identified related to missing functions in the application state. * **1:53 PM EST** Rollback performed, restoring chat functionality. * **2:54 PM EST** Root cause found and fixed **May 15, 2025** * **10:30 PM EST** All impacted sessions were identified AND cleared from cache ## **Lessons/Improvements** * **Enhanced Testing** - Automated regression tests will be added to prevent future regressions related to the application state. QA testing procedures will be updated to always include testing the ability to start a new conversation after logout. * **Status**: In Progress * **Improved Documentation** - Documentation regarding the expected behavior of the logout function will be enhanced. * **Status**: Done

Read the full incident report →

Started May 08, 2025, 08:09 PM UTC· resolved May 08, 2025, 10:21 PM UTC

1. investigating May 08, 2025, 08:09 PM UTC

   Kustomer is aware of an event affecting our WhatsApp Integration that may cause latency in sending messages. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

2. identified May 08, 2025, 08:58 PM UTC

   Kustomer has identified an event in our WhatsApp Integrations that may cause delays in messaging. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

3. monitoring May 08, 2025, 09:36 PM UTC

   Kustomer has implemented an update to address an event affecting WhatsApp Channel that caused long delays in sending WhatsApp messages and multiples of the same message being sent. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email or Chat if you have additional questions or concerns.

4. resolved May 08, 2025, 10:21 PM UTC

   Kustomer has resolved an event affecting WhatsApp Channel that caused long delays in messages to be delivered. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Email or Chat if you have additional questions or concerns.

5. postmortem May 20, 2025, 03:13 PM UTC

   # **Summary** Between April 24 and May 8, our system experienced three instances of delayed WhatsApp message processing caused by sudden increases in send volume. While scaling eventually resolved these issues, we have taken steps to prevent future occurrences. These include increased scaling for high-volume events like Mother's Day and the implementation of rate limiting. # **Root Cause** The root cause was an overwhelming surge in message volume, leading to the send queue growing faster than our processing capacity. The autoscaling configuration's response time was insufficient to prevent client impact. To address this, we implemented per-user rate limiting to control queue ingestion and increased baseline resource availability during the Mother's Day period. # **Timeline** **April 24, 2025 - 11:00 AM ET -** Slowdown of sending for WhatsApp messages. **April 24, 2025 - 4:00 PM ET -** Scaling allowed the system to recover. **May 7, 2025 - 12:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 7, 2025 - 3:00 PM ET -** Additional scaling allowed our system to recover. **May 8, 2025 - 2:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 8, 2025 - 5:00 PM ET -** Refined the auto scaling configuration to allow the system to recover. **May 9, 2025 - 3:00 PM ET -** Rate limiting imposed on sending of WhatsApp messages. **May 9, 2025 - 3:30 PM ET -** Increased scaling limit temporarily for holiday weekend # **Lessons/Improvements** * **Rate Limiting** - In order to mitigate the impact of several sends coming at the same time, we imposed rate limiting on WhatsApp sends. Initial implementation was 400 sends per user per minute, with further tuning to come. * **Scaling** - Services connected to the sending of WhatsApp messages were scaled up temporarily for the Mother's Day holiday weekend.

Read the full incident report →

Started May 07, 2025, 06:55 PM UTC· resolved May 07, 2025, 07:12 PM UTC

1. monitoring May 07, 2025, 06:55 PM UTC

   Kustomer has implemented an update to address an event affecting WHATSAPP Chat Channel in PROD 1 that caused long delays in sending and multiples of the same messages being sent in WhatsApp conversations. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email OR Chat if you have additional questions or concerns.

2. resolved May 07, 2025, 07:12 PM UTC

   Kustomer has resolved an event affecting WhatsApp Chat in Prod 1 that caused long delays in sending messages and multiple of the same message going out. To resolve this issue, our team has scaled up our traffic allowance with WhatsApp messaging service to alleviate this issue. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Email or Chat if you have additional questions or concerns.

3. postmortem May 20, 2025, 03:15 PM UTC

   # **Summary** Between April 24 and May 8, our system experienced three instances of delayed WhatsApp message processing caused by sudden increases in send volume. While scaling eventually resolved these issues, we have taken steps to prevent future occurrences. These include increased scaling for high-volume events like Mother's Day and the implementation of rate limiting. # **Root Cause** The root cause was an overwhelming surge in message volume, leading to the send queue growing faster than our processing capacity. The autoscaling configuration's response time was insufficient to prevent client impact. To address this, we implemented per-user rate limiting to control queue ingestion and increased baseline resource availability during the Mother's Day period. # **Timeline** **April 24, 2025 - 11:00 AM ET -** Slowdown of sending for WhatsApp messages. **April 24, 2025 - 4:00 PM ET -** Scaling allowed the system to recover. **May 7, 2025 - 12:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 7, 2025 - 3:00 PM ET -** Additional scaling allowed our system to recover. **May 8, 2025 - 2:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 8, 2025 - 5:00 PM ET -** Refined the auto scaling configuration to allow the system to recover. **May 9, 2025 - 3:00 PM ET -** Rate limiting imposed on sending of WhatsApp messages. **May 9, 2025 - 3:30 PM ET -** Increased scaling limit temporarily for holiday weekend # **Lessons/Improvements** * **Rate Limiting** - In order to mitigate the impact of several sends coming at the same time, we imposed rate limiting on WhatsApp sends. Initial implementation was 400 sends per user per minute, with further tuning to come. * **Scaling** - Services connected to the sending of WhatsApp messages were scaled up temporarily for the Mother's Day holiday weekend.

Read the full incident report →

Started Apr 24, 2025, 06:05 PM UTC· resolved Apr 24, 2025, 08:17 PM UTC

1. investigating Apr 24, 2025, 06:05 PM UTC

   Kustomer has identified an event in Whatsapp that may cause delays in messages being sent. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

2. investigating Apr 24, 2025, 06:11 PM UTC

   We are continuing to investigate this issue.

3. investigating Apr 24, 2025, 06:12 PM UTC

   We are continuing to investigate this issue.

4. monitoring Apr 24, 2025, 06:39 PM UTC

   Kustomer has implemented an update to address an event affecting latency in WhatsApp affecting Prod 1 that caused latency in messaging. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. monitoring Apr 24, 2025, 07:28 PM UTC

   Kustomer has implemented an update to address an event affecting latencies in WhatsApp messaging that caused messages to be delayed and/or not persist. Our team is continuing to monitor this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. resolved Apr 24, 2025, 08:17 PM UTC

   Kustomer has resolved an event affecting WhatsApp Prod 1 that caused latency in messages being delivered. To resolve this issue, our team has processed a redrive. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. postmortem May 20, 2025, 03:16 PM UTC

   # **Summary** Between April 24 and May 8, our system experienced three instances of delayed WhatsApp message processing caused by sudden increases in send volume. While scaling eventually resolved these issues, we have taken steps to prevent future occurrences. These include increased scaling for high-volume events like Mother's Day and the implementation of rate limiting. # **Root Cause** The root cause was an overwhelming surge in message volume, leading to the send queue growing faster than our processing capacity. The autoscaling configuration's response time was insufficient to prevent client impact. To address this, we implemented per-user rate limiting to control queue ingestion and increased baseline resource availability during the Mother's Day period. # **Timeline** **April 24, 2025 - 11:00 AM ET -** Slowdown of sending for WhatsApp messages. **April 24, 2025 - 4:00 PM ET -** Scaling allowed the system to recover. **May 7, 2025 - 12:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 7, 2025 - 3:00 PM ET -** Additional scaling allowed our system to recover. **May 8, 2025 - 2:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 8, 2025 - 5:00 PM ET -** Refined the auto scaling configuration to allow the system to recover. **May 9, 2025 - 3:00 PM ET -** Rate limiting imposed on sending of WhatsApp messages. **May 9, 2025 - 3:30 PM ET -** Increased scaling limit temporarily for holiday weekend # **Lessons/Improvements** * **Rate Limiting** - In order to mitigate the impact of several sends coming at the same time, we imposed rate limiting on WhatsApp sends. Initial implementation was 400 sends per user per minute, with further tuning to come. * **Scaling** - Services connected to the sending of WhatsApp messages were scaled up temporarily for the Mother's Day holiday weekend.

Read the full incident report →

Started Apr 03, 2025, 03:11 AM UTC· resolved Apr 03, 2025, 04:01 AM UTC

1. identified Apr 03, 2025, 03:11 AM UTC

   Kustomer has identified an event that may cause latency in sending and receiving chat messages, as well as in processing workflows. Our team is actively working on implementing a resolution. Please expect further updates within the next 30 minutes. If you have any questions or require additional information, please reach out to Kustomer Support at [email protected].

2. monitoring Apr 03, 2025, 03:24 AM UTC

   Kustomer has implemented an update to address an event affecting chat message delivery and workflow processing that caused latency. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. resolved Apr 03, 2025, 04:01 AM UTC

   Kustomer has resolved an event affecting chat message delivery and workflow processing that caused latency. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. postmortem May 23, 2025, 06:04 PM UTC

   # **Summary** On April 2, our background worker services were unable to scale up to meet increased demand because their auto-scaling alarms had not been in place. As a result, queues backed up and processing latency spiked. After manually scaling the services and restoring the missing alarms, throughput and responsiveness returned to normal within an hour. # **Root Cause** During a recent migration of services to a new cluster, the CloudWatch alarms that trigger scale-out and scale-in actions were inadvertently removed and never recreated. Without those alarms, the auto-scaling system never responded to rising load. # **Timeline** **04/02 9:49 pm EST** - PagerDuty alert triggered: workflow service has a high number of messages in the queue. **04/02 9:50 pm EST -** On-call engineers began investigating the issue. **04/02 10:43 pm EST** - Workflow containers were manually scaled out. **04/02 10:57 pm EST** - Multiple customers reported Conversation Assistants were not responding. **04/02 10:57 pm EST** - Message volume in the workflow queue began decreasing. **04/02 11:20 pm EST** - Workflow service processed all pending messages and caught up with the queue. **04/02 11:19 pm EST** - Root cause identified: missing CloudWatch alarms due to Terraform configuration. A temporary fix was applied by manually creating an alarm and verifying that scaling worked as expected. **04/03 1:06 am EST** - A permanent fix \(redeploying the affected services\) was identified and tested successfully with one service. **04/03 1:07 am EST** - Rollout of permanent fix initiated for remaining impacted services. # **Lessons / Improvements** **Recreate Critical Alarms**Ensure auto-scaling alarms are reattached whenever services are migrated or redeployed. **Distinct Naming Conventions**Incorporate cluster or environment identifiers in alarm names to avoid accidental deletion. **Pre-Migration Verification**Add a checklist step to confirm all scaling policies exist before cutting over to a new environment.

Read the full incident report →

Started Mar 31, 2025, 05:33 PM UTC· resolved Apr 01, 2025, 01:43 AM UTC

1. monitoring Mar 31, 2025, 05:33 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting Kustomer Voice that may cause voice calls to fail within the platform. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. Please follow Twilio's status page (https://status.twilio.com/) for more details. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. monitoring Mar 31, 2025, 08:03 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting Kustomer Voice. Twilio has implemented an update to address an event that caused calls to potentially fail and are monitoring the situation. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. Please follow Twilio's status page (https://status.twilio.com/) for more details. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Mar 31, 2025, 11:01 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting Kustomer Voice. Twilio has implemented an update to address an event that may have caused calls to fail and is continuing to monitor the situation. Our team continues to monitor the incident, and we are working with the vendor where possible to resolve the issue. Please follow Twilio's status page (https://status.twilio.com/) for more details. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. resolved Apr 01, 2025, 01:43 AM UTC

   We are pleased to inform you that Kustomer Voice is no longer experiencing issues related to the recent event reported by Twilio. While Twilio continues to monitor their update, which was implemented to address call failures, we have confirmed that our services are functioning normally. We will keep a close eye on the situation alongside our vendor to ensure continued stability. Please check Twilio's status page (https://status.twilio.com/) for ongoing updates. If you have any questions or need further assistance, feel free to contact our support team at [email protected].

Read the full incident report →

Started Mar 06, 2025, 03:22 PM UTC· resolved Mar 06, 2025, 05:42 PM UTC

1. monitoring Mar 06, 2025, 03:22 PM UTC

   Kustomer has implemented an update to address an event affecting Twilio Whatsapp & SMS messages that caused delayed sending. The update has been successfully completed. Any impacted messages that have not sent have will been identified and re-sent. We are still in the process of re-sending and will continue to post updates. Please and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. monitoring Mar 06, 2025, 03:54 PM UTC

   Kustomer has implemented an update to address an event affecting Twilio Whatsapp & SMS messages in Prod 1 that caused delayed sending. The update has been successfully completed. Any impacted messages that have not sent have will been identified and re-sent. We are still in the process of re-sending and will continue to post updates. Please and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Mar 06, 2025, 04:23 PM UTC

   Kustomer has implemented an update to address an event affecting Twilio Whatsapp & SMS messages in Prod 1 that caused delayed sending. The update has been successfully completed. Any impacted messages that have not sent have will been identified and re-sent. Our team is still in the process of re-sending any impacted messages and will continue to post updates. Please and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. monitoring Mar 06, 2025, 04:53 PM UTC

   Kustomer has applied an update aimed at resolving an issue in Prod 1 that was delaying the sending of Twilio Whatsapp & SMS messages. This update has been successfully executed. We have identified any messages that were affected and have not yet been sent, and are actively resending them. Our team will continue working on this and will provide further updates as necessary. Should you have any further questions or concerns, please contact Kustomer support at [email protected].

5. monitoring Mar 06, 2025, 05:22 PM UTC

   Kustomer has implemented an update to fix a problem in Prod 1 that was causing delays in the delivery of Twilio Whatsapp & SMS messages. The update has been completed successfully. We have pinpointed messages that were delayed and not yet sent and are currently in the process of resending them. Our team remains committed to addressing this issue and will keep posting updates as needed. If you have any more questions or concerns, please reach out to Kustomer support at [email protected].

6. resolved Mar 06, 2025, 05:42 PM UTC

   Kustomer has addressed an issue impacting Twilio Whatsapp & SMS messages that resulted in delayed transmissions. To rectify this, our team has resent all messages that were not sent during the disruption. Following thorough monitoring, we have confirmed that our systems have been fully restored and all affected messages have been successfully resent. If you have any further questions or concerns, please contact Kustomer support at [email protected].

7. postmortem Mar 10, 2025, 05:47 PM UTC

   ## **Summary** A subset of Twilio SMS and WhatsApp drafts sent from the Kustomer platform between 9:25 AM EST and 9:53 AM EST on March 6th 2025 failed to send and were stuck in a pending state until they were repaired by Kustomer engineering at 12:40PM. ## **Root Cause** Kustomer introduced a change, intended to improve error handling for Twilio SMS and WhatsApp channels. This change introduced a scenario where some drafts would get incorrectly identified as being misconfigured, which prevented them from being sent. This left them in a “Sending” state in the UI until they were manually rescheduled. ## **Timeline** **Mar 6, 2025** **9:20 AM EST** Triggered deployment intended to improve error handling in Kustomer’s Twilio service **9:25 AM EST** Deployment finished **9:41 AM EST** Kustomer was alerted that Twilio SMS and WhatsApp drafts/messages were stuck in “Sending” state, began rolling back change \(as it was the only recent change\) **9:53 AM EST** Rollback is completed, Twilio SMS and WhatsApp drafts/messages are confirmed to be sending again, began monitoring Twilio service **10:00 AM EST** - Started to redrive the Twilio SMS and WhatsApp drafts/messages that were not delivered during the incident window **12:40 PM EST** Finished redriving the Twilio SMS and WhatsApp drafts/messages that were not delivering, all messages sent ## **Lessons/Improvements** * **Automated Tests** - We are adding additional automated tests that test the Twilio service against a broader array of configurations to prevent further regressions of this type. Kustomer is identifying all relevant parameters to ensure thorough test case coverage. **Status**: In Progress * **Runbook for Redriving drafts** - Kustomer will optimize our runbook for redriving undelivered drafts to minimize expected recovery time in future cases of orphaned drafts. **Status**: In Progress * **Additional Review Gates** - Message delivery is of the utmost importance. We have added an additional review step for changes that impact logic that can change the deliverability of a message. **Status**: Complete

Read the full incident report →

Started Jan 29, 2025, 08:52 AM UTC· resolved Jan 29, 2025, 11:45 AM UTC

1. investigating Jan 29, 2025, 08:52 AM UTC

   Kustomer is aware of an issue that is causing errors when accessing the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any further questions or updates.

2. investigating Jan 29, 2025, 09:26 AM UTC

   We are still working to resolve the issue that is causing errors when accessing the platform. Kustomer is actively working to identify the cause and implement a resolution. Please expect additional updates within the next 30 minute and reach out to Kustomer Support at [email protected] for any further questions or updates.

3. identified Jan 29, 2025, 10:11 AM UTC

   Kustomer has identified the source of the issue with users not being able to access the platform as an issue with our cloud provider, who is actively working on a fix. Our team continue to work with our vendor to implement a resolution. We will provide an estimated resolution time as soon as we have any further updates from our provider. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any further queries or concerns.

4. identified Jan 29, 2025, 10:27 AM UTC

   Kustomer has identified the cause of the issue accessing the platform as being due to an issue from our cloud provider AWS. We are actively working with them to arrive at a resolution as quickly as possible. Please feel free to check AWS's statuspage at https://health.aws.amazon.com/health/status for further details. Our team will continue to share updates until this issue is resolved. In the meantime, please reach out to Kustomer Support at [email protected] for any additional concerns or queries.

5. investigating Jan 29, 2025, 11:11 AM UTC

   Kustomer is are aware of the ongoing AWS CloudFront issue affecting users in the EU region. AWS has identified the root cause of the CloudFront errors affecting customers and is actively applying mitigations. We will continue monitoring the situation and expect another update from AWS within the next 30-45 minutes. You can also check their statuspage for further details: https://health.aws.amazon.com/health/status As a potential workaround, impacted users may be able to bypass the issue by using a VPN to connect through a non-EU region, such as the United States. This change may restore access while AWS resolves the issue. Before using a VPN, please check your internal policies to ensure compliance with your organization’s security and usage guidelines. Please reach out to Kustomer Support at [email protected] for any further questions or concerns.

6. identified Jan 29, 2025, 11:11 AM UTC

   Kustomer is are aware of the ongoing AWS CloudFront issue affecting users in the EU region. AWS has identified the root cause of the CloudFront errors affecting customers and is actively applying mitigations. We will continue monitoring the situation and expect another update from AWS within the next 30-45 minutes. You can also check their statuspage for further details: https://health.aws.amazon.com/health/status As a potential workaround, impacted users may be able to bypass the issue by using a VPN to connect through a non-EU region, such as the United States. This change may restore access while AWS resolves the issue. Before using a VPN, please check your internal policies to ensure compliance with your organization’s security and usage guidelines. Please reach out to Kustomer Support at [email protected] for any further questions or concerns.

7. identified Jan 29, 2025, 11:18 AM UTC

   Kustomer is are aware of the ongoing AWS CloudFront issue affecting users in the EU region. AWS has identified the root cause of the CloudFront errors affecting customers and is actively applying mitigations. We will continue monitoring the situation and expect another update from AWS within the next 30-45 minutes. Direct updates are also available on AWS's statuspage: https://health.aws.amazon.com/health/status As a potential workaround, impacted users may be able to bypass the issue by using a VPN to connect through a non-EU region, such as the United States. This change may restore access while AWS resolves the issue. Before using a VPN, please check your internal policies to ensure compliance with your organization’s security and usage guidelines. Please reach out to Kustomer Support at [email protected] for any further questions or concerns.

8. resolved Jan 29, 2025, 11:45 AM UTC

   Kustomer has resolved the AWS CloudFront issue which impacted users in the EU region after AWS released a fix restoring the service to full functionality. Please find further details of AWS's update in their statuspage: https://health.aws.amazon.com/health/status After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

9. postmortem Feb 18, 2025, 07:36 PM UTC

   # **Summary** On January 29th, 2025, users in the EU experienced login failures due to an AWS CloudFront issue. The issue was addressed, with steps in place to investigate further, and additional measures planned to prevent recurrence. # **Root Cause** On January 29, 2025, at around 3:30 AM EST, users in the EU experienced login failures due to an AWS CloudFront issue preventing access to S3 static website origins. This resulted in some customers in the EU region being unable to access the platform for several hours. Following the incident, we are investigating fallback options for CloudFront and enhancing monitoring. We have reviewed the information provided by AWS and confirmed that the issue originated within their systems. # **Timeline** **Jan 29, 2025** * 3:30 AM ET: Incident was declared, and investigation started * 4:32 AM ET: Created support issue with AWS to investigate potential AWS-related causes * 5:00 AM ET: AWS confirmed the incident was caused by a CloudFront issue on their end, affecting EU logins * 6:06 AM ET: Recovery confirmed as affected customers regained access ‌ **Lessons/Improvements** * **\[Completed\] AWS Coordination:** Worked with AWS to obtain and review their postmortem. Kustomer and AWS are aligned that the root cause was entirely within AWS, and no action items were identified for Kustomer * **\[Planned\] Fallback Strategy:** Explore fallback options and evaluate alternative solutions when Cloudfront is experiencing connectivity problems * **\[Planned\] Monitoring & Resilience:** Improve monitoring, alerting, and disaster recovery planning

Read the full incident report →