[PLATFORM] Issue Accessing Platform

Update timeline

1. investigating Jan 29, 2025, 08:52 AM UTC

   Kustomer is aware of an issue that is causing errors when accessing the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any further questions or updates.

2. investigating Jan 29, 2025, 09:26 AM UTC

   We are still working to resolve the issue that is causing errors when accessing the platform. Kustomer is actively working to identify the cause and implement a resolution. Please expect additional updates within the next 30 minute and reach out to Kustomer Support at [email protected] for any further questions or updates.

3. identified Jan 29, 2025, 10:11 AM UTC

   Kustomer has identified the source of the issue with users not being able to access the platform as an issue with our cloud provider, who is actively working on a fix. Our team continue to work with our vendor to implement a resolution. We will provide an estimated resolution time as soon as we have any further updates from our provider. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any further queries or concerns.

4. identified Jan 29, 2025, 10:27 AM UTC

   Kustomer has identified the cause of the issue accessing the platform as being due to an issue from our cloud provider AWS. We are actively working with them to arrive at a resolution as quickly as possible. Please feel free to check AWS's statuspage at https://health.aws.amazon.com/health/status for further details. Our team will continue to share updates until this issue is resolved. In the meantime, please reach out to Kustomer Support at [email protected] for any additional concerns or queries.

5. investigating Jan 29, 2025, 11:11 AM UTC

   Kustomer is are aware of the ongoing AWS CloudFront issue affecting users in the EU region. AWS has identified the root cause of the CloudFront errors affecting customers and is actively applying mitigations. We will continue monitoring the situation and expect another update from AWS within the next 30-45 minutes. You can also check their statuspage for further details: https://health.aws.amazon.com/health/status As a potential workaround, impacted users may be able to bypass the issue by using a VPN to connect through a non-EU region, such as the United States. This change may restore access while AWS resolves the issue. Before using a VPN, please check your internal policies to ensure compliance with your organization’s security and usage guidelines. Please reach out to Kustomer Support at [email protected] for any further questions or concerns.

6. identified Jan 29, 2025, 11:11 AM UTC

   Kustomer is are aware of the ongoing AWS CloudFront issue affecting users in the EU region. AWS has identified the root cause of the CloudFront errors affecting customers and is actively applying mitigations. We will continue monitoring the situation and expect another update from AWS within the next 30-45 minutes. You can also check their statuspage for further details: https://health.aws.amazon.com/health/status As a potential workaround, impacted users may be able to bypass the issue by using a VPN to connect through a non-EU region, such as the United States. This change may restore access while AWS resolves the issue. Before using a VPN, please check your internal policies to ensure compliance with your organization’s security and usage guidelines. Please reach out to Kustomer Support at [email protected] for any further questions or concerns.

7. identified Jan 29, 2025, 11:18 AM UTC

   Kustomer is are aware of the ongoing AWS CloudFront issue affecting users in the EU region. AWS has identified the root cause of the CloudFront errors affecting customers and is actively applying mitigations. We will continue monitoring the situation and expect another update from AWS within the next 30-45 minutes. Direct updates are also available on AWS's statuspage: https://health.aws.amazon.com/health/status As a potential workaround, impacted users may be able to bypass the issue by using a VPN to connect through a non-EU region, such as the United States. This change may restore access while AWS resolves the issue. Before using a VPN, please check your internal policies to ensure compliance with your organization’s security and usage guidelines. Please reach out to Kustomer Support at [email protected] for any further questions or concerns.

8. resolved Jan 29, 2025, 11:45 AM UTC

   Kustomer has resolved the AWS CloudFront issue which impacted users in the EU region after AWS released a fix restoring the service to full functionality. Please find further details of AWS's update in their statuspage: https://health.aws.amazon.com/health/status After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

9. postmortem Feb 18, 2025, 07:36 PM UTC

   # **Summary** On January 29th, 2025, users in the EU experienced login failures due to an AWS CloudFront issue. The issue was addressed, with steps in place to investigate further, and additional measures planned to prevent recurrence. # **Root Cause** On January 29, 2025, at around 3:30 AM EST, users in the EU experienced login failures due to an AWS CloudFront issue preventing access to S3 static website origins. This resulted in some customers in the EU region being unable to access the platform for several hours. Following the incident, we are investigating fallback options for CloudFront and enhancing monitoring. We have reviewed the information provided by AWS and confirmed that the issue originated within their systems. # **Timeline** **Jan 29, 2025** * 3:30 AM ET: Incident was declared, and investigation started * 4:32 AM ET: Created support issue with AWS to investigate potential AWS-related causes * 5:00 AM ET: AWS confirmed the incident was caused by a CloudFront issue on their end, affecting EU logins * 6:06 AM ET: Recovery confirmed as affected customers regained access ‌ **Lessons/Improvements** * **\[Completed\] AWS Coordination:** Worked with AWS to obtain and review their postmortem. Kustomer and AWS are aligned that the root cause was entirely within AWS, and no action items were identified for Kustomer * **\[Planned\] Fallback Strategy:** Explore fallback options and evaluate alternative solutions when Cloudfront is experiencing connectivity problems * **\[Planned\] Monitoring & Resilience:** Improve monitoring, alerting, and disaster recovery planning