Kustomer incident

[PLATFORM] [403 ERROR] [PROD 1 & 2]

Minor Resolved View vendor source →

Kustomer experienced a minor incident on August 29, 2025 affecting Web Client and Web Client, lasting 7m. The incident has been resolved; the full update timeline is below.

Started
Aug 29, 2025, 05:46 PM UTC
Resolved
Aug 29, 2025, 05:54 PM UTC
Duration
7m
Detected by Pingoru
Aug 29, 2025, 05:46 PM UTC

Affected components

Web ClientWeb Client

Update timeline

  1. investigating Aug 29, 2025, 05:46 PM UTC

    Kustomer is aware of an event affecting access to the that may cause access issue to the platform returning a 403. Affecting PROD 1 & 2 Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support Email for any further questions or updates.

  2. resolved Aug 29, 2025, 05:54 PM UTC

    Kustomer has resolved an event affecting access to the platform affecting all PRODs that caused permission errors when using the platform. To resolve this issue, our team has rolled back a code update around securities. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Chat or Email if you have additional questions or concerns.

  3. postmortem Sep 07, 2025, 04:50 PM UTC

    # **Summary** On August 29, 2025, a security policy update unintentionally restricted access to web assets, resulting in HTTP 403 errors for the site for 26 minutes. The API was unaffected. During this window, some users were unable to log in; users already logged in were generally unaffected. The update was then reverted and service was restored. # **Root Cause** A security policy update inadvertently restricted access required for public web content. Reverting the update restored normal behavior. # **Timeline \(Eastern\)** **1:27 pm** – Security policies updated in production. **1:28 pm** – Monitoring detected elevated 403 errors. **1:34 pm** – Confirmed impact limited to web; API unaffected. **1:53 pm** – Update reverted; recovery observed. # **Lessons/Improvements** While there are safeguards and automated checks for access-related changes, this incident exposed a narrow gap; to cover such edge cases, the following actions are being taken: * **\[IN-PROGRESS\]** Broaden existing change-review rules for updates that could affect platform access. * **\[IN-PROGRESS\]** Extending our automated policy validations and guardrails to detect and block these edge cases before rollout.