Ricoh incident

[EU] Service Outage

Notice Resolved View vendor source →

Ricoh experienced a notice incident on March 24, 2026 affecting Device Management UI (EU), lasting 7d 12h. The incident has been resolved; the full update timeline is below.

Started
Mar 24, 2026, 05:00 PM UTC
Resolved
Apr 01, 2026, 05:10 AM UTC
Duration
7d 12h
Detected by Pingoru
Mar 24, 2026, 05:00 PM UTC

Affected components

Device Management UI (EU)

Update timeline

  1. investigating Mar 24, 2026, 05:00 PM UTC

    We are receiving a high number of alerts. The Device Management UI might be inaccessible. We are investigating the issue with the highest priority.

  2. monitoring Mar 25, 2026, 05:20 AM UTC

    Incident Summary An incident occurred in which the CloudStream DM service became inaccessible and returned 404 Not Found errors. This issue occurred after a tenant was added or removed and affected tenants within the same region. The service did not recover automatically. The issue has been resolved, and the service is currently operating normally across all production environments. Root Cause The issue was caused by an inconsistency in the routing configuration following a recent infrastructure change. When tenant configuration changes (such as tenant addition or removal) occurred, part of the routing information was not updated as expected, resulting in external requests not being correctly routed to backend services. Resolution The routing configuration handling for tenant changes has been corrected, and the fix was deployed to all production environments at 00:16 on March 25 (UTC). This update ensures that routing information is always properly updated when tenant changes occur. The fix has been validated, and the stability of the service has been confirmed. Preventive Measures To reduce the risk of recurrence, we are strengthening safety measures for updates that involve configuration changes. Specifically, we have improved the reliability of configuration update processing during tenant additions and deletions, expanded mandatory verification items during releases, and enhanced monitoring and alerting for operations with significant impact. Taking this incident as an opportunity, we will continue to improve both the resilience of the platform and our incident response on an ongoing basis.

  3. resolved Apr 01, 2026, 05:10 AM UTC

    Following the deployment of the fix, the service remained stable throughout the monitoring period, with no recurrence of the issue observed.

  4. postmortem Apr 17, 2026, 05:36 AM UTC

    Following this incident, we conducted a detailed investigation to identify the root cause and define preventive measures to avoid recurrence. A full outage report, including the root cause analysis and implemented preventive actions, is available at the link below: [https://supportsite.eu.cloudstream.ricoh.com/wp-content/uploads/2026/04/EU-CloudStream-DM-All-Tenant-Service-Outage-Report.pdf](https://supportsite.eu.cloudstream.ricoh.com/wp-content/uploads/2026/04/EU-CloudStream-DM-All-Tenant-Service-Outage-Report.pdf)