[App/Intranet] [German System] - Delayed Push Notifications to mobile apps

Update timeline

1. resolved Apr 09, 2025, 03:26 PM UTC

   Affected feature: Push Notifications to mobile apps On Monday, April 7th, customers may have observed delays in push notifications sent between 10:00 AM and 11:00 AM CET. Our development team addressed the issue as soon as they became aware of it, and at 11:00 AM CET on Monday, April 7th, all push notifications were sent. There was no loss of push notifications. To check which system your application is hosted on, go to Studio > Settings > API Access and review the tag next to the title. We apologize for the inconvenience. If you have any questions regarding this matter, please contact our support team at [email protected].

2. postmortem Jun 04, 2025, 11:43 AM UTC

   **Summary:** As our investigation of the incident is complete, we would like to share more details about the push notification delay that occurred on April 7th, between 10:00 AM and 11:00 AM CET. ‌ **What happened:** A high number of requests to the notification service caused the service to slow down, leading to an increased response time. Additionally, some of the notification jobs failed, which led to multiple retries, increasing the load on the service even further. ‌ **Impact:** During the specified period, some customers observed unusual delays in sending push notifications to their end users. ‌However, no push notifications have been lost, and as soon as the service stabilized, all notifications were sent. ‌ **Future Improvements:** We are taking steps to implement additional monitoring and alerting to the notification service while making the jobs overall more robust to help prevent job failures and raise awareness for potential issues earlier, aiming to prevent similar incidents in the future. Please contact us at [[email protected]](mailto:[email protected]) if you have any further questions regarding this topic or if any feature is still impacted against our expectations.