Pronto incident

Notification delay

Resolved after 8 hours 22 minutes, affecting Pronto Platform, on July 22, 2022.

resolved Jul 23, 2022, 12:30 AM UTC

Since our last update we have been carefully monitoring notification delays and the numbers look great and have stayed that way for about 4 hours now. We believe that this issue is now fully resolved and will close this incident. The root cause was that a database upgrade appears to have reset some of the optimizations that were used to ensure fast database access. After reconfiguring the database with the proper optimizations, performance returned to normal levels.

Started
Jul 22, 2022, 04:07 PM UTC
Resolved
Jul 23, 2022, 12:30 AM UTC
Duration
8h 22m
Detected by Pingoru
Jul 22, 2022, 04:07 PM UTC

Affected components

Pronto Platform

Update timeline

  1. investigating Jul 22, 2022, 04:07 PM UTC

    We are once again experiencing delayed notifications, due to some slow database queries. We are working again with our vendor to investigate and will update here as we learn more. We sincerely apologize for the ongoing issue. The team is working diligently to solve this issue for good.

  2. monitoring Jul 22, 2022, 07:57 PM UTC

    After making some more database optimizations all our numbers are looking good again. We will continue monitoring for the next few hours, but for now notifications are back to normal delivery times.

  3. resolved Jul 23, 2022, 12:30 AM UTC

    Since our last update we have been carefully monitoring notification delays and the numbers look great and have stayed that way for about 4 hours now. We believe that this issue is now fully resolved and will close this incident. The root cause was that a database upgrade appears to have reset some of the optimizations that were used to ensure fast database access. After reconfiguring the database with the proper optimizations, performance returned to normal levels.

Your product runs on more than Pronto

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →