Risotto incident

Intermittent failures due to Slack outage

Major Resolved View vendor source →

Risotto experienced a major incident on February 26, 2025 affecting Dashboard and Agent, lasting 2d 5h. The incident has been resolved; the full update timeline is below.

Started
Feb 26, 2025, 04:44 PM UTC
Resolved
Feb 28, 2025, 10:20 PM UTC
Duration
2d 5h
Detected by Pingoru
Feb 26, 2025, 04:44 PM UTC

Affected components

DashboardAgent

Update timeline

  1. investigating Feb 26, 2025, 04:44 PM UTC

    We are seeing an elevated error rate due to a Slack incident (https://slack-status.com/2025-02/1b757d1d0f444c34). This is resulting in sporadic unresponsiveness from Risotto's Slackbot, as well as repeated messages. We are monitoring and will post updates to this page.

  2. identified Feb 26, 2025, 06:04 PM UTC

    We are seeing some recovery taking place, though Slack is still working to restore service (https://slack-status.com/2025-02/1b757d1d0f444c34)

  3. identified Feb 26, 2025, 08:38 PM UTC

    Slack is still working to restore service (https://slack-status.com/2025-02/1b757d1d0f444c34).

  4. identified Feb 26, 2025, 11:48 PM UTC

    Slack is still working to restore service (https://slack-status.com/2025-02/1b757d1d0f444c34).

  5. identified Feb 27, 2025, 01:23 AM UTC

    Slack has resolved the issue, but we are still getting reports of the Risotto chatbot not functioning correctly. We are investigating.

  6. identified Feb 27, 2025, 04:18 AM UTC

    Slack has reported a separate issue (https://slack-status.com/2025-02/d41e4bfd1ccae26a) that continues to impact Risotto Slackbot functionality. They are reprocessing a backlog of messages, so Risotto may respond late or sporadically. We will continue to monitor the incident and will update this once service is fully restored.

  7. monitoring Feb 27, 2025, 02:07 PM UTC

    Slack has reported that the incident with messaging has been resolved (https://slack-status.com/2025-02/d41e4bfd1ccae26a) and we began to see some recovery of Slackbot functionality around 2 AM Pacific, however we are monitoring and will resolve the alert when Slack resolves the incident completely.

  8. monitoring Feb 27, 2025, 07:45 PM UTC

    Risotto Slackbot continues to operate as expected, despite the continuing Slack incident (https://slack-status.com/2025-02/d41e4bfd1ccae26a). We will resolve this incident once Slack has given the clear.

  9. resolved Feb 28, 2025, 10:20 PM UTC

    This incident is confirmed resolved.