Geekbot incident

Performance issues reported

Minor Resolved View vendor source →

Geekbot experienced a minor incident on December 9, 2019 affecting API, lasting 3d 18h. The incident has been resolved; the full update timeline is below.

Started
Dec 09, 2019, 07:25 PM UTC
Resolved
Dec 13, 2019, 01:32 PM UTC
Duration
3d 18h
Detected by Pingoru
Dec 09, 2019, 07:25 PM UTC

Affected components

API

Update timeline

  1. investigating Dec 09, 2019, 07:25 PM UTC

    We're looking into reports of performance and lag issues for some customers. We'll keep you in the loop as we continue to investigate this further.

  2. investigating Dec 09, 2019, 07:26 PM UTC

    We are continuing to investigate this issue.

  3. identified Dec 10, 2019, 09:10 AM UTC

    For those affected, we're happy to report that the issue has been identified and a fix has been implemented.

  4. monitoring Dec 10, 2019, 12:35 PM UTC

    A fix has been implemented and we are monitoring the results. Please bear with us a bit longer.

  5. monitoring Dec 11, 2019, 07:42 PM UTC

    A new fix is about to deploy, please bear with us a little longer.

  6. monitoring Dec 12, 2019, 01:25 PM UTC

    We are happy to say that we are seeing signs of improvement. For those affected, we apologize for keeping you away from your work during this time.

  7. resolved Dec 13, 2019, 01:32 PM UTC

    Thank you so much for your patience as we worked on this. Looks like we're finally out of the woods after numerous failed attempts to fix the issue. We tracked down that Slack was not sending requests to the correct events URL which affected the bot's responsiveness and caused performance issues. Once we settled this with Slack's support and ensured that requests are sent to the correct events URL we determined that this had resolved the issue for all customers.