Dashboard & Widgets Service Outage

Update timeline

1. investigating May 22, 2024, 11:12 AM UTC

   Our dashboard and widgets are currently not responding. We are investigating the issue and will update the status page with more information. Our APIs are still functional and we are still processing orders and points successfully.

2. investigating May 22, 2024, 12:51 PM UTC

   We are still looking into the issue. The root cause is still unknown.

3. identified May 22, 2024, 09:31 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring May 22, 2024, 10:23 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 22, 2024, 10:42 PM UTC

   This incident has been resolved.

6. postmortem May 22, 2024, 10:42 PM UTC

   # Incident summary Between approximately 10:30 UTC and 22:30 UTC on May 22, the dashboard and widgets experienced a major outage. The event was triggered by a deployment at 10:15 UTC. The deployment contained a code change for a repair to the sign up process for collaborators. The deployment triggered a package update in the package tree of a third party package we use for error logging. We mitigated the event by upgrading the third-party package to a later major version. # Lead up At approximately 10:30 UTC, we started receiving reports from users and error spike alerts about a potential issue with the dashboard. We initially thought this was a partial outage but due to caching rules, the event become more severe as time went on and cached responses expired. We had recently deployed a code change at 10:15 UTC to fix an issue with out collaborators feature, where collaborators were unable to create user accounts. # Impact The dashboard app was the single app that failed in this event. However, the widgets relied on an endpoint that the dashboard serves to get data required for initializing all widgets which therefore meant all widgets were inaccessible. Our backend remained unaffected and therefore we were still processing events from API, Shopify and BigCommerce successfully. As such, points, stamps and referrals were being awarded. # Fault We started seeing certain pages within the dashboard returning HTTP 500 status code but without any HTML. As time went on and cache rules cleared, more and more dashboard pages started going under. This eventually also caused widgets to go down due to an initialization network request that is called for widget setup. The endpoint is hosted on the same project and codebase as dashboard. # Response The engineering team started trawling through logs to track down the error. Since this issue affected our error logging tool, it became difficult to diagnose the problem as quickly as we would like. This lead us to believe it may have been a problem with our infrastructure provider and we started opening communications with them. After helping us identify the issue with our error logging tool’s third party package, we quickly moved to upgrade the package. We were unable to identify why the package was causing this error as the issue had been patched in a previous minor release. We moved to upgrade the package to the next major version, which saw a rewrite of their codebase which removed the use of the third-party package. # Recovery At 21:45 UTC, we began working on upgrading the package version to the next major revision which we then deployed at 22:10 UTC. By 22:30 UTC, our monitoring indicated we were no longer serving 5XX HTTP status codes and dashboard and widgets were fully operational again. We are actively working on looking into how our error logging tool was able to cause a catastrophic downtime and are reviewing other third-party tools that we are utilising within our projects at Influence.io[.](http://Influence.io) # Final Notes We want to sincerely apologise for the extended period of downtime experienced. We are working hard to ensure this matter does not happen again and are striving for a more reliable platform with minimal unexpected downtime.