Subsplash experienced a major incident on July 23, 2024 affecting Mobile apps and Subsplash Giving and 1 more component, lasting 2h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 23, 2024, 11:37 AM UTC
We are currently investigating an issue preventing Clients from logging into the Dashboard. Clients may receive an error when attempting to log into the Dashboard. This issue is also affecting other Subsplash services from loading such as Mobile Apps, SnapPages, Giving, and Media & Live.
- investigating Jul 23, 2024, 12:30 PM UTC
We are continuing to investigate this issue.
- identified Jul 23, 2024, 01:06 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 23, 2024, 01:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 23, 2024, 01:48 PM UTC
This incident has been resolved.
- postmortem Jul 31, 2024, 12:48 PM UTC
On July 23, 2024, Subsplash experienced an unexpected infrastructure change to a core networking system. While this networking system was interrupted, many of our Subsplash backend services were unavailable. The Dashboard, Giving, Live, SnapPages, Mobile Apps, and Messaging were among these services. This interruption affected both Clients managing the services and End-Users interacting with them. Recovering the services required manual intervention to properly reconfigure the networking system. We are prioritizing the improvement of our misconfiguration alerts and network resilience.