Partial outage due to external providers

Update timeline

1. investigating Jun 12, 2025, 06:48 PM UTC

   We are currently experiencing service disruptions affecting multiple systems: - User Authentication: Sign up and sign in functionality is currently unavailable - Dynamic DNS API: Experiencing elevated failure rates and intermittent connectivity issues Our engineering team is actively investigating these issues and working to restore full service as quickly as possible. We are monitoring the situation closely and implementing mitigation measures.

2. investigating Jun 12, 2025, 06:49 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 12, 2025, 07:00 PM UTC

   We are pleased to report that all affected systems are returning to normal operation: - User Authentication: Sign up and sign in functionality has been restored - Dynamic DNS API: Service levels have returned to normal with failure rates back to baseline Our monitoring indicates that all systems are now operating normally. We continue to monitor closely to ensure stability.

4. resolved Jun 12, 2025, 07:06 PM UTC

   All systems have been restored to normal operation: - User Authentication: Sign up and sign in functionality is fully operational - Dynamic DNS API: Service has returned to normal performance levels Our monitoring confirms that all affected services are now stable and operating as expected.

5. postmortem Jun 13, 2025, 01:12 PM UTC

   ## Summary On June 12, 2025, Red Sift experienced service disruptions affecting user authentication and Dynamic DNS functionality. Users were unable to sign up or sign in to the platform, and Dynamic DNS experienced elevated response times and API failures. The incident was caused by a widespread Google Cloud Platform Identity and Access Management \(IAM\) service outage that impacted multiple GCP products globally. ## Root Cause The incident was caused by a widespread Google Cloud Platform Identity and Access Management \(IAM\) service outage that began on June 12, 2025. According to [Google's incident report](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW), this affected multiple GCP products globally across all regions. Red Sift's authentication system and Dynamic DNS service \(both API and DNS components\) depend on Google Cloud Platform services that were impacted by this IAM service disruption. ## Impact ### User Authentication * 100% of sign-in and sign-up attempts failed during the incident * Existing authenticated sessions remained functional ### Dynamic DNS * DNS: 95th percentile response time increased from ~13ms to 245ms during the incident * API: Elevated failure rates and degraded performance * Both DNS resolution and API components significantly impacted ### Business Impact * New user acquisitions temporarily halted * No data loss or security breaches occurred ## Timeline All times are in BST timezone. * **18:55** - Service degradation detected * **19:02** - Root cause identified as external cloud provider issue * **19:22** - Incident declared on status page: "Partial outage due to external providers" * **20:00** - Recovery observed * **20:06** - Full service restoration confirmed ## Resolution The incident resolved automatically as Google Cloud Platform restored their IAM services. Red Sift's systems returned to normal operation without requiring any manual intervention on our infrastructure. ## Customer Impact Mitigation * Existing user sessions remained active throughout the incident * All user data and configurations remained intact * No security breaches or data loss occurred * Transparent communication provided through status page updates We sincerely apologize for any inconvenience this outage may have caused. While this incident was due to an external cloud provider issue beyond our direct control, we remain committed to providing reliable service to our customers.