Firetext experienced a critical incident on April 1, 2020, lasting 1h 26m. The incident has been resolved; the full update timeline is below.
Update timeline
- monitoring Apr 01, 2020, 12:05 PM UTC
A temporary outage of the system occurred for approx 6 mins between 12:56 to 13:02. Messages sent or access to the system to this time would have been affected. We are extremely sorry for this unusual blip. The team have resolved the current situation and are monitoring this now.
- resolved Apr 01, 2020, 01:32 PM UTC
The team can confirm this issue is resolved. Increased latency was observed earlier today, resulting in slower than normal distribution to the networks. The team decided to mitigate this by triggering a failover. Whilst the failover occurred in under 1s as expected, dns was still directing traffic to the affected areas. This resulted in a short period of 5xx responses to affected api users, and affected app users seeing a maintenance page. Email to SMS users were unaffected. The team were quick to identify the issue and manually updated the affected dns records thereby resolving the issue. We apologise for any inconvenience during this time. As always, full monitoring will continue by the team.