InEvent experienced a critical incident on March 9, 2023 affecting UK Server Region, lasting 1h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 09, 2023, 10:00 AM UTC
We are investigating an issue with the European Servers that are causing a response delay and Gateway Timeouts on certain requests.
- identified Mar 09, 2023, 10:05 AM UTC
We have identified a DNS issue that caused our server requests to queue and severely delay their responses on API requests. The API microservices had to wait more than anticipated for a DNS resolution and that caused timeouts with a Gateway Timeout HTTP error. We have deployed another DNS server to be used and this may take up to 30 minutes to take effect and requests starts to normalize. We will be monitoring the resolution development.
- monitoring Mar 09, 2023, 10:09 AM UTC
The DNS servers are now responsive and our Load Balancer is now draining pending requests and flushing old requests in favor of new ones. New API calls will be handled normally and we will be monitoring the results.
- monitoring Mar 09, 2023, 10:19 AM UTC
We are continuing to monitor the servers. The API calls are now 100% responsive and without degraded performance.
- resolved Mar 09, 2023, 10:25 AM UTC
This incident has been resolved. Our European Servers are fully operational.