Gridly experienced a minor incident on August 29, 2021, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 30, 2021, 04:02 AM UTC
We're experiencing an elevated level of API timeout. This issue has been resolved on infrastructure side.
- postmortem Aug 30, 2021, 04:10 AM UTC
### Timeline on 2021-08-29 UTC * 05:13 AM - Degraded API performance and timeout connections * 05:26 AM - Timeout connections are under control. API is back to normal * 06:15 AM - Deployed hotfix and improvements to production cluster. ### Root cause analysis \(RCA\) * Internal endpoint rotated IP addresses unexpectedly * Our edge proxy servers had timeout connections on networking between services because of expired IPs * Reload proxy layer for re-init connections to latest IPs * Deploy hotfixes & improvements on DNS caching layer for actively update changes from internal endpoints