Clearhaus experienced a major incident on August 30, 2020 affecting Gateway API (gateway.clearhaus.com), lasting 4h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 30, 2020, 01:43 PM UTC
We are seeing increased error rates and are investigating.
- identified Aug 30, 2020, 02:15 PM UTC
The Gateway API experienced sporadic timeouts towards upstream service providers approximately 2020-08-30 12:57-13:40 UTC due to a network issue. We have worked around the issue while we are waiting for our network provider to solve it. The MPI has experienced an increase in timeouts towards Visa's directory servers starting at approximately 2020-08-30 10:02 UTC. From approximately 2020-08-30 11:44 UTC and onwards all connections towards Visa's directory servers are experiencing timeouts. We have reached out to Visa. Mastercard’s directory servers are not affected.
- monitoring Aug 30, 2020, 02:49 PM UTC
At 2020-08-30 14:21:56 UTC Visa's directory servers once again became reachable and started responding successfully, and the MPI is no longer experiencing timeout issues. We will continue to monitor the situation.
- resolved Aug 30, 2020, 06:09 PM UTC
The outage was caused by the general internet problems experienced today, August 30th 2020. It appears the problem was caused by a CenturyLink misconfiguration.