IDVerse incident

IDKit DNS Caching Issue - IDKit API Clients Using AWS Infrastructure

IDVerse experienced a major incident on August 13, 2024 affecting OCR & DFA Engine API and OCR & DFA Engine API and 1 more component, lasting 2h 30m. The incident has been resolved; the full update timeline is below.

Started: Aug 13, 2024, 12:28 AM UTC
Resolved: Aug 13, 2024, 02:58 AM UTC
Duration: 2h 30m
Detected by Pingoru: Aug 13, 2024, 12:28 AM UTC

Affected components

OCR & DFA Engine APIOCR & DFA Engine APIOCR & DFA Engine APILiveness Engine APILiveness Engine APILiveness Engine APIFace Match Engine APIFace Match Engine APIFace Match Engine API

Update timeline

identified Aug 13, 2024, 12:28 AM UTC

As an unforeseen consequence of routine IDKit infrastructure maintenance between 9:00 PM on August 13th and 1:50 AM AEST on August 14th, we have been made aware of a potential DNS issue for IDKit customers using our API endpoints specifically behind AWS infrastructure. Such customers may be experiencing issues or inability to progress past certain flow stages, especially at the detail check stage with address input and validation. IDVerse technical support and infrastructure teams have investigated and verified that whilst DNS changes have been propagated globally and the product flow stages are currently accessible without error, IDKit clients impacted are instructed to flush their server DNS cache to ensure they are not caching outdated data. IDVerse is presently following up with AWS directly to better understand the nature of this issue Please do not hesitate to contact our support teams for any further questions or concerns
monitoring Aug 13, 2024, 12:46 AM UTC

Our Infrastructure teams have provided further analysis, noting that we have fully identified that lingering DNS caching inside AWS is the definitive cause of DNS resolution errors for customers trying to call the API from inside the AWS network. The issue is resolving as the cache expires inside AWS. From testing, launching new resources (i.e. replacing the EC2 box) should resolve the issue. The correct NS records for api.au.idkit.com should be as follows: ns-1512.awsdns-61.org. ns-2025.awsdns-61.co.uk. ns-209.awsdns-26.com. ns-857.awsdns-43.net.
resolved Aug 13, 2024, 03:28 AM UTC

AWS Support have advised IDVerse Infrastructure and Operations teams that they have now flushed the DNS cache of their internal resolvers. Success rates for connections have now dramatically increased. Combined with prior advice, this issue is now considered resolved. Please reach out to IDVerse Support for any further questions or concerns.