Geoscape Australia experienced a major incident on July 27, 2020 affecting Predictive API and Addresses API and 1 more component, lasting 1h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 27, 2020, 06:11 AM UTC
The issue has been identified as a certificate expiry on api.psma.com.au. A certificate renewal is being implemented.
- monitoring Jul 27, 2020, 06:31 AM UTC
The new certificate has been deployed and the service has been restored. We are monitoring the results.
- resolved Jul 27, 2020, 07:42 AM UTC
This incident has been resolved.
- postmortem Jul 29, 2020, 07:20 AM UTC
## **What happened?** * The Production API services experienced an outage from 3:11 PM to 4:32 PM AEST due to the expiry of the [_api.psma.com.au_](http://api.psma.com.au) SSL certificate. * All API services were unavailable to all API customers. **3:40 pm** * Customers begin reporting an issue with the Predictive address API. Reports suggested \(and later confirmed\) that issues commenced shortly after 3:00 PM. * Service monitoring did not indicate any errors. * PSMA API developers identify and report the certificate has expired. **4:15 PM** * A new certificate was obtained and installation commenced. **4:32 PM** \(end of outage\) * Testing confirms the services have returned to normal. ## **What did we learn?** * We need to improve our certificate management practices. * The service monitoring was operating in a way that masked notifications for SSL certificate expiry. ## **What are we going to do \(have already done\)?** * The service monitoring now checks and send notifications for approaching certificate expiry. * We are creating an accessible register of all certificates in use across our organisation \(not just for APIs\), noting their expiry dates and team ownership. * We are implementing recurring tasks and calendar events to support the renewal of certificates.