Geoscape Australia incident
Address Verification issues - PSMA Cloud and Address API
Resolved after 2 hours 4 minutes, affecting Addresses API, on March 18, 2019.
Background The deployment of updated PSMA data to the PRODUCTION environment involves the 'switching' of database tables containing previous data to tables containing new data. This typically does not cause any perceivable outage to services for customers. The tables should have associated indexes which maintain search performance. Incident Description The deployment of updated GNAF data on 18 March initially appeared to be successful but shortly after the deployment, address verification services began to degrade significantly to failure. On investigation it was found that the tables containing the new data did not have their associated indexes which resulted in unacceptably poor performance. An attempt was made to create the indexes on the tables in-situ, however, this was unsuccessful due to active sessions on the database effectively 'locking' the table and preventing the creation of indexes. A decision was made to restart the database which dropped all active sessions and allowed the indexes to be created. Shortly after the restart, all address verification services returned to normal expected operation. Lessons learnt and actions A step has been added to the new data deployment process to confirm required indexes exist on tables before they are deployed to any environment. The deployment of new data took place during business hours. This was not normal practice but was done as the deployment was way overdue. This incident highlights the importance of conducting deployments out-of-hours to minimise disruption to customers in the event of unexpected issues.
Affected components
Update timeline
- investigating Mar 18, 2019, 04:31 AM UTC
PSMA Cloud and Addresses API are experiencing address verification issues (no response). We are currently investigating and hope to have all services back online ASAP.
- resolved Mar 18, 2019, 06:36 AM UTC
Address verification services have now been restored. The post mortem details for this incident will be updated in the next 24hrs.
- postmortem Mar 20, 2019, 01:10 AM UTC
Background The deployment of updated PSMA data to the PRODUCTION environment involves the 'switching' of database tables containing previous data to tables containing new data. This typically does not cause any perceivable outage to services for customers. The tables should have associated indexes which maintain search performance. Incident Description The deployment of updated GNAF data on 18 March initially appeared to be successful but shortly after the deployment, address verification services began to degrade significantly to failure. On investigation it was found that the tables containing the new data did not have their associated indexes which resulted in unacceptably poor performance. An attempt was made to create the indexes on the tables in-situ, however, this was unsuccessful due to active sessions on the database effectively 'locking' the table and preventing the creation of indexes. A decision was made to restart the database which dropped all active sessions and allowed the indexes to be created. Shortly after the restart, all address verification services returned to normal expected operation. Lessons learnt and actions A step has been added to the new data deployment process to confirm required indexes exist on tables before they are deployed to any environment. The deployment of new data took place during business hours. This was not normal practice but was done as the deployment was way overdue. This incident highlights the importance of conducting deployments out-of-hours to minimise disruption to customers in the event of unexpected issues.
Your product runs on more than Geoscape Australia
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services