SMX Cybersecure Email experienced a major incident on December 18, 2022 affecting Mail Filtering and SMX Portal, lasting 4h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 18, 2022, 10:22 PM UTC
We are aware of problems affecting the SMX NZ/AU Filtering gateway. SMX technical staff are investigating and will provide a further update in 30 minutes or as new information becomes available.
- investigating Dec 18, 2022, 10:26 PM UTC
We believe the problem relates to corruption of an internal configuration database. We are working to restore the database from a recent backup now. We have manually disabled the SMX Email Admin Portal to prevent writes to the database while it is being restored.
- identified Dec 18, 2022, 10:52 PM UTC
Database restore and re-index is in progress, after which we will perform basic health checking before bringing it back into serivce. We will provide a further update in 30 minutes or when service is restored, which ever comes first.
- identified Dec 18, 2022, 11:13 PM UTC
We have completed the database restore on 1 of 6 nodes, and mail flow through SMX Filtering is now working again, albeit with degraded performance. We are now working to configure replication from the database primary host to the remaining 5 read-only replicas, which will restore the service to full working capacity. There will be further work required to manually recover system changes made between the most recent backup (6AM this morning) and the time at which the database became corrupted.
- identified Dec 18, 2022, 11:15 PM UTC
Access to the SMX email admin portal has been restored.
- monitoring Dec 18, 2022, 11:37 PM UTC
Normal service of the SMX Filtering Gateway has now been restored. We have identified 43 customer configuration changes made between the most recent backup and the outage that we will manually restore over the course of the day.
- resolved Dec 19, 2022, 03:13 AM UTC
This incident has been resolved.