Ontraport experienced a major incident on August 10, 2022 affecting Isolated Accounts, lasting 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 10, 2022, 04:08 PM UTC
We are currently investigating a database issue that may result in account/service disruption for some accounts.
- identified Aug 10, 2022, 04:19 PM UTC
We have identified the issue and are verifying fix.
- monitoring Aug 10, 2022, 04:36 PM UTC
Services have been restored, we will continue to monitor and address any lingering issues as they appear or are identified.
- resolved Aug 10, 2022, 04:56 PM UTC
The systems and services look like they are working normally, and no additional errors are being observed or reported.
- postmortem Aug 10, 2022, 04:57 PM UTC
At approximately 08:58 PST, one of our database servers experienced an unexpected restart event. This resulted in an isolated number of accounts losing some account functionality. The infrastructure team was alerted immediately to the loss and began standard recovery procedures, which resulted in services being restored fully by 09:28 PST. The nature of the event, while understood, has a yet unknown cause. Issue tickets with the hardware vendor have been opened and we will continue to investigate the root cause. Logs, monitoring, and hardware events do not indicate the source of the restart; as such, this will be continued investigation until we are satisfied that we understand the full sequence of events.