Ontraport incident
Increased error rates and slowness for isolated accounts
Ontraport experienced a notice incident on October 16, 2023 affecting Isolated Accounts, lasting 66d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 16, 2023, 04:30 PM UTC
We are currently investigating an issue that is impacting isolated accounts which result in error messages and slow loading of content. We are aware of the underlying issue and are working to find a fix.
- identified Oct 16, 2023, 05:36 PM UTC
We are working on a fix for this and have a path forward. Updates to follow as timelines for implementation are identified.
- monitoring Oct 16, 2023, 07:45 PM UTC
The fix is being applied and we will be observing if it fully resolves the issue.
- monitoring Oct 16, 2023, 10:25 PM UTC
We are continuing to monitor for any further issues.
- monitoring Oct 17, 2023, 01:54 AM UTC
The initial fix reduced incidence of problems, but did not fully resolve the issue. Efforts to address this are continuing.
- monitoring Oct 17, 2023, 02:37 AM UTC
Around 23:00 Pacific time, we will be doing some triage that will result in a couple minutes of downtime to address this; while our initial fixes did reduce impact, they were not complete, so some emergency maintenance will be necessary.
- monitoring Oct 17, 2023, 04:27 AM UTC
The majority of issues have been resolved, but if you believe your account to be impacted by this service issue please contact [email protected] for further investigation. Efforts will continue until complete resolution.
- monitoring Oct 17, 2023, 06:30 AM UTC
The majority of issues have been resolved. If you believe that your account is impacted, please contact [email protected]. This issue will be left in "minor" and "monitoring" states until we are certain that per-account issues have been resolved.
- monitoring Oct 17, 2023, 03:12 PM UTC
We are continuing to monitor the impacted accounts.
- identified Oct 18, 2023, 10:27 AM UTC
Currently addressing a serious issue with some underlying database hardware.
- identified Oct 18, 2023, 10:46 AM UTC
We are bringing some degraded services back online currently. Resolution is expected soon.
- identified Oct 18, 2023, 11:26 AM UTC
The impacted database services are currently undergoing automated verification.
- monitoring Oct 18, 2023, 11:31 AM UTC
Services have been restored, we will continue to observe performance and stability.
- monitoring Oct 18, 2023, 01:02 PM UTC
Services appear to be running optimally. Resolving this issue, but will continue to monitor for any changes.
- resolved Dec 21, 2023, 08:28 PM UTC
This issue has long been resolved; the "monitoring" status was left up erroneously.