Aptoma experienced a major incident on September 23, 2024 affecting DrPublish - GUI and DrPublish - API /io, lasting 4h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 23, 2024, 07:57 AM UTC
We are currently investigating this issue.
- investigating Sep 23, 2024, 08:26 AM UTC
We are still investigating this issue, and we expect the incident to last for some more time. We'll post updates immediately when we have new information, and we'll update you every 30 minutes regardless of status.
- identified Sep 23, 2024, 08:56 AM UTC
The issue has been identified, and we are working to resolve it. It will take some more time for all systems to fully recover.
- monitoring Sep 23, 2024, 09:05 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Sep 23, 2024, 10:13 AM UTC
All DrPublish users should expect DrPublish to function as normal. We are still monitoring the situation.
- resolved Sep 23, 2024, 12:18 PM UTC
This incident has been resolved. We will later publish a public postmortem here once we have concluded.
- postmortem Sep 26, 2024, 09:48 AM UTC
Our investigations into the incident Monday revealed a certain use of DrPublish that caused the front-end to create an exponentially increasing amount of requests back to DrPublish's APIs. This in turn revealed a weakness in the error-logging mechanisms, which further contributed to the problems. These two factors together created a load on the backend beyond what autoscaling could mitigate. This caused a major outage for some customers, and decreased service operations for others. We have now corrected everything that played a direct role in this incident to prevent incidents from the same or similar causes in the future. We are sorry for the problems caused by this incident.