Maxio experienced a critical incident on July 1, 2024 affecting Main Application and Billing Portal and 1 more component, lasting 4h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 01, 2024, 04:43 PM UTC
We are currently investigating an outage with the Advanced Billing platform
- monitoring Jul 01, 2024, 05:04 PM UTC
A fix has been implemented and the application is back up. We are closely monitoring and will provide another update as soon as possible.
- monitoring Jul 01, 2024, 05:19 PM UTC
While the application is back up, we are not yet fully operational. Background jobs are currently paused. Webhooks, emails, and processing are delayed. We are actively working to bring the application back to full operational status.
- monitoring Jul 01, 2024, 06:38 PM UTC
We are continuing to monitor for any further issues.
- monitoring Jul 01, 2024, 07:35 PM UTC
We have re-enabled background jobs and processing for subscriptions should begin to resume, though delays are still expected. Exports have been paused for the time being, and our engineers are continuing to investigate some slowness within the application. We will provide more information as soon as we have another update to share.
- resolved Jul 01, 2024, 09:09 PM UTC
We are happy to report that the application is back to operating at full capacity. This incident is resolved. During the full outage period, which occurred for a period of time between 12:30 PM EST and 1:01 PM EST, subscriptions attempting to process likely failed and will have encountered a processing error. Our team will remediate any processing errors that occurred for any affected subscriptions, and once the processing errors are cleared, the subscriptions will resume processing as normal and bill for their renewal. No action is necessary to resolve these errors on your end unless our Sr. Product Support Consultant team reaches out directly. API calls, and signups/subscription creation attempts for sites during the full outage period time frame may have failed, and will need to be retried. Our team is happy to provide API logs from this time period for your site(s) if needed. Thank you for your patience as our team worked to resolve this matter, and our sincere apologies for the disruption. Should you have any questions or concerns, please contact our Support Team for assistance.
- postmortem Jul 02, 2024, 04:38 PM UTC
On July 1, Maxio’s Advanced Billing module encountered a temporary issue due to an unexpected surge in load. This caused a small number of subscription renewals to fail. The disruption was traced to a specific part of our infrastructure. Upon identification, our team promptly resolved the issue, and full service has been restored. We have initiated a thorough post-incident review. This review will include enhancements to our monitoring systems, reevaluation of certain infrastructure configurations, and other internal process improvements to prevent future occurrences. We are currently addressing all affected subscriptions to ensure their renewals are processed correctly. You do not need to take any action unless our Support team reaches out to you for further assistance. We appreciate your patience and apologize for any inconvenience this incident may have caused. Thank you for your understanding as we work to fully resolve this matter.