Resolved after 6 hours 36 minutes, affecting One Church Giving, on March 28, 2023.
**Letter from Ken Keller, the SVP/CTO of Usio** I am writing to follow up on the recent system outage that occurred on March 28. We understand that the outage may have caused inconvenience and disruption to your organization, and we apologize for any inconvenience this may have caused. Here is a summary of the outage and what we have done to address the issue: * The outage occurred at 12:36 PM CT on 3/28/2023 * Our technical team was immediately alerted and began working to resolve the issue. * The system was down for approximately 6 hours, during which time we worked to identify and address the outage: * We identified our internal firewall configuration experienced an automated failover after a faulty deployment release * This failover triggered an unexpected change in IP addresses, leading to mismatched routing and lack of internal service communication * We have taken steps to ensure that similar outages do not occur in the future including: * Availability options for internal services * Strengthening deployment gates * Continued investigation into automated process We recognize that downtime is a serious matter, and we take it very seriously. We would like to assure you that we are implementing enhanced steps to prevent similar issues. We are reviewing our incident response procedures to ensure that we are better prepared for any future outages. We will continue to monitor our systems closely and take proactive steps to ensure that our services remain secure and reliable. Thank you for your continued support. Sincerely, Ken Keller SVP, CTO Usio, Inc.
Affected components
Update timeline
- investigating Mar 28, 2023, 05:38 PM UTC
Online payment processing is currently resulting in a 502 Bad Gateway for some of our customers. Usio, our merchant services provider, is currently investigating this and we'll post additional information as it becomes available.
- investigating Mar 28, 2023, 05:54 PM UTC
We are continuing to investigate this issue.
- investigating Mar 28, 2023, 08:33 PM UTC
Usio is still actively investigating the issue and hopes to have more information soon on the outage.
- identified Mar 28, 2023, 10:29 PM UTC
Usio has identified an issue with their application firewall and an issue with a routing table. They are actively working on rebuilding this resource.
- resolved Mar 29, 2023, 12:15 AM UTC
Usio reports that the application firewall has been re-deployed and is functioning normally at this time. Online payment processing should be working again for all customers. If you experience any continuing issues, please reach out to support.
- postmortem Mar 29, 2023, 06:59 PM UTC
**Letter from Ken Keller, the SVP/CTO of Usio** I am writing to follow up on the recent system outage that occurred on March 28. We understand that the outage may have caused inconvenience and disruption to your organization, and we apologize for any inconvenience this may have caused. Here is a summary of the outage and what we have done to address the issue: * The outage occurred at 12:36 PM CT on 3/28/2023 * Our technical team was immediately alerted and began working to resolve the issue. * The system was down for approximately 6 hours, during which time we worked to identify and address the outage: * We identified our internal firewall configuration experienced an automated failover after a faulty deployment release * This failover triggered an unexpected change in IP addresses, leading to mismatched routing and lack of internal service communication * We have taken steps to ensure that similar outages do not occur in the future including: * Availability options for internal services * Strengthening deployment gates * Continued investigation into automated process We recognize that downtime is a serious matter, and we take it very seriously. We would like to assure you that we are implementing enhanced steps to prevent similar issues. We are reviewing our incident response procedures to ensure that we are better prepared for any future outages. We will continue to monitor our systems closely and take proactive steps to ensure that our services remain secure and reliable. Thank you for your continued support. Sincerely, Ken Keller SVP, CTO Usio, Inc.
Your product runs on more than One Church Software
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services