Paragon Outage History

Started Feb 13, 2025, 04:52 PM UTC· resolved Feb 13, 2025, 05:08 PM UTC

1. investigating Feb 13, 2025, 04:52 PM UTC

   We are currently investigating this issue.

2. resolved Feb 13, 2025, 05:08 PM UTC

   This incident has been resolved.

3. postmortem Feb 13, 2025, 11:38 PM UTC

   On February 13, 2025, at approximately 8:38 AM PST, a spike in database connections to one of our core services resulted in degraded performance across multiple parts of our system. This led to downtime affecting key product areas, including the dashboard and connect portal. The issue was identified and resolved by our engineering team within a short time frame, and all systems were fully restored. ‌ Please reach out to our Support team if you’d like a full RCA.

Read the full incident report →

Started Feb 05, 2025, 07:35 PM UTC· resolved Feb 05, 2025, 08:45 PM UTC

1. investigating Feb 05, 2025, 07:35 PM UTC

   We are currently investigating this issue.

2. identified Feb 05, 2025, 08:00 PM UTC

   We are currently experiencing an incident where a number of workflows were unexpectedly un-deployed. Our team has identified the affected workflow IDs and is actively working on redeploying them now. We apologize for any disruption this may have caused and appreciate your patience as we work to resolve the issue. We will provide further updates as soon as we have more information. Thank you for your understanding.

3. identified Feb 05, 2025, 08:24 PM UTC

   We've implemented a solution and should have it shortly resolved. We apologize for any disruption this may have caused and appreciate your patience as we work to resolve this. We will continue to monitor the issue and expect the issue to be fully resolved in the next few minutes.

4. resolved Feb 05, 2025, 08:45 PM UTC

   This incident has been resolved.

5. postmortem Feb 11, 2025, 09:23 PM UTC

   # Undeployed Workflows: Public RCA ## **Summary** On Wednesday, February 5th, at approximately 19:10 UTC, we identified an issue where a subset of customer workflows were unexpectedly undeployed. Our engineering team immediately began investigating and restoring affected workflows, with full resolution completed shortly thereafter. ## **Impact** A number of customer workflows were temporarily set to an undeployed state, requiring manual reactivation. Some customers may have noticed disruptions in workflow execution during this period. ## **Root Cause** The issue was triggered by an administrative deployment management action from that did not correctly validate parameters under specific conditions. This resulted in certain workflows being marked as undeployed. ## **Resolution & Response** * Our automated monitoring systems, including BetterStack and Grafana, immediately alerted our team to the issue. * Within 1 minute of the alert, members of our executive team and key engineers began actively investigating the root cause and identifying the full scope of impact. * An internal tool was found as the source and was immediately taken offline. * A status page was posted to notify customers of the disruption. * A dedicated response team quickly developed and deployed a fix to restore all affected workflows using Paragon’s built in version and deployment history. * The issue was fully resolved in a timely manner, ensuring that customers could resume normal operations. ## **Preventative Measures** To prevent this from happening again, we have implemented the following safeguards: * **Enhanced Validation:** We have updated our deployment management process to enforce stricter input validation, preventing similar scenarios. * **Process Audits:** A comprehensive review of related administrative workflows is underway to identify and address potential gaps. * **Ongoing Monitoring Improvements:** We are strengthening our real-time monitoring and alerting systems to further reduce response times. ## **Next Steps** We are committed to continuously improving our platform’s reliability and ensuring minimal disruptions to customer operations. Our engineering team is actively reviewing additional safeguards and process improvements to enhance system resilience. We appreciate our customers’ patience and trust as we work to deliver a seamless, dependable experience. If you have any questions or concerns, please reach out to our support team.

Read the full incident report →

Started Feb 03, 2025, 04:41 PM UTC· resolved Feb 04, 2025, 03:15 PM UTC

1. investigating Feb 03, 2025, 04:41 PM UTC

   We are currently investigating this issue.

2. identified Feb 03, 2025, 06:02 PM UTC

   Cron triggers are currently stuck and not firing. We've identified the issue and looking for a root cause.

3. monitoring Feb 03, 2025, 11:48 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 04, 2025, 03:15 PM UTC

   All cron triggers have been restored with no data loss. Workflow triggers have resumed normal operation.

Read the full incident report →

Started Jan 31, 2025, 07:03 PM UTC· resolved Jan 31, 2025, 07:30 PM UTC

1. investigating Jan 31, 2025, 07:03 PM UTC

   We are currently investigating this issue.

2. monitoring Jan 31, 2025, 07:27 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 31, 2025, 07:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 20, 2024, 08:49 AM UTC· resolved Dec 20, 2024, 04:15 PM UTC

1. investigating Dec 20, 2024, 08:49 AM UTC

   We are currently investigating elevated request error rates to API requests for api.useparagon.com and zeus.useparagon.com. Dashboard views, Connect Portal requests, and other requests to the API may fail to load due to request failures.

2. identified Dec 20, 2024, 09:46 AM UTC

   We have identified a potential issue and are working on implementing a fix.

3. resolved Dec 20, 2024, 04:15 PM UTC

   We experienced a service disruption due to database performance issues that affected our SDK authentication system. The issue was resolved by 2:49am PST, and our team continued to monitor throughout the night to ensure stability. We've implemented additional safeguards to prevent similar incidents in the future.

Read the full incident report →

Started Dec 13, 2024, 05:31 PM UTC· resolved Jan 22, 2025, 03:58 PM UTC

1. investigating Dec 13, 2024, 05:31 PM UTC

   We are currently investigating an issue with our services that may be causing Request-triggered Workflows and Task History views to respond slowly.

2. investigating Dec 13, 2024, 11:07 PM UTC

   We are continuing to investigate this issue and have confirmed that its impact is limited to long-running workflow executions. Task History is not affected by this incident. If you are impacted by this incident, you may see the following error in your workflow as a false positive: “Processing has failed repeatedly due to CPU and memory constraints on the server. Please either refactor your workflow to process less records or contact support.” Replaying the workflow may resolve the issue temporarily for that execution, but the fix we are working on will prevent any false positive occurrences for this error. For almost all customers, this incident is affecting a small number of your total tasks (fewer than 1%). If we have identified your integrations as impacting a high volume of workflow executions, our team will reach out. We will provide an update by end of day Saturday, December 14. If you have any questions or concerns, please reach out to your Customer Success Manager.

3. investigating Dec 15, 2024, 08:16 AM UTC

   We are continuing to investigate this issue and are working on a fix. At this time, this incident is still impacting a small number of total executions, and replaying the workflow may temporarily resolve any issues.

4. monitoring Dec 16, 2024, 04:59 AM UTC

   We have resolved the issue in our systems at this time. We are continuing to monitor to ensure systems remain normal for any affected users.

5. monitoring Dec 19, 2024, 01:43 AM UTC

   We have identified elevated error rates affecting Request-triggered Workflows and Task History views. Our team is actively monitoring the situation.

6. monitoring Dec 19, 2024, 02:26 AM UTC

   We have resolved the issue in our systems at this time. We are continuing to monitor to ensure systems remain normal for any affected users.

7. resolved Jan 22, 2025, 03:58 PM UTC

   This incident has been resolved.

Read the full incident report →