Evidos Outage History
Evidos is up right nowPingoru has recorded 50 incidents for Evidos: 22 in the last 12 months and 2 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 83 of 720 (12%) |
| May 2026 | 3 | 10 of 744 (1%) |
| June 2026 | 3 | 5 of 720 (1%) |
| July 2026 | 2 | 2 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 1 | 68 of 720 (9%) |
| October 2026 | 1 | 22 of 527 (4%) |
Incidents, newest first
Major October 1, 2026 Email delivery issues 21h 36m API, Portal
Started Oct 01, 2026, 05:17 PM UTC· resolved Oct 02, 2026, 02:54 PM UTC
- investigating Oct 01, 2026, 05:17 PM UTC
We are currently investigating this issue.
- monitoring Oct 01, 2026, 05:27 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 02, 2026, 02:54 PM UTC
This incident has been resolved.
Notice September 4, 2026 Email Delivery Delays 2d 20h Mail service
Started Sep 04, 2026, 10:12 AM UTC· resolved Sep 07, 2026, 06:18 AM UTC
- investigating Sep 04, 2026, 10:12 AM UTC
We are currently experiencing an issue that is causing delays in the delivery of emails sent from our platform. As a result, emails may take longer than expected to reach recipients.
- investigating Sep 04, 2026, 12:49 PM UTC
We are investigating delays affecting email notifications sent by Signhost. The Signhost platform remains fully operational and document processing is not impacted. The issue is currently limited to email delivery, which may result in notifications reaching recipients later than usual. We are working with our email service provider to resolve the issue as quickly as possible
- identified Sep 04, 2026, 02:28 PM UTC
We have identified that the delays affecting email notifications sent by Signhost are related to a broader Microsoft 365 Exchange Online incident. At this time, the Signhost platform remains fully operational, and document processing, signing, and transaction handling are not affected. The impact is limited to email notifications being delivered later than usual. Recipients may experience delays in receiving notifications. We are continuing to monitor the situation closely together with our email service provider. We will provide the next update on Monday morning.
- resolved Sep 07, 2026, 06:18 AM UTC
The issue affecting email notification delivery has been resolved. Email notifications are now being delivered as expected, with no further delays observed.
- postmortem Sep 07, 2026, 08:29 AM UTC
### What happened? On September 4th 2026, some Signhost email notifications, including signing invitations and status updates, were delivered later than usual. During this period, the Signhost platform remained fully operational. Document signing, transaction processing and API functionality were not affected. ### What caused the issue? Our investigation confirmed that the delays were caused by a broader Microsoft 365 Exchange Online incident affecting email processing and delivery. Microsoft reported delays and “server busy“ errors affecting Exchange Online infrastructure. The issue originated outside of Signhost and was not related to the Signhost application, infrastructure or services. It is important to note that Signhost does not use Microsoft Exchange Online to send email notifications. Our email delivery is handled through SparkPost, our email service provider. More information about the Microsoft incident can be found here: [Exchange Online outage causes email delays and server busy errors](https://www.bleepingcomputer.com/news/microsoft/exchange-online-outage-causes-email-delays-server-busy-errors/) ### What we did? We worked closely with our email service provider to monitor the situation and verify the root cause. Throughout the incident, we continuously monitored Signhost services and confirmed that core platform functionality remained unaffected. Once Microsoft resolved the Exchange Online incident, email delivery returned to normal. We have verified that email notifications are now being delivered as expected, with no further delays observed. We would like to reassure customers that no documents, transactions or customer data were impacted and no action is required on your part.
Notice July 23, 2026 Issues with itsme Identification (AES) verification method. 1h 29m itsme Identification
Started Jul 23, 2026, 12:49 PM UTC· resolved Jul 23, 2026, 02:18 PM UTC
- investigating Jul 23, 2026, 12:49 PM UTC
We are currently experiencing an issue affecting the itsme Identification (AES) verification method within Signhost. As a result, transactions that use this verification method cannot be completed at this time. We are investigating this issue.
- identified Jul 23, 2026, 01:41 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 23, 2026, 01:57 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 23, 2026, 02:18 PM UTC
This incident has been resolved.
Notice July 17, 2026 Issue with iDIN and iDEAL verification methods 53m iDIN, iDEAL
Started Jul 17, 2026, 02:25 PM UTC· resolved Jul 17, 2026, 03:19 PM UTC
- investigating Jul 17, 2026, 02:25 PM UTC
We are currently experiencing an issue affecting iDIN and iDEAL services within Signhost. As a result, transactions that use these methods cannot be completed at this time. Our team is actively investigating the issue and working to restore normal service as quickly as possible. We will provide further updates as more information becomes available.
- identified Jul 17, 2026, 02:44 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 17, 2026, 02:47 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 17, 2026, 03:19 PM UTC
This incident has been resolved.
Major June 18, 2026 Issue with verification method Itsme Identification (AES) 2h itsme Identification
Started Jun 18, 2026, 07:16 AM UTC· resolved Jun 18, 2026, 09:16 AM UTC
- investigating Jun 18, 2026, 07:16 AM UTC
We are currently experiencing issues with the verification method itsme Identification (AES). As a result, signers may be unable to complete the signing process. The itsme Sign (QES) method via CSC Qualified verification is not affected. Our team is actively investigating the issue.
- monitoring Jun 18, 2026, 09:02 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 18, 2026, 09:16 AM UTC
This incident has been resolved.
Critical June 16, 2026 ID Proof - Outage 41m ID Proof, Webforms
Started Jun 16, 2026, 01:27 PM UTC· resolved Jun 16, 2026, 02:09 PM UTC
- investigating Jun 16, 2026, 01:27 PM UTC
We are currently experiencing problems with ID Proof. We are investigating this issue. ID Proof is unable to be used momentarily.
- monitoring Jun 16, 2026, 01:55 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 16, 2026, 02:09 PM UTC
This incident has been resolved.
Notice June 3, 2026 Major outage 2h API, UI / View
Started Jun 03, 2026, 04:55 AM UTC· resolved Jun 03, 2026, 06:55 AM UTC
- investigating Jun 03, 2026, 04:55 AM UTC
Currently investigating an outage of the API and signing service. Transaction creation is not possible. Sorry for the inconvenience. This has our highest attention and we'll keep you updated.
- identified Jun 03, 2026, 05:22 AM UTC
The incident has been identified at our hosting provider, TransIP, causing a temporary outage in service. Recovery work is in progress. For live updates on the incident, please visit https://transnoc.nl.
- monitoring Jun 03, 2026, 05:33 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jun 03, 2026, 05:49 AM UTC
Looks like the fix is holding. We'll keep monitoring the situation.
- resolved Jun 03, 2026, 06:55 AM UTC
This incident has been resolved.
Notice May 21, 2026 Investigating Delays In Transaction Processing 33m API, Portal
Started May 21, 2026, 11:05 PM UTC· resolved May 21, 2026, 11:38 PM UTC
- investigating May 21, 2026, 11:05 PM UTC
Currently we're investigating an issue causing delays in transaction processing. A subset of requests may take longer than expected to complete, and in some cases may time out before processing finishes. We are actively working to reduce the delay and restore normal performance. We will provide another update as soon as we have more information.
- identified May 21, 2026, 11:25 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 21, 2026, 11:28 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 21, 2026, 11:38 PM UTC
This incident has been resolved.
Critical May 5, 2026 Signhost QES - Outage 1h 34m Signhost QES
Started May 05, 2026, 07:07 AM UTC· resolved May 05, 2026, 08:42 AM UTC
- identified May 05, 2026, 07:07 AM UTC
Our provider is experiencing issues. The Signhost QES verification cannot be completed for now.
- monitoring May 05, 2026, 08:11 AM UTC
The issue has been fixed. We are monitoring the situation.
- resolved May 05, 2026, 08:42 AM UTC
Signhost QES is operational again.
Minor April 20, 2026 Verification method itsme is currently experiencing issues 7m itsme sign (qualified signature), itsme Identification
Started Apr 20, 2026, 07:52 AM UTC· resolved Apr 20, 2026, 08:00 AM UTC
- investigating Apr 20, 2026, 07:52 AM UTC
Itsme is currently experiencing a technical issue affecting 25% or more of their transactions since 9:25 AM CET. Their technical teams are actively investigating the problem and working to resolve it as quickly as possible. We understand the inconvenience this may cause and will continue to monitor the situation closely. Updates will be provided as soon as more information becomes available.
- monitoring Apr 20, 2026, 07:57 AM UTC
Itsme has identified the root cause of the issue. Their systems are currently recovering, and their technical teams are closely monitoring stability to ensure a full and safe resolution. We will continue to monitor the situation and share further updates as soon as they become available.
- resolved Apr 20, 2026, 08:00 AM UTC
This incident has been resolved.
Minor April 10, 2026 ID Proof - ID validation fails to start in Signhost Android app 2d 21h ID Proof, Signhost App
Started Apr 10, 2026, 01:29 PM UTC· resolved Apr 13, 2026, 10:35 AM UTC
- identified Apr 10, 2026, 01:29 PM UTC
Some customers may experience issues starting ID validation in the Signhost app on Android devices. Instead of launching the ID validation flow, users are prompted to log in. Our team is working on a fix and we will provide further updates as progress is made. Our signing services are not affected.
- resolved Apr 13, 2026, 10:35 AM UTC
A fix has been released to the Google app store and the deeplink problem is solved. Updating the Android app lets users finish identification.
Critical March 23, 2026 DigiD outage 1h 32m DigiD
Started Mar 23, 2026, 12:29 PM UTC· resolved Mar 23, 2026, 02:01 PM UTC
- investigating Mar 23, 2026, 12:29 PM UTC
There are currently issues with logging into DigiD. Logius is investigating the issue, we will communicate any updates.
- monitoring Mar 23, 2026, 01:38 PM UTC
The issues with logging into DigiD have been solved by Logius.
- resolved Mar 23, 2026, 02:01 PM UTC
The issues with DigiD have been resolved by Logius.
Minor March 17, 2026 Degraded Performance – CSC Qualified Verification 25m itsme sign (qualified signature), ZealiD Qualified, Signhost QES
Started Mar 17, 2026, 09:34 AM UTC· resolved Mar 17, 2026, 09:59 AM UTC
- monitoring Mar 17, 2026, 09:34 AM UTC
We see degraded performance with the CSC Qualified verification method. Some signers may encounter issues completing the signing process. A fix has been implemented, and we are now monitoring the system to ensure full recovery. We will update this status once we confirm that the issue has been fully resolved.
- resolved Mar 17, 2026, 09:59 AM UTC
This incident has been resolved.
Minor February 12, 2026 Issues with signing transactions and downloading documents. 5h 14m API, Portal
Started Feb 12, 2026, 10:06 AM UTC· resolved Feb 12, 2026, 03:21 PM UTC
- investigating Feb 12, 2026, 10:07 AM UTC
We are currently investigating this issue.
- investigating Feb 12, 2026, 10:44 AM UTC
Unfortunately, we have not yet been able to identify the problem. We are continuing to investigate this issue.
- investigating Feb 12, 2026, 11:58 AM UTC
We have identified a possible cause and are in contact with our provider. We are continuing to investigate the issue with the highest priority.
- investigating Feb 12, 2026, 01:00 PM UTC
Unfortunately, we have not yet been able to identify the exact cause of the issue. We are continuing our investigation in collaboration with our hosting provider.
- investigating Feb 12, 2026, 02:05 PM UTC
We are still investigating the issue and have not yet identified a root cause. Downloading documents may fail initially but generally succeeds after several retries. At this time, approximately 3–5% of transactions are impacted. For signing transaction, we are observing an increase in queue length. These queued transactions will be retried once the issue is fully resolved. This incident remains our highest priority, and we continue to work on identifying and resolving the underlying cause.
- monitoring Feb 12, 2026, 03:08 PM UTC
We are currently not seeing any errors and will continue to monitor the situation.
- resolved Feb 12, 2026, 03:21 PM UTC
This incident has been resolved.
- postmortem Feb 18, 2026, 09:45 AM UTC
### What happened? On Thursday, February 12th, around 10:30 CEST, we observed an increase in errors affecting the file download and file preview functionalities. As a result, customers experienced issues downloading documents or receipts and viewing documents during signing. In many cases, the action succeeded after retrying. After resolving the incident, we retried open transactions on our end resulting in under 1% of transactions ending in an error state. In some cases, this may have been caused by transactions that were deleted in the meantime. ### What we did Once the increase in errors was detected, our team immediately started investigating the issue. Initial analysis focused on our application and infrastructure, but after ruling out internal causes, we escalated the issue to our hosting provider. ### What caused the issue? The incident was caused by a misconfiguration at our hosting provider during a migration. Because of this incorrect setting: * Larger data packets within the data center were dropped. * This led to failures on our side when retrieving or previewing files. ### What are we doing next? To resolve the incident, the hosting provider corrected the configuration on the affected switches, fully mitigating the issue. To reduce the likelihood of recurrence and improve detection going forward, the following improvements are being implemented: * Improved monitoring: Thresholds in the monitoring system of our hosting provider that tracks specific error rates will be adjusted so similar issues can be detected earlier. * Faster escalation: We are reviewing escalation procedures with our hosting provider to shorten investigation time for infrastructure related incidents. * Hosting migration: We are migrating our file services to another hosting provider this quarter to have more control over configuration.
Major January 23, 2026 Issues with logging in to Signhost web portal 48m Portal, Auth and login service
Started Jan 23, 2026, 09:21 AM UTC· resolved Jan 23, 2026, 10:09 AM UTC
- investigating Jan 23, 2026, 09:21 AM UTC
Customers are experiencing issues with logging in to the Signhost web portal. After filling in log in credentials an error message is received: "Invalid credentials" We are investigating the issue.
- monitoring Jan 23, 2026, 09:38 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 23, 2026, 10:09 AM UTC
This incident has been resolved.
Major November 25, 2025 Issues with creating transactions in webportal using Onfido verification 1h 38m IDV Sign (Onfido)
Started Nov 25, 2025, 10:33 AM UTC· resolved Nov 25, 2025, 12:12 PM UTC
- investigating Nov 25, 2025, 10:33 AM UTC
We are currently investigating this issue.
- identified Nov 25, 2025, 12:07 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Nov 25, 2025, 12:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 25, 2025, 12:12 PM UTC
This incident has been resolved.
Notice November 18, 2025 Issues when signing with CSC Qualified verification 29m itsme sign (qualified signature), ZealiD Qualified
Started Nov 18, 2025, 12:08 PM UTC· resolved Nov 18, 2025, 12:37 PM UTC
- investigating Nov 18, 2025, 12:08 PM UTC
Signers are experiencing issues when signing with a CSC Qualified verification method. We are investigating the issues.
- identified Nov 18, 2025, 12:31 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Nov 18, 2025, 12:33 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 18, 2025, 12:37 PM UTC
This incident has been resolved.
Critical November 11, 2025 issue with signing transactions 31m API, UI / View, Portal
Started Nov 11, 2025, 02:40 PM UTC· resolved Nov 11, 2025, 03:12 PM UTC
- identified Nov 11, 2025, 02:40 PM UTC
The issue has been identified and a fix is being implemented.
- identified Nov 11, 2025, 02:51 PM UTC
We see a very high peak load and have put our systems in maintenance until this subsides.
- resolved Nov 11, 2025, 03:12 PM UTC
This incident has been resolved.
Critical November 6, 2025 Issues with ID Proof web form 32m ID Proof
Started Nov 06, 2025, 09:20 AM UTC· resolved Nov 06, 2025, 09:53 AM UTC
- investigating Nov 06, 2025, 09:20 AM UTC
We are currently experiencing an issue with our ID Proof web forms where not all text and options are displaying correctly Our provider is investigating the cause.
- monitoring Nov 06, 2025, 09:32 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 06, 2025, 09:53 AM UTC
This incident has been resolved.
Critical October 21, 2025 Issues with ID Proof 2h 17m ID Proof
Started Oct 21, 2025, 07:09 AM UTC· resolved Oct 21, 2025, 09:27 AM UTC
- identified Oct 21, 2025, 07:09 AM UTC
ID Proof validations are not resulting in transactions being created. A cause has been identified with a third party. They are working on a fix.
- monitoring Oct 21, 2025, 09:13 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 21, 2025, 09:27 AM UTC
This incident has been resolved.
Critical September 3, 2025 Issues with creating transactions and accessing the web portal. 1h 12m API, UI / View, Portal
Started Sep 03, 2025, 03:18 PM UTC· resolved Sep 03, 2025, 04:31 PM UTC
- investigating Sep 03, 2025, 03:18 PM UTC
We are experiencing issues with creating transactions and entering the portal. We are investigating the issues.
- investigating Sep 03, 2025, 03:38 PM UTC
We have placed the platform in maintenance as we continue our investigation.
- investigating Sep 03, 2025, 03:38 PM UTC
We are continuing to investigate this issue.
- monitoring Sep 03, 2025, 04:08 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 03, 2025, 04:31 PM UTC
This incident has been resolved.
- postmortem Sep 12, 2025, 01:28 PM UTC
### What happened? On September 2 and 3, 2025, our platform experienced a major service outage that significantly impacted availability and transaction processing. The disruption was caused by a code deployment that introduced a bug resulting in excessive duplicate message processing. This overwhelmed our database connections and led to downtime across both days. ### What we did On September 2, we deployed a change and observed an outage shortly after. The team initiated a rollback to restore functionality. After reviewing the change, we re-deployed the same change later that day, believing the issue was unrelated. However, on September 3, a second major outage occurred around the same time. During this time, all SQL queries across the Signhost services failed due to database saturation. After analyzing message logs, we had to conclude the code change was the root cause. The deployment intermittently triggered bursts of hundreds of thousands of duplicate messages for the same transaction events. These spikes occurred after hours of normal operation, making the issue difficult to detect early. Once confirmed, we permanently rolled back to the previous stable version. The system has remained stable since. ### What caused the issue? The root cause was a still unidentified bug in the change we deployed on September 2, that sporadically generated massive volumes of duplicate messages. These bursts overwhelmed the database, causing complete service outages. The duplicates were not caused by infinite loops but occurred in sudden, high-volume spikes. ### What are we doing next? To prevent similar incidents and improve our deployment safety for these kinds of changes, we are taking the following steps: Deployment safeguards: * Gradual rollout strategy: Implement phased rollouts with real-time monitoring to detect anomalies before full deployment. Monitoring improvements: * Monitoring system: We are transitioning to a more robust monitoring system, which will provide improved insights, faster anomaly detection, and more precise alerting across our infrastructure. Root cause analysis: * Deep dive: Continue investigating the specific bug that caused the duplicate message generation to prevent recurrence in future deployments.
Critical September 3, 2025 Issues with creating and signing transactions 2h 9m API, UI / View, Portal, Signhost App
Started Sep 03, 2025, 11:23 AM UTC· resolved Sep 03, 2025, 01:32 PM UTC
- investigating Sep 03, 2025, 11:23 AM UTC
We are experiencing issues with creating and signing transactions. We are investigating the issues.
- investigating Sep 03, 2025, 11:42 AM UTC
We have placed the platform in maintenance as we continue our investigation.
- monitoring Sep 03, 2025, 12:05 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 03, 2025, 01:32 PM UTC
This incident has been resolved.
Notice July 9, 2025 Transaction in status progress 1h 30m API, Portal
Started Jul 09, 2025, 06:59 PM UTC· resolved Jul 09, 2025, 08:30 PM UTC
- investigating Jul 09, 2025, 06:59 PM UTC
We are currently investigating this issue.
- investigating Jul 09, 2025, 07:11 PM UTC
We are continuing to investigate this issue.
- identified Jul 09, 2025, 07:49 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 09, 2025, 08:01 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 09, 2025, 08:30 PM UTC
This incident has been resolved.
- postmortem Jul 11, 2025, 03:01 PM UTC
### What happened? On July 9, 2025, between 20:59 and 22:01 CEST, our platform experienced a service disruption that impacted availability. During this time, customers were unable to access the platform, and transactions were temporarily halted. ### What we did Our team noticed an increase in specific errors on the platform and immediately began investigating the root cause. We observed that queues were building up rapidly, and the platform was not processing the signing of transactions as expected. To prevent further degradation and ensure data integrity, we proactively placed the platform into maintenance mode. During this window, we identified a misconfiguration in our new containerized environment. The platform was running solely on one document signing service due to this configuration error. Although a second document signing service was active, it was not receiving any traffic. Once the issue was identified, we corrected the configuration and restored the affected server. The API was restored and operational within approximately 15 minutes, as there were no issues with creating transactions. However, access to the portal and signing transactions remained unavailable. The entire platform was fully brought back online only after the queues had been processed and stability was verified. ### What caused the issue? The root cause was a configuration error introduced during our transition to a containerized environment. Due to a misconfiguration, only one document signing service was actively handling traffic. When that server failed, the second server—although healthy—was not utilized, leading to a service outage. ### What are we doing next? To prevent similar issues from happening again and to ensure service continuity, we are taking the following steps: 1. Audit on configuration: Conducting a thorough audit of our containerized environment configuration to identify and correct any potential misconfigurations. 2. Prominent validation checks: Adding this scenario as a key validation checkpoint in our containerization rollout process to ensure traffic routing and failover mechanisms are correctly configured. Moving to containerized architecture is a critical part of our broader cloud migration strategy. Our ultimate vision is to fully transition to Entrust’s EU hosting infrastructure. This migration will allow us to leverage high-availability capabilities and failover mechanisms to ensure service continuity. Additionally, it will provide a scalable and resilient platform that can meet evolving customer needs while adhering to strict regional compliance requirements to ensure that data resides in designated jurisdictions. We expect to finalize the migration to a fully containerized environment this quarter.
Major June 23, 2025 Issue creating transactions –
Started Jun 23, 2025, 07:30 PM UTC· resolved Jun 23, 2025, 07:30 PM UTC
- resolved Jun 24, 2025, 07:08 AM UTC
We are experiencing an issue with creating transactions in our portal and API.
- postmortem Jun 24, 2025, 04:02 PM UTC
### What happened? On June 23, 2025, between 7:40 PM and 9:40 PM, we experienced an interruption in our API service. Message queues began growing rapidly, indicating that incoming requests were not being handled promptly. Upon investigation, it was determined that two of our containerized application servers were extremely slow in responding or had become effectively unresponsive. While the queued messages were eventually reprocessed, this was done with significant delays. ### What we did Due to limitations in our monitoring, it took some time before the issue was flagged. However, once the first alerts came in, our team quickly began troubleshooting by analyzing logs, reviewing queue metrics, and closely examining the performance of the affected servers. During the investigation, it became apparent that memory issues in the two affected containerized servers were leading to degraded performance. To mitigate the issue, the problematic servers were rebooted. To mitigate the issue, we performed a rollback to our non-containerized platform. This action restored normal performance and allowed the system to begin processing the backlog of queued messages. By 9:40 PM, all services had returned to full functionality, and all delayed messages had been successfully processed. ### What caused the issue? The performance degradation was traced back to memory management issues in two docker containers. These memory issues arose as part of challenges associated with our recent migration to a containerized environment. During this process, resource allocation parameters were not fully optimized for production workloads, leading to memory pressure that caused the containers to slow down significantly. Additionally, our monitoring systems did not flag the incident early enough due to thresholds that did not effectively capture the early signs of degradation. This delay extended the time required to resolve the issue. ### What are we doing next? To prevent similar issues from happening again and to ensure service continuity, we are taking the following steps: 1. Roll Back to Non-Containerized Platform \(Temporary Measure\): To stabilize services in the short term, we have reverted to our non-containerized platform until we have identified and resolved the underlying memory issues within our containerized environment. 2. Fix Memory Issues in Containerized Environment: We are reviewing and optimizing resource allocation parameters, particularly memory limits, to ensure the infrastructure can handle production workloads reliably without degradation. 3. Enhance Monitoring Systems: 1. We will recalibrate monitoring thresholds to detect early signs of resource strain, enabling proactive intervention before any impact on performance occurs. 2. To support this, we are transitioning to a more robust monitoring system, which will provide improved insights, faster anomaly detection, and more precise alerting across our infrastructure. Moving to containerized architecture is a critical part of our broader cloud migration strategy. Our ultimate vision is to fully transition to Entrust’s EU hosting infrastructure. This migration will allow us to leverage high-availability capabilities and failover mechanisms to ensure service continuity. Additionally, it will provide a scalable and resilient platform that can meet evolving customer needs while adhering to strict regional compliance requirements to ensure that data resides in designated jurisdictions.
Critical March 13, 2025 Issues creating and viewing transactions 1h 37m API, Portal
Started Mar 13, 2025, 12:21 PM UTC· resolved Mar 13, 2025, 01:58 PM UTC
- investigating Mar 13, 2025, 12:21 PM UTC
We are currently investigating this issue.
- monitoring Mar 13, 2025, 12:42 PM UTC
A fix is implemented and we're monitoring the results.
- resolved Mar 13, 2025, 01:58 PM UTC
This incident has been resolved.
- postmortem Mar 14, 2025, 01:59 PM UTC
**What happened?** Yesterday, March 13, some of our customers experienced issues with transactions failing to complete and occasional unreliability in our Portal when accepting user input. We understand how frustrating this can be, and we sincerely apologize for any inconvenience caused. **What we did** Our team immediately began investigating by analyzing logs, webserver telemetry, and other platform metrics. At 13:09 CET, we identified and restarted a web application node that was not functioning correctly. This initially resolved the errors. However, we continued monitoring and noticed other issues, such as web connections growing steadily without decreasing as expected. To ensure stability, we removed the problematic node from our load balancer, which restored normal platform operations. **What caused the issue?** The issue stemmed from a release we deployed on Thursday, March 13, at 12:28 CET. One webserver node began experiencing intermittent database connection failures, which led to the errors observed. Over time, these database connection issues persisted, causing further complications. Upon further investigation, it turned out that our routine application reboot process that is fired upon reboot and release did not cleanly close and reopen all connections, causing connections to keep rising and in the end destabilize our platform. **What are we doing next?** To prevent similar issues and ensure uninterrupted service, we are taking the following steps: * **Enhanced monitoring:** Improving our system health monitoring to detect issues faster and reduce resolution times. * **Service provider collaboration:** Working with our service provider to address the root cause of the database connection errors. * **Platform improvements:** We identified some core improvements in how our services reconnect upon reboot. This way continuous releasing and rebooting continues to work as expected: without any platform impact with failovers to our multiple backup nodes. These measures are part of our ongoing commitment to providing a reliable, high-availability platform. We are dedicated to delivering a seamless signing service 24/7 and will continue to prioritize improvements to meet your expectations. We follow a continuous releasing process that allows us to release multiple times a day and even a week, and we are committed to keep this a flawless and fast process to enhance and secure our platform without any scheduled, let alone accidental, downtime. Thank you for your patience and understanding as we work to make our platform even better.