Cova Outage History

Started Jun 04, 2025, 02:10 PM UTC· resolved Jun 04, 2025, 02:30 PM UTC

1. identified Jun 04, 2025, 02:10 PM UTC

   We have identified an issue were there is a delay in some invoices from the POS processing to the cloud for HUB reporting. Our teams are investigating and we will provide updates as we have an estimate when the backlog of sales will be processed. The POS and front end services have no impact and all delayed orders will be processed once services have normalized.

2. resolved Jun 04, 2025, 02:30 PM UTC

   We have identified the issue with the service preventing some orders from processing and have applied a fix. Any failed orders are being processed and will post to the HUB momentarily. Thank you for your patience. -Cova Support Team

Read the full incident report →

Started Jun 03, 2025, 04:22 PM UTC· resolved Jun 03, 2025, 07:51 PM UTC

1. monitoring Jun 03, 2025, 04:22 PM UTC

   Hello, We have received notification from BioTrack that they are currently having a disruption with DNS connectivity in New Mexico. The formal statement is below and we will continue to monitor and provide updates as they work to resolve the connectivity issues. Currently Cova customers are unable to access traceability settings in Cova HUB, receive manifests, or process adjustments via API. Sales may also fail to submit to BioTrack, but can be reprocessed from the Traceability Dashboard once BioTrack and the state resolve the DNS problems. Thank you, -Cova Support Team "Dear New Mexico Operators, We’re aware that you may be experiencing issues accessing the patient portal and the trace seed-to-sale system. The disruption is due to the State's DNS. Our team is actively working with the State to resolve this as quickly as possible. We’ll provide an update as soon as access is fully restored. We apologize for any inconvenience and appreciate your patience in the meantime. -The BioTrack Team"

2. resolved Jun 03, 2025, 07:51 PM UTC

   Hello, We've received communication from BioTrack that the issues related to the State's DNS have been resolved. Manifests are now populating in HUB and sales are being submitted to BioTrack as expected. We encourage all New Mexico retailers to check their Traceability Dashboard to verify all sales have been resubmitted from disruption and to contact our support team with any questions. The communication from BioTrack is below for reference. Thank you, -Cova Support Team "Dear New Mexico Operators, Access to the patient portal and the trace seed-to-sale system has been restored. The earlier disruption was related to the State's DNS, which has now been resolved. Thank you for your patience and understanding as we worked with the State to address this. -The BioTrack Team"

Read the full incident report →

Started May 28, 2025, 03:33 PM UTC· resolved May 28, 2025, 03:33 PM UTC

1. resolved May 28, 2025, 03:33 PM UTC

   Good morning, We have received the necessary data for the AGLC Catalog Subscription Service in Alberta and product creation is currently taking place for subscribed companies. Please watch for the email output file to indicate the catalog for your company has been updated. Thank you for your patience and we apologize for any inconvenience this delay may have caused. -Cova Support

Read the full incident report →

Started May 27, 2025, 10:55 PM UTC· resolved May 27, 2025, 10:55 PM UTC

1. resolved May 27, 2025, 10:55 PM UTC

   Hello and greetings from Cova Support. Due to a delay in obtaining the AGLC catalog data this week, the Catalog Subscription service for Alberta is delayed. We will provide an update once the process has been initiated and we apologize for any inconvenience. -Cova Support Team

Read the full incident report →

Started Apr 28, 2025, 03:43 PM UTC· resolved Apr 28, 2025, 07:29 PM UTC

1. identified Apr 28, 2025, 03:43 PM UTC

   Our teams have identified an issue causing errors when trying to navigate pricing, catalog and availability in HUB. We are restarting the impacted services and will provide updates as we have resolution.

2. identified Apr 28, 2025, 04:08 PM UTC

   Although services have improved, we are continuing to see intermittent errors with pricing and availability components in HUB. We are continuing to troubleshoot and will provide additional updates as we are closer to resolution.

3. monitoring Apr 28, 2025, 05:19 PM UTC

   We are continuing to see improved performance with pricing and availability. Our teams are monitoring both services and will provide an all clear once we are comfortable all issues have been resolved.

4. monitoring Apr 28, 2025, 06:06 PM UTC

   Our teams are continuing to investigate transient errors in pricing and availability, however, services have stabilized. We are continuing to troubleshoot any outstanding issues before we resolve this incident.

5. resolved Apr 28, 2025, 07:29 PM UTC

   Our teams have confirmed that the pricing and availability services have returned to business as usual and as such we will be closing this incident. Should we experience any additional issues we will continue to communicate through a new posting.

Read the full incident report →

Started Apr 21, 2025, 06:54 PM UTC· resolved Apr 22, 2025, 06:33 AM UTC

1. monitoring Apr 21, 2025, 06:54 PM UTC

   We are seeing that the Cova Pay issues are beginning to re-occur due to a second round of DDOS attacks against our payment processor and we are reopening this incident to continue looking into this. We apologize for the confusion and inconvenience during this time. We will continue to provide updates as they become available.

2. monitoring Apr 21, 2025, 08:24 PM UTC

   We are still seeing some Cova Pay payments fail to successfully finish on payment terminals and are continuing to monitor the issue. This status incident will be closed once we are more confident that this is no longer occurring and we will continue to provide updates in the meantime.

3. monitoring Apr 21, 2025, 09:25 PM UTC

   We are still seeing some Cova Pay payments fail to successfully finish on payment terminals and are continuing to monitor the issue. This status incident will be closed once we are more confident that this is no longer occurring and we will continue to provide updates in the meantime.

4. monitoring Apr 21, 2025, 11:32 PM UTC

   Although we have not seen the issue resolve at this point, we can see that almost all Cova Pay transactions are going through successfully. We will continue to monitor this issue after end of business day as we are not yet confident this has been fully resolved.

5. resolved Apr 22, 2025, 06:33 AM UTC

   Our payment processor declared the issue fully resolved at 8:30 PM PST, so this incident is now being closed as we believe the issue should no longer be occurring. Thank you for your patience as we waited for a full resolution of this issue.

Read the full incident report →

Started Apr 21, 2025, 05:22 PM UTC· resolved Apr 21, 2025, 06:21 PM UTC

1. identified Apr 21, 2025, 05:22 PM UTC

   We are currently seeing issues with the Cova Pay integrated payment solution. Affected users may see that sales made using the Cova Pay payment method may not go through successfully.The appropriate teams have been alerted and are investigating further.

2. identified Apr 21, 2025, 05:38 PM UTC

   Our payment processor has identified that they are expericing a DDOS (distributed denial of service) attack, which is currently overwhelming their services and is causing the issues we are seeing with Cova Pay. This kind of attack does not expose sensitive information, but instead simply aims to shut down services by making too many requests. We will continue to provide updates as they become available.

3. identified Apr 21, 2025, 05:56 PM UTC

   We are beginning to see more successful Cova Pay transactions occurring but not 100% of attempts are successful yet. We will continue to monitor and provide updates as they become available.

4. resolved Apr 21, 2025, 06:21 PM UTC

   We are no longer seeing any issues related to Cova Pay integrated payments. Thank you for your patience as we waited for this issue to be resolved.

Read the full incident report →

Started Apr 21, 2025, 05:19 PM UTC· resolved Apr 21, 2025, 06:42 PM UTC

1. identified Apr 21, 2025, 05:19 PM UTC

   We are currently seeing some issues related to the Pricing service within Cova. This service affects many other parts of Cova and affected users may see errors when attempting to set prices in the Cova hub, or delays in saved changes being represented in the Cova app. We are aware of the issue and working on a resolution but cannot currently provide an ETA on a solution.

2. monitoring Apr 21, 2025, 05:52 PM UTC

   We have seen that the Pricing issue is mostly cleared up but we are continuing to monitor affected services before calling the all-clear. Thank you for your patience and we will continue to provide updates until we are confident this is no longer occurring.

3. monitoring Apr 21, 2025, 06:35 PM UTC

   We are continuing to monitor Pricing services to ensure that related services are behaving normally again. We believe that the issues are no longer occurring but we are continuing to monitor to ensure service continues smoothly as many users are setting new prices after the weekend.

4. resolved Apr 21, 2025, 06:42 PM UTC

   We believe the issue with Pricing services in Cova is now fully resolved. We apologize for the inconvience and thank you for your patience as we worked to resolve this.

Read the full incident report →

Started Apr 17, 2025, 02:00 AM UTC· resolved Apr 17, 2025, 12:05 PM UTC

1. monitoring Apr 17, 2025, 02:00 AM UTC

   Our monitoring of the pricing service continues to show intermittent errors when accessing the pricing tab via my catalog in HUB. The team will continue to resolve any spikes, however, we will leave this notification active until we are confident any lingering disruptions are resolved. If you receive an error, attempt to reload the page or toggling between products should help populate pricing.

2. resolved Apr 17, 2025, 12:05 PM UTC

   After monitoring the service overnight, our Incident Management team experienced slightly higher error rates around both 8pm and 9pm MDT which were both addressed by moving traffic between data centers. There was no additional alerting beyond the two spikes and the pricing service has normalized. We will continue to monitor throughout the day and provide updates should we see any indications of slowness. Thank you for your understanding as we approached this disruption with an abundance of caution.

Read the full incident report →

Started Apr 17, 2025, 12:36 AM UTC· resolved Apr 17, 2025, 12:47 AM UTC

1. identified Apr 17, 2025, 12:36 AM UTC

   We are again seeing intermittent timeouts when accessing the pricing tab via my catalog. Our teams are taking corrective measures to resolve the issue and we will provide updates as the service returns to normal.

2. resolved Apr 17, 2025, 12:47 AM UTC

   The pricing service slowness has been resolved. Should we see any further impact we will provide additional updates.

Read the full incident report →

Started Apr 16, 2025, 02:12 PM UTC· resolved Apr 16, 2025, 09:54 PM UTC

1. investigating Apr 16, 2025, 02:12 PM UTC

   We are aware of an issue causing slow loading or no loading of the pricing tab in Catalog. We are actively investigating the degraded performance and will update as soon as we know more.

2. monitoring Apr 16, 2025, 03:23 PM UTC

   We have identified the issue with the Pricing service and are seeing some initial improvement. We are still monitoring before declaring the issue resolved. For users still seeing issues loading prices, reloading the webpage will probably help as most requests are going through successfully.

3. monitoring Apr 16, 2025, 05:22 PM UTC

   We continue to monitor the known issue with the Pricing service and while we have seen a large amount of improvement, we have not yet seen a full recovery. Thank you for your continued patience as we wait to see this service behaving at normal levels again.

4. monitoring Apr 16, 2025, 07:03 PM UTC

   We continue to monitor the known issue with the Pricing service and while we have seen a large amount of improvement, we have not yet seen a full recovery. Thank you for your continued patience as we wait to see this service behaving at normal levels again.

5. resolved Apr 16, 2025, 09:54 PM UTC

   We have finished monitoring the Pricing issue from earlier today and we are confident that the service has returned to normal operational levels. Thank you for your patience as we monitored and resolved this issue.

Read the full incident report →

Started Apr 07, 2025, 02:41 PM UTC· resolved Apr 08, 2025, 12:37 AM UTC

1. identified Apr 07, 2025, 02:41 PM UTC

   We have been notified by Vector Payments, one of our Point of Banking providers that one of their processors is experiencing DNS connection errors this morning. This service disruption will impact some, but not all Vector Point of Banking customers. These is no impact to Vector Integrated credit processing. As updates are provided by Vector, we will continue to provide details here. You can also reach out to Vector Payments support at [email protected].

2. identified Apr 07, 2025, 05:55 PM UTC

   We received an update from Vector that the DNS issues has been identified and the processor is taking steps to mitigate the issue. We will continue to provide updates and we monitor this situation and work with our vendor towards resolution.

3. identified Apr 07, 2025, 08:52 PM UTC

   We have not received any additional updates from Vector on when the DNS issue will be resolved with the impacted processor. As we know more we will provide additional updates.

4. resolved Apr 08, 2025, 12:37 AM UTC

   We have received confirmation from the Vector team that the DNS issue has been resolved and all Point of Banking customers are able to successfully process transactions. Any terminals impacted by the service disruption should be rebooted in order to ensure connectivity is restored. Thank you for your patience while the processor worked to resolve this issue.

Read the full incident report →

Started Apr 05, 2025, 06:25 PM UTC· resolved Apr 05, 2025, 06:59 PM UTC

1. investigating Apr 05, 2025, 06:25 PM UTC

   We are currently investigating an issue with our catalog service, which is causing slowness in navigation and product creation. Our teams are investigating and will continue to provide updates as we have more details.

2. identified Apr 05, 2025, 06:30 PM UTC

   We have moved the impacted service to a new data center and are monitoring performance. We will continue to provide additional updates as services return to normal.

3. identified Apr 05, 2025, 06:35 PM UTC

   We are continuing to see intermittent errors in catalog after failing over data centers and are taking additional action to mitigate the derogated performance.

4. monitoring Apr 05, 2025, 06:51 PM UTC

   We are continuing to see improved performance and we will continue to monitor the service as we return to business as usual.

5. resolved Apr 05, 2025, 06:59 PM UTC

   Our monitoring has confirmed that the catalog service has returned to business as usual. Thank you for your patience while we worked to resolve this disruption.

Read the full incident report →

Started Mar 26, 2025, 04:41 PM UTC· resolved Mar 26, 2025, 05:13 PM UTC

1. investigating Mar 26, 2025, 04:41 PM UTC

   We are currently seeing intermittent connectivity to our authentication services preventing login to HUB and other dependent actions within our services. Our teams have been engaged and are working towards a resolution. We will post updates here, but in the meantime POS devices can be logged into using the bubble picker and if necessary work in offline mode to continue to process sales.

2. identified Mar 26, 2025, 04:59 PM UTC

   Our teams are rerouting services to our redundant data center and are monitoring traffic. As we continue to mitigate the issue we will provide updates on estimate return to service.

3. identified Mar 26, 2025, 05:05 PM UTC

   Our teams have identified the root cause and are adding additional resources to remedy the issue. Again, as we have an return we will continue to provide updates.

4. monitoring Mar 26, 2025, 05:08 PM UTC

   We are beginning to see services recover and users able to log into HUB and POS. We will continue to monitor and provide additional updates.

5. resolved Mar 26, 2025, 05:13 PM UTC

   We believe issues related to being able to log into Cova are now fully resolved. Thank you for your patience as we worked to resolve this.

Read the full incident report →

Started Mar 03, 2025, 05:29 PM UTC· resolved Mar 03, 2025, 08:26 PM UTC

1. identified Mar 03, 2025, 05:29 PM UTC

   We have identified an issue with Microsoft hosting services occurring outside of Cova's control that may affect some users. Affected users might have issues logging into Cova. In addition, it's not currently possible to call Cova Support because the routing services are hosted by Microsoft as well. As these issues are occurring with services outside of Cova, we cannot currently provide an ETA on a resolution but will continue to provide updates as they become available.

2. identified Mar 03, 2025, 07:44 PM UTC

   We are seeing much improvement with users logging into Cova, but we are still experiencing issues with users being able to call into the Cova Support Phone line. For anyone needing to reach Support, please email [email protected] or initiate a chat by going to https://support.covasoftware.com/covasupport/s/ and using the Chat functionality in the bottom right corner of the page. Thank you for your continued patience as we await a resolution.

3. resolved Mar 03, 2025, 08:26 PM UTC

   We've received notification from Microsoft that services have returned to normal. Users should no longer experience issues logging into the POS or when trying to contact Cova Support via phone. Thank you for your patience while we waited for services to be restored.

Read the full incident report →

Started Feb 27, 2025, 05:31 PM UTC· resolved Feb 27, 2025, 06:01 PM UTC

1. investigating Feb 27, 2025, 05:31 PM UTC

   We are currently experiencing issues with our authentication provider, which is impacting logging into Cova POS and HUB, among other services. We are working with our authentication provider on a resolution and will provide updates as we have more information.

2. monitoring Feb 27, 2025, 05:35 PM UTC

   Our teams are beginning to see successful logins and systems returning to normal. We will continue to monitor and provide updates as business returns to normal.

3. monitoring Feb 27, 2025, 05:45 PM UTC

   Services are continuing to recover, however, we continue to see some latency. We are continuing to monitor and will provide an all clear once systems have fully normalized.

4. resolved Feb 27, 2025, 06:01 PM UTC

   We are seeing all authentication services behaving as normal again. Thank you for your patience as we worked to resolve this.

Read the full incident report →

Started Feb 13, 2025, 03:55 PM UTC· resolved Feb 13, 2025, 04:42 PM UTC

1. identified Feb 13, 2025, 03:55 PM UTC

   We have become aware of an issue with the login authentication services with Cova that allow users to login. The relevant teams are currently investigating the cause in order to find a solution. We cannot currently provide an ETA on a resolution. Thank you for your patience as we work to resolve this.

2. monitoring Feb 13, 2025, 04:19 PM UTC

   We have seen some improvement with the login services and some users might now be able to login as normal, with some longer loading times as services continue to recover. We will continue to monitor for improvements in the service before this issue is fully resolved.

3. resolved Feb 13, 2025, 04:42 PM UTC

   We are now observing a full resolution on the issues with logging into Cova. Thank you for your patience as we worked to resolve this.

Read the full incident report →

Started Dec 06, 2024, 03:29 PM UTC· resolved Dec 06, 2024, 08:26 PM UTC

1. identified Dec 06, 2024, 03:29 PM UTC

   We have identified an issue caused by the service disruption this morning with the pricing service in HUB. There is a back log of pricing changes that need to process and post to the POS + API's. The teams are adding additional resources to clear the back log and pricing updates will begin to push to the tablets and external services via API. We will continue to update as the back log clears and services return to normal

2. monitoring Dec 06, 2024, 04:23 PM UTC

   Additional resources have been fully ramped up to help process the back log of pricing messages. Retailers should see updates flowing through to the tablet and other services via APT. We expect all services to be caught up within the next 90 minutes. Thank you for your patience as we continue to work through the back log.

3. monitoring Dec 06, 2024, 05:57 PM UTC

   We are continuing to see the back log of pricing messages process and reduce, however, the services are taking longer than expected. We will post additional updates as we have a more concise timeline for the back log to be cleared.

4. monitoring Dec 06, 2024, 07:07 PM UTC

   Our services are continuing to process the back log of pricing messages. More updates will follow as we have an ETR.

5. resolved Dec 06, 2024, 08:26 PM UTC

   Our teams have confirmed all pricing messages in the back log have been processed and we are back to business as usual. We apologize for any inconvenience and thank you for your patience while the teams worked to resolve this issue.

Read the full incident report →

Started Dec 06, 2024, 01:50 PM UTC· resolved Dec 06, 2024, 02:30 PM UTC

1. investigating Dec 06, 2024, 01:50 PM UTC

   We are currently seeing issues when setting or updating prices in HUB. Our teams have been notified and are investigating. We will provide updates when we have identified the issue.

2. identified Dec 06, 2024, 02:07 PM UTC

   We have identified the issue, and teams are working to resolve it and restore all services.

3. monitoring Dec 06, 2024, 02:21 PM UTC

   The issue has been identified, addressed, and we are monitoring services as they return to normal levels.

4. resolved Dec 06, 2024, 02:30 PM UTC

   All services have returned to normal, the root cause has been identified, and additional measures have been taken to prevent future disruptions from this issue.

Read the full incident report →