Atlas Device Synchronisation

Update timeline

1. investigating Jul 23, 2026, 05:58 AM UTC

   There is an issue with the Atlas device sync service. Our technical teams are investigating.

2. investigating Jul 23, 2026, 08:14 AM UTC

   We are continuing to investigate this issue as our highest priority. We understand the impact this is having on your teams and apologise for the disruption this is causing. We will provide a further update as soon as we have more information to share.

3. identified Jul 23, 2026, 08:14 AM UTC

   We are continuing to work on the issues with Atlas device synchronisation as a critical priority. Our engineering team are working with our infrastructure partners to restore service as quickly as possible. We sincerely apologise for the disruption to this crucial service and thank you for your continued patience and understanding.

4. monitoring Jul 23, 2026, 09:40 AM UTC

   Device synchronisation is now improving. Customers may get intermittent synchronisation failures whilst the performance scales up. We recommend waiting 5 minutes, then trying again if this happens.

5. monitoring Jul 23, 2026, 10:05 AM UTC

   We are continuing to monitor for any further issues.

6. resolved Jul 23, 2026, 10:41 AM UTC

   The performance issues affecting device synchronisation have now been fully resolved. All services are operating normally. We are sorry for the disruption this caused today. We know that reliability is essential when you are administering medication, and we do not take that responsibility lightly. We will be reviewing what happened to help prevent a similar issue in future. If you have any concerns or continue to notice anything unusual, please do not hesitate to contact our Support team.

7. postmortem Aug 05, 2026, 02:20 PM UTC

   # Atlas Sync Unavailability Following Scheduled SQL Infrastructure Migration **Summary** Scheduled maintenance was carried out on the night of Wednesday 22nd July 2026 to move the Atlas SQL database onto new infrastructure. The migration itself completed largely as planned, but on the morning of Thursday 23rd July the Sync service became unavailable across both the Sync A and Sync B pods, affecting all customers. Care homes were unable to synchronise their devices and, in some cases, were unable to administer medication through the application. Guidance was issued to move devices into offline mode. This was effective for many customers, although its success varied depending on how many devices were in use per unit, some customers reported that the application did not present the offline mode option and instead remained in a continuous loading state. Those customers moved to their paper-based fallback process. From a customer perspective the issue began at approximately 07:00 with poor performance and escalated from 08:00 as morning medication rounds started. The service was fully restored by 10:30 the same morning. **Root Cause** The root cause was that the newly provisioned SQL infrastructure was still building up caches and under the live production workload a disk throughput bottleneck appeared once the full morning load was applied to the new environment. The new servers had a different disk layout to the original servers which made this issue more difficult to resolve under full load. Adjusting this then triggered a failover which we were not ready for yet after the migration, a contributing factor to this was the Capa printing issue which had taken priority. This extended the resolution time. **Resolution** The bottleneck was identified at 08:30 as a disk capacity and throughput constraint. Disk capacity was increased first, which did not resolve the issue, so the database load was redistributed across multiple disks as well as increasing the memory cache; this work completed by 10:00. This work triggered the partial failover described above and a short period of login failures. These were resolved by 10:30, at which point the Sync service was fully restored and operating normally. Records entered while devices were in offline mode synchronised successfully once the service was restored. **Going Forward - Preventative Measures** PCS is committed to ensuring that future infrastructure changes of this type do not result in customer-facing disruption. The following measures have been taken or are in place: * Future infrastructure migrations will be deliberately over-provisioned at cutover, with capacity monitored and tuned back down once the environment has proven stable, rather than being sized to a planning estimate. The short-term additional infrastructure cost is accepted as the correct trade-off against customer impact. This approach will be applied to comparable future work. * High availability configuration is now fully in place on the new infrastructure, so that scaling operations no longer risk a partial failover. * Database consistency checks have been moved entirely off the primary server onto the secondary server, and overlapping scheduled jobs have been re-timed so that they no longer collide. Backups are also now taken from the secondary server rather than the primary. This reduces load at key points in the day and was one of the goals of the new infrastructure. * The behaviour where the application did not present the offline mode option on some devices is being investigated as a separate product improvement, so that the offline fallback is dependable in any future service interruption. The ability to distribute database workload across multiple servers was not available on the previous infrastructure. Although the migration caused this disruption, it has provided the resilience and capacity headroom that the previous environment could no longer offer. **Timeline of Events** | **Date** | **Event Description** | | --- | --- | | **22nd July 2026, 23:00** | Scheduled maintenance began to move the Atlas SQL database onto new infrastructure. The migration completed largely as planned. | | **23rd July 2026, 06:55** | Internal alert raised on the performance issues channel; first customer reports began arriving through support. | | **23rd July 2026, 07:00** | Start of customer-facing impact. Sync performance reduced across both Sync A and Sync B pods. | | **23rd July 2026, 08:00** | Reports escalated sharply as morning medication rounds began. Incident escalated internally and guidance issued to customers to move devices into offline mode. | | **23rd July 2026, 10:00** | Root cause identified as a disk capacity and throughput bottleneck. Disk capacity increased, Database load redistributed across multiple disks. Memory limit doubled. | | **23rd July 2026, 10:30** | Login failures resolved and the Sync service fully restored. Normal service resumed. | We apologise for the disruption this caused and thank you for your patience and collaboration while we investigated and resolved the issue.