Intermittent issues with delivery of event notifications within Order workflows

Update timeline

1. monitoring Sep 13, 2022, 02:31 AM UTC

   We are experiencing intermittent issues with delivery of event notifications for Order workflows between our App and POS partners. Orders submitted to Doshii may experience delays in being received by merchant POS, and subsequently be cancelled. The issue has been identified and we have taken steps to address the issue. We are currently monitoring to ensure it has been resolved.

2. monitoring Sep 13, 2022, 02:55 AM UTC

   We are continuing to monitor for any further issues.

3. resolved Sep 13, 2022, 03:56 AM UTC

   This incident has been resolved.

4. postmortem Sep 16, 2022, 06:33 AM UTC

   At 11.10am on Tuesday 13 September, Doshii’s cloud provider experienced a hardware outage and subsequently initiated a successful fail-over. Twenty minutes after this event, Doshii services were back online and operational, however for those venues using a subset of Point of Sale leveraging real-time socket communication the impact may have been extended, potentially until 12.55pm. During the outage window, orders sent to Doshii were not received by a venue’s POS, resulting in orders being cancelled as per each app partner’s order timeout rules. Some venues may have also experienced a backlog of pending orders being printed at once, up to 70 minutes from the order being placed. This impact was limited to app partners who have not implemented order timeouts in their Doshii integration. We are constantly improving our processes and as a result of this event our teams are extending our monitoring and scaling mechanisms to enable better resilience and recovery of our infrastructure . We are also working with some partners to ensure their integrations with Doshii utilise best practices to minimise venue impacts in situations like this. If you have further questions please contact us at [[email protected]](mailto:[email protected])