Pay & Connect experienced a major incident on November 1, 2019 affecting Pay & Connect Service, lasting 4h 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 01, 2019, 06:30 AM UTC
A breaking change was made by our partner which has resulted in voucher issuances failing today
- identified Nov 01, 2019, 06:33 AM UTC
We are continuing to work on a fix for this issue and will let you know when the issue has been resolved.
- identified Nov 01, 2019, 08:27 AM UTC
We are continuing to work on the issue and will provide a further update as soon as we can. This issue only affects voucher issuance and meal entry scanning remains unaffected.
- monitoring Nov 01, 2019, 09:43 AM UTC
A fix has been implemented and we're monitoring the results.
- resolved Nov 01, 2019, 11:22 AM UTC
This incident has been resolved.
- postmortem Nov 01, 2019, 04:30 PM UTC
This morning we experienced an interruption to the voucher issuance service due to a breaking change on one of our third party integrations that was deployed late last night. The change hampered our ability to successfully process transactions. Our service provider has since to have fixed the issue and the system was back to normal at 10:44 am, in time for the lunch rush. The voucher issuance cutoff had to be extended to 1pm to allow students the opportunity to issue themselves vouchers for the day. We would like to sincerely apologise for the inconvenience caused as a result of this incident. We have been assured that the issue has been resolved, but we are following up for more information regarding the details of the recent deployment. We will continue to monitor the situatuon closely, and have requested that mitigating measures are taken in collaboration with us in order to prevent such incidents going forward. Thank you for your patience and understanding.