StarRez incident

Degraded Performance - StarRez Cloud Payment Service

Major Resolved View vendor source →

StarRez experienced a major incident on May 21, 2024 affecting Australia East and StarRez Cloud Payment Service and 1 more component, lasting 1d 14h. The incident has been resolved; the full update timeline is below.

Started
May 21, 2024, 07:30 PM UTC
Resolved
May 23, 2024, 10:09 AM UTC
Duration
1d 14h
Detected by Pingoru
May 21, 2024, 07:30 PM UTC

Affected components

Australia EastStarRez Cloud Payment ServiceAustralia SoutheastCanada CentralCanada EastCentral USEast USEast US 2France CentralNorth Central US

Update timeline

  1. investigating May 21, 2024, 07:30 PM UTC

    We are currently investigating an issue with degraded performance with the StarRez Cloud Payment Service as a matter of urgency. -Engineers are actively reviewing this issue with our upstream provider. -Next update expected within the next 2 hours, or as warranted by a change of events.

  2. investigating May 21, 2024, 09:42 PM UTC

    We are currently investigating an issue with degraded performance with the StarRez Cloud Payment Service that is resulting in successful payments not being saved within StarRez. - Engineers are actively investigating the issue. - Next update expected within the next 2 hours, or as warranted by a change of events.

  3. identified May 21, 2024, 10:23 PM UTC

    - Engineers are currently reviewing solutions - Engineers are also building plans for reconciling missing payments within StarRez - Next update expected within the next 2 hours, or as warranted by a change of events.

  4. identified May 21, 2024, 11:03 PM UTC

    We have identified the issue and are are scheduling a deployment to resolve. We anticipate this deployment to occur in the next 180 minutes or soon thereafter. If you would prefer to pause payment processing in PortalX temporarily, please contact StarCare for assistance. Our team is also developing a plan to reconcile any missing payments within StarRez and we will work with impacted customers to assist once complete.

  5. identified May 21, 2024, 11:14 PM UTC

    We are continuing to work on a fix for this issue.

  6. identified May 22, 2024, 01:30 AM UTC

    We have prepared a fix for the identified issue and will commence rolling deployments to affected customers shortly.

  7. identified May 22, 2024, 09:39 AM UTC

    The fix has been rolled out to Week A customers in the following regions: - Canada Central - Canada East - Central US - East US - East US 2 - France Central - North Central US - North Europe - South Central US - Switzerland North - UK South - UK West - West Central US - West Europe - West US - West US 2 The fix will be deployed to Week A customers in the following regions shortly: - Australia East - Australia Southeast - Southeast Asia

  8. monitoring May 22, 2024, 05:27 PM UTC

    A fix for the issue that was identified in 10.9.77, has been rolled out for all impacted customers. At this time we have not seen any related errors since 15:02 UTC, 11:02 EDT, 01:02 AEST. Engineers will continue to monitor to confirm no further issues emerge. We are also developing a plan to work with any impacted customers for reconciliation. We will provide an additional update within the next 8 hours.

  9. identified May 22, 2024, 06:38 PM UTC

    A new issue has been identified that is causing intermittent errors within the Cloud Payment Service. We are actively investigating the errors and will provide an update within the next 2 hours.

  10. monitoring May 22, 2024, 09:48 PM UTC

    The second issue has been resolved. This issue started at 16:48 UTC, 12:48 EDT, 02:48 AEST and was resolved at 18:46 UTC, 14:46 EDT, 04:46 AEST. This would have resulted in intermittent errors when redirected to payment gateways. No reconciliation will be needed for this time period. We are continuing to monitor to ensure stability across the platform. We will provide an additional update within the next 8 hours.

  11. monitoring May 23, 2024, 09:05 AM UTC

    All outstanding issues have now been resolved and payment processing has been observed to be stable since 18:46 UTC 22nd May, 14:46 EDT 22nd May, 04:46 AEST 23rd May. Engineers will continue to monitor to confirm no further issues emerge. We are continuing to develop a plan to work with any impacted customers for reconciliation.

  12. resolved May 23, 2024, 10:09 AM UTC

    All outstanding issues have now been resolved and payment processing has been observed to be stable for over 15 hours since 18:46 UTC 22nd May, 14:46 EDT 22nd May, 04:46 AEST 23rd May. We are continuing to develop a plan to work with any impacted customers for reconciliation.