CartonCloud incident

Issues with searching

Minor Resolved View vendor source →

CartonCloud experienced a minor incident on December 11, 2023 affecting Web Application Portal / APIs (Asia Pacific) and Search, lasting 16h 12m. The incident has been resolved; the full update timeline is below.

Started
Dec 11, 2023, 02:48 AM UTC
Resolved
Dec 11, 2023, 07:01 PM UTC
Duration
16h 12m
Detected by Pingoru
Dec 11, 2023, 02:48 AM UTC

Affected components

Web Application Portal / APIs (Asia Pacific)Search

Update timeline

  1. investigating Dec 11, 2023, 02:48 AM UTC

    Searching Issues

  2. identified Dec 11, 2023, 03:06 AM UTC

    Searching may be intermittent, we're working to resolve the issue.

  3. identified Dec 11, 2023, 03:26 AM UTC

    Searching may be slower than usual

  4. monitoring Dec 11, 2023, 03:37 AM UTC

    Search and system speed have recovered, we're continuing to monitor for issues.

  5. monitoring Dec 11, 2023, 05:17 AM UTC

    This afternoon/evening we are needing to repair some data within our search system as today's issue caused corruption of some search results. To minimise disruption, we will be reloading data in the following order: Customers, Run Sheets, Suburbs, Consignments. We are now reloading Suburb & Consignment Data.

  6. monitoring Dec 11, 2023, 06:48 AM UTC

    We are repairing some data within our search system as today's issue caused corruption of some search results. We are now reloading Suburb & Consignment Data, while this is occurring searching for these will be impacted and may show no results.

  7. resolved Dec 11, 2023, 07:01 PM UTC

    This incident has been resolved.

  8. postmortem Dec 11, 2023, 08:52 PM UTC

    Refer slack incident thread: [https://cartoncloud.slack.com/archives/C068TG9CTUH](https://cartoncloud.slack.com/archives/C068TG9CTUH) Confluence post mortem: [https://cartoncloud.atlassian.net/wiki/spaces/PDA/pages/232620033/Postmortem\+2023-12-11\+-\+Search](https://cartoncloud.atlassian.net/wiki/spaces/PDA/pages/232620033/Postmortem+2023-12-11+-+Search)