Customer.io incident

Maintenance to improve reliability of our search indexes

Notice Resolved View vendor source →

Customer.io experienced a notice incident on October 1, 2015, lasting 3d 23h. The incident has been resolved; the full update timeline is below.

Started
Oct 01, 2015, 05:30 PM UTC
Resolved
Oct 05, 2015, 05:16 PM UTC
Duration
3d 23h
Detected by Pingoru
Oct 01, 2015, 05:30 PM UTC

Update timeline

  1. scheduled Oct 01, 2015, 05:30 PM UTC

    We'll be taking down an our search indexes to upgrade and reindex our people search index. This will remove your ability to apply filters to your customer profiles. When viewing results from the search index, you'll see a degraded message until your search indexes are restored. We expect all but a few accounts will be fully restored within a few hours, however large accounts may take longer to fully recover.

  2. in progress Oct 02, 2015, 12:00 AM UTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  3. in progress Oct 02, 2015, 09:22 PM UTC

    Upgrading the search indexes is taking longer than expected. Our engineering team is working on speeding up the process, but in the meantime we want to assure you that campaigns and newsletters continue to send to the correct recipients and the incorrect number of users you see when selecting segments for a newsletter is just a display issue. If you want to check on the number of recipients before sending a message, this is still possible by exporting the segments as .csv via the People section. We’re really sorry about the confusion and stress caused by this. We’ll add a new update once the upgrade is complete.

  4. in progress Oct 05, 2015, 06:33 AM UTC

    The search infrastructure upgrade is still in progress. Unfortunately, we cannot speed up the process, but we're happy to confirm most of the accounts are already switched to the improved version and the remaining ones will be too in the following hours.

  5. completed Oct 05, 2015, 05:16 PM UTC

    The search update was completed successfully. You should now be able to filter and search users without any problems. If you continue to see errors or have trouble with any segments or user searches, please let us know at [email protected]