INKY incident

Database misconfiguration causes issues with VIP Spoofing

Minor Resolved View vendor source →

INKY experienced a minor incident on February 14, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
Feb 14, 2022, 03:57 PM UTC
Resolved
Jan 19, 2022, 03:30 PM UTC
Duration
—
Detected by Pingoru
Feb 14, 2022, 03:57 PM UTC

Update timeline

  1. resolved Feb 14, 2022, 03:57 PM UTC

    At 0600 UTC on 16 January 2022, following maintenance a configuration issue resulted in issues for some queries. This resulted in VIP spoofing for some users to not work consistently until the issue was caught and rectified.

  2. postmortem Feb 14, 2022, 03:57 PM UTC

    # Post incident report: Start: 16-January-2022 0600 UTC End: 19-January-2022 1452 UTC Duration: 3 days ## Summary: At 0600 UTC on 16 January 2022, following maintenance a configuration issue resulted in issues for some queries. This resulted in VIP spoofing for some users to not work consistently until the issue was caught and rectified. ## Root Cause: During the rollback of the initial attempt to transfer to a new database system an internal configuration was missed causing some of the queries to fail sporadically. ## Customer Impact: Incoming customer emails during this period may not have been properly checked against the VIP Spoofing list and therefore were not all flagged correctly. ## Mitigation Action: The missed configuration that resulted in the failure was corrected. ## Follow-up Items and Preventative Measures: 1\) A full check of internal configurations will be made after any changes to those records are required for changes and/or rollbacks to changes.