Issue importing contacts with custom fields.

Update timeline

1. identified Oct 24, 2024, 06:15 PM UTC

   We are working on an issue with importing, saving, and creating new contacts in inboxes that use custom fields, as well as creating new fields in your inbox. As a temporary workaround, removing custom field columns from CSV uploads and/or API requests will resolve the issue. App load times may be slower than usual while we resolve the underlying issue.

2. monitoring Oct 24, 2024, 09:09 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 24, 2024, 10:19 PM UTC

   This incident has been resolved.

4. postmortem Oct 24, 2024, 11:11 PM UTC

   ## What Happened An underlying database table hit a primary key limit that had not properly been migrated to 8 bytes. Today, we hit the limit on the maximum primary key for that table. This resulted in inboxes using Custom Fields feature see intermittent failures with various features that tried to update the value of custom fields on their contacts. This included creating contacts via API, uploading contacts to broadcast audience lists, editing contacts in the app, and handling answers to survey questions if the surveys stored data on custom fields. Inboxes that did not use the Custom Fields feature were not impacted. Sending messages with $custom\_field embeded in the message were not impacted. Contact uploads for that did not include custom field column names were not impacted even if they had custom fields, and should have finished accordingly. ## Resolution We have fully migrated the impacted table to use the correct primary key size limit and existing uploads continued as usual and reindex the results. We were able to do this without much disruption, though load times for inboxes with many contacts with custom fields may have been slower than usual during the day. Contact uploads that failed were retried after our database migration completed, so if you had a broadcast audience upload that failed, please double check your audience, as it should have all the contacts \(and their custom fields\) accounted for. Impacted surveys will have properly continued down their execution path. ## Next Steps All our new tables for the past few years already use 8 bytes to represent primary keys where appropriate. However, we plan on migrating all existing integer primary keys to use 8 bytes to avoid this issue in the future for any other tables. Thanks for your patience and trusting Avochato with your contact data, Christopher