Connect message history reindexing

Update timeline

1. scheduled Jun 17, 2016, 06:48 PM UTC

   Connect messages will be re-indexed from MySQL to ElasticSearch. This will trigger a momentary unavailability of message history (search results and scrollback) and interruption to the delivery of message notifications for a few minutes. Message history will be indexed in reverse chronological order, meaning recent history will be available quickly (with notifications resuming as soon as indexing begins) while the oldest history will be filled in over time.

2. scheduled Jun 19, 2016, 10:59 AM UTC

   Maintenance will begin as scheduled in 60 minutes.

3. scheduled Jun 19, 2016, 11:35 AM UTC

   Changes are going through CI and data is being backed up.

4. scheduled Jun 19, 2016, 11:43 AM UTC

   Backups complete. Another will be taken immediately prior to reindexing.

5. in progress Jun 19, 2016, 12:55 PM UTC

   Production deployment is rolling.

6. in progress Jun 19, 2016, 01:02 PM UTC

   Changes are deployed. Reindexing beginning.

7. in progress Jun 19, 2016, 01:26 PM UTC

   Reindexing is running, is up to 38,000 messages indexed of 737,319. Thus, message history as visible in clients is being filled in. At this import rate, all messages should be available in approximately 2 hours. Unfortunately a bug has been discovered which results in the current impossibility of indexing the message bodies of 2079 messages whose bodies are exceptionally long. The majority of these are messages to rooms related to Nitro development, so the set of affected users is small. Additionally, there is no data loss, so we can resolve the issue once the import has completed. We're working on a resolution for these messages.

8. in progress Jun 19, 2016, 01:59 PM UTC

   Reindexing is up to 287,000 messages, and a fix for lost message bodies has been proposed at https://github.com/powerhome/nitro-web/pull/2428. I have requested peer review of this by John Rose and Wade, and it could be deployed to recover lost message bodies once the reindexing is complete.

9. in progress Jun 19, 2016, 03:11 PM UTC

   Reindexing is complete. We're preparing to recover the ~2000 lost message bodies.

10. completed Jun 19, 2016, 05:56 PM UTC

    Missing message bodies have been restored.