Signable incident
Uploading documents during the Envelope Send Flow process
Signable experienced a critical incident on May 21, 2024 affecting Web App and Document Processing, lasting 1h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 21, 2024, 03:51 PM UTC
We are currently investigating this issue.
- monitoring May 21, 2024, 05:11 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring May 21, 2024, 05:14 PM UTC
We are continuing to monitor for any further issues.
- resolved May 21, 2024, 05:18 PM UTC
This incident has been resolved.
- postmortem May 28, 2024, 10:28 AM UTC
On May 21, 2024, at 15:22 UTC, a spike in indexing latency in ElasticSearch caused significant delays in document processing. This latency increase led to a backlog in the job queues, which impacted the overall performance of our app. After manually scaling several parts of our infrastructure, all job queues were fully processed and normal service resumed at 17:12 UTC. # Customer Impact A severe slowdown was experienced in the following services: * Document processing * Syncing to Elasticsearch * Bulk sends * Envelope expiries and reminders * Integrations with Google Drive and Dropbox * Zapier integration upon signup # Lessons Learned and Improvements To prevent a recurrence, we implemented the following changes: 1. **Queue Prioritisation**: Adjusted the priorities of our busiest queues and increased the priority of essential jobs like Document Processing so a slowdown in ElasticSearch does not impact other core services. 2. **Incident Response**: Improved documentation and procedures for scaling services \(cache, database, search\). 3. **Scaling Enhancements:** Enhanced Redis cluster scaling mechanisms. 4. **Latency Investigation:** Initiated an investigation into the cause of high latency on a specific Elasticsearch shard. We apologise for the inconvenience caused and are committed to improving our services to prevent such incidents in the future. Thank you for your understanding and continued support. For any further questions or concerns, please contact our support team at [[email protected]](mailto:[email protected])