Archbee Outage History

A scheduled migration script acquired an ACCESS EXCLUSIVE lock on the documents table in production PostgreSQL, blocking all reads and writes. The application became unresponsive as connection pool slots filled with queries waiting on the lock. Users experienced timeouts and 5xx errors across the platform for approximately 10 minutes.

We use a hosted/managed database service from Azure. Because we are careful with customer data, and want to fulfill our end to end encryption promise, we use SSL when connecting to any database, this one included. A few weeks ago, we saw a notification about a SSL certificate change in our of our test databases, and we made the SSL certificate change for that specific database. Because we were 100% sure that the same thing will happen soon with our main database, we made a change preemptively while making sure multiple times that the change is backwards compatible, using both certificates in the certificate chain. After deploying the change, our systems went on undisturbed for over two weeks. Today, while having a planned database update from Azure, the system suddenly stopped working due to the new change we had made.