Mews incident

Downgraded Performance

Resolved after 1 hour 24 minutes, on September 9, 2026.

postmortem Oct 07, 2026, 02:50 PM UTC

**Problem** On 9 September between 21:07 and 21:25 CEST, the Mews platform experienced a performance degradation lasting about 18 minutes. About half of requests to the application and API timed out, including sign-in. Customers saw slow or unresponsive pages and intermittent errors. Some operations kept working, and no data was lost. **Action** Our automated monitoring detected the issue within seven minutes, and our support team was engaged within minutes to help affected customers. We kept customers informed through our status page and monitored until the service fully recovered at 21:25 CEST. We opened a case with our cloud provider the same evening. **Causes** A routine access change to our production database caused the incident. Because of how our tooling re-applied existing permissions, one small change became a large batch of permission updates. That batch caused the database replica that serves read traffic to stall. Our cloud provider has confirmed that the replica should have recovered far sooner than it did, and is working on improvements to how it recovers from this situation. The replica recovered on its own after about 18 minutes, and operations that write data were largely unaffected. **Solutions** * We are changing our database tooling to apply only the specific change being made, rather than re-applying every existing permission. * We are adding timing, announcement and approval controls for production database changes such as this one. * We are making infrastructure changes visible to our incident tooling, so similar changes are identified immediately. * We have added alerting on this type of database change, and are improving alerting on database connection errors. * We are working with our cloud provider on faster recovery from this replica behaviour, and adding a fallback so read traffic can continue if the replica is unavailable.

Started
Sep 09, 2026, 07:30 PM UTC
Resolved
Sep 09, 2026, 08:54 PM UTC
Duration
1h 24m
Detected by Pingoru
Sep 09, 2026, 07:30 PM UTC

Update timeline

  1. investigating Sep 09, 2026, 07:30 PM UTC

    We have noticed a downgraded performance of the system. We are currently looking into it. Thank you for your patience.

  2. monitoring Sep 09, 2026, 07:38 PM UTC

    We're investigating the issue.

  3. monitoring Sep 09, 2026, 07:42 PM UTC

    We've been able to resolve the cause of the incident. The app had around 10-second response time or was unresponsive for the end users. The system is functional, we are working on containing the root cause. More updates soon.

  4. monitoring Sep 09, 2026, 08:19 PM UTC

    We have narrowed the problem to be related to a maintenance event performed by our cloud provider Microsoft Azure on our production database. We are cooperating with them to address the root cause.

  5. resolved Sep 09, 2026, 08:54 PM UTC

    We've been able to identify the cause the performance issues out to be an unplanned migration work on the side of our cloud provider, Microsoft Azure.

  6. postmortem Oct 07, 2026, 02:50 PM UTC

    **Problem** On 9 September between 21:07 and 21:25 CEST, the Mews platform experienced a performance degradation lasting about 18 minutes. About half of requests to the application and API timed out, including sign-in. Customers saw slow or unresponsive pages and intermittent errors. Some operations kept working, and no data was lost. **Action** Our automated monitoring detected the issue within seven minutes, and our support team was engaged within minutes to help affected customers. We kept customers informed through our status page and monitored until the service fully recovered at 21:25 CEST. We opened a case with our cloud provider the same evening. **Causes** A routine access change to our production database caused the incident. Because of how our tooling re-applied existing permissions, one small change became a large batch of permission updates. That batch caused the database replica that serves read traffic to stall. Our cloud provider has confirmed that the replica should have recovered far sooner than it did, and is working on improvements to how it recovers from this situation. The replica recovered on its own after about 18 minutes, and operations that write data were largely unaffected. **Solutions** * We are changing our database tooling to apply only the specific change being made, rather than re-applying every existing permission. * We are adding timing, announcement and approval controls for production database changes such as this one. * We are making infrastructure changes visible to our incident tooling, so similar changes are identified immediately. * We have added alerting on this type of database change, and are improving alerting on database connection errors. * We are working with our cloud provider on faster recovery from this replica behaviour, and adding a fallback so read traffic can continue if the replica is unavailable.

Your product runs on more than Mews

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →