Degraded external message streaming in us-east-2

Update timeline

1. identified Mar 09, 2023, 06:04 PM UTC

   At approximately 7:30am CT our team noticed that the queues backing the external log message streaming services had grown to have a substantial backlog. We have increased the size of the job pool associated with the service and are working through the backlog, but users may notice a substantial delay in the delivery of some messages. We will continue to monitor the situation to ensure that it works through the remaining backlog in a timely fashion, and will investigate more sophisticated strategies for handlig the underlying issue(s) that resulted in the accumulation.

2. resolved Mar 09, 2023, 06:43 PM UTC

   The service has successfully worked through the backlog and continues to operate normally. We will continue to watch for alerts on queue depth and take the necessary actions should the problem return. We will also continue work on more sophisticated strategies for mitigating the underlying issue(s) that resulted in the original issue.