Hi Marley incident

Media message failures

Resolved after 1 day 17 hours, affecting Messaging, on February 7, 2023.

resolved Feb 09, 2023, 02:25 PM UTC

A hot fix was released last night and we're experiencing inbound media messages being processed as expected, along with quicker replays for previously failed messages. Some replayed messages may display out of order, but new inbound messages will process and display in your chats as expected.

Started
Feb 07, 2023, 08:43 PM UTC
Resolved
Feb 09, 2023, 02:25 PM UTC
Duration
1d 17h
Detected by Pingoru
Feb 07, 2023, 08:43 PM UTC

Affected components

Messaging

Update timeline

  1. investigating Feb 07, 2023, 08:43 PM UTC

    We've received a few reports of media messages failing at a higher rate than normal. Our team is working on a diagnosis and a fix and hoping to turn that around as soon as tonight. We will provide additional updates once we have them.

  2. identified Feb 07, 2023, 10:35 PM UTC

    We've identified the cause of the issue and a hot fix has been scheduled for tonight. This issue is expected to be resolved by tomorrow morning but we will do some additional monitoring to confirm the fix worked as expected.

  3. identified Feb 08, 2023, 03:42 PM UTC

    We deployed an attempted fix for the increased errors in inbound media handling last night, and in some manual testing today, inbound media message are being received, but there's additional monitoring and work being done to sure up this fix and to verify that high volume of media messages won't be a problem going forward. In addition, previously failed media messages will be replayed automatically.

  4. resolved Feb 09, 2023, 02:25 PM UTC

    A hot fix was released last night and we're experiencing inbound media messages being processed as expected, along with quicker replays for previously failed messages. Some replayed messages may display out of order, but new inbound messages will process and display in your chats as expected.

Your product runs on more than Hi Marley

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →