Beekeeper incident

Commenting and reacting to posts now fixed, temporarily missing notifications created before Jul 1st

Minor Resolved View vendor source →

Beekeeper experienced a minor incident on July 1, 2024 affecting Data Center (Oregon), lasting 2d 4h. The incident has been resolved; the full update timeline is below.

Started
Jul 01, 2024, 02:07 PM UTC
Resolved
Jul 03, 2024, 06:08 PM UTC
Duration
2d 4h
Detected by Pingoru
Jul 01, 2024, 02:07 PM UTC

Affected components

Data Center (Oregon)

Update timeline

  1. identified Jul 01, 2024, 02:07 PM UTC

    The commenting and reacting features of posts are currently working with limitations. Whenever a user comments or reacts on a post, the user will see an error message and the comment or reaction disappears immediately. The comment and reaction however will be visible to others. We have identified the root cause and working on the resolution. We will send an additional update in 60 minutes.

  2. identified Jul 01, 2024, 02:25 PM UTC

    We are continuing to work on a fix for this issue.

  3. identified Jul 01, 2024, 03:42 PM UTC

    We have restored platform functionality. All stream related features will work normally, including commenting, posting, and reactions to posts. As part of the fix we have temporarily disabled access to old notifications that were created before Jul 1. All new notifications from campaigns, surveys, streams, and chats should work as normal. Notifications created before Jul 1 will not appear in the notification tab, and will be restored by 8 am ET tomorrow.

  4. monitoring Jul 01, 2024, 03:44 PM UTC

    We have restored platform functionality. All stream related features will work normally, including commenting, posting, and reactions to posts. As part of the fix we have temporarily disabled access to old notifications that were created before Jul 1. All new notifications from campaigns, surveys, streams, and chats should work as normal. Notifications created before Jul 1 will not appear in the notification tab, and will be restored by 8 am ET tomorrow.

  5. monitoring Jul 02, 2024, 02:24 PM UTC

    Today we have restored notifications created between June 12th and Jul 1st - those should now appear in the notification tab. During off-peak hours we will restore another batch of notifications created since May 1st. This process should be completed by 8 am ET tomorrow.

  6. monitoring Jul 03, 2024, 12:54 PM UTC

    Today, we have restored another batch of notifications, and the process was completed by 8 am ET. All users now have access to the notifications created after May 1st.

  7. resolved Jul 03, 2024, 06:08 PM UTC

    We have resolved the issue. All users now have access to the notifications created after May, 1st.