RingCentral Engage Digital incident

Facebook has issue with it's Page API, some page synchronization is failing

Minor Resolved View vendor source →

RingCentral Engage Digital experienced a minor incident on March 13, 2020 affecting Facebook page synchronization, lasting 4d 22h. The incident has been resolved; the full update timeline is below.

Started
Mar 13, 2020, 07:45 PM UTC
Resolved
Mar 18, 2020, 05:48 PM UTC
Duration
4d 22h
Detected by Pingoru
Mar 13, 2020, 07:45 PM UTC

Affected components

Facebook page synchronization

Update timeline

  1. identified Mar 14, 2020, 01:30 PM UTC

    Bug has been reported to Facebook, it seems to be widespread worldwide, Facebook has not reacted, main ticket for reference is https://developers.facebook.com/support/bugs/643788149784639/ The issue on Facebook side started March 13 at 12:44am PST. It seems to affect about 25% of pages. We are waiting for Facebook to fix the issue. Outgoing comments from Engage Digital are working, new import of Post and comments are disrupted by the faulty API. Meanwhile direct use of the Facebook page is working as a workaround. We apologize for the inconvenience.

  2. identified Mar 14, 2020, 06:50 PM UTC

    Facebook responded and is escalating the bug internally. Still no ETA to resolution. You can follow their support here https://developers.facebook.com/support/bugs/643788149784639/

  3. identified Mar 16, 2020, 12:34 PM UTC

    Given that there is not much activity from Facebook on their API bug, we are currently carefully deploying an experimental workaround in order to restart the synchronization of Facebook sources. We are rewriting part of our Facebook synchronization engine to use different API endpoints that seems to be less affected by the Facebook API endpoint bug. It's going ok so far but it seems that the Facebook bug is also impacting /feed api endpoint (user posts on wall) on a smaller set of sources (the ticket was about /posts api endpoint which suffer from the highest number of failure). Therefore we need a second workaround in order to restart the synchronization of some FB Page. We keep you posted.

  4. identified Mar 16, 2020, 03:35 PM UTC

    The workaround Facebook bug on our side is progressively deployed, only 26 sources are still in error. We keep on adjusting it.

  5. monitoring Mar 17, 2020, 04:31 PM UTC

    Facebook did not update https://developers.facebook.com/support/bugs/643788149784639/ for a while. Our workaround made most source working decently for 24h now. We still have random failures due to FB api that slows down synchronization for ~20 sources but this is working. We are monitoring the situation to ensure it remains stable.

  6. monitoring Mar 18, 2020, 10:14 AM UTC

    The incident has been resolved on Facebook's side. We are now monitoring the situation to ensure that everything is back to normal.

  7. resolved Mar 18, 2020, 05:48 PM UTC

    No issue was noticed since the resolution of the incident on Facebook's side earlier today. The incident has been resolved.