brainCloud incident
Logging cluster having troubles keeping up...
brainCloud experienced a minor incident on November 15, 2023 affecting Logging Services, lasting 6h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 15, 2023, 01:54 PM UTC
We are seeing some issues with the performance of our logging cluster. We are temporarily disabling request+response logging in the meantime to alleviate the load on the system. All other logs (errors, server-to-server, login and audit logs) remain fully operational. This has no impact on API services themselves. Apologies for the inconvenience!
- identified Nov 15, 2023, 06:02 PM UTC
We have identified the issue. It relates to a software change to make our logging configuration more flexible - but has an unexpected performance impact when devs attempt to bulk delete users from the App > Design > Admin Tools > Data Deletion page. We have temporarily disabled (i.e. hidden) that page from the Portal - and are working on a fix. We will re-enable request+response logging soon... (after a few other configuration changes are complete).
- resolved Nov 15, 2023, 08:22 PM UTC
Request+response logging has been re-enabled.