Bucks County Community College incident
One Login/Twilio - SMS Outgoing Traffic Impacted
Bucks County Community College experienced a major incident on March 26, 2024 affecting OneLogin - SSO Handler, lasting 23h 59m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 26, 2024, 01:06 PM UTC
Investigating - We have identified the primary cause, taken remediating action and are beginning to see recovery from this incident with queued messages beginning to be sent. We expect to provide another update in 1 hour or as soon as more information becomes available.
- monitoring Mar 26, 2024, 01:19 PM UTC
Monitoring - We are observing an improvement on SMS outgoing traffic. Our engineers are monitoring the issue. We will provide another update in 1/2 hour or as soon as more information is available.
- monitoring Mar 26, 2024, 02:04 PM UTC
We are continuing to monitor for any further issues.
- monitoring Mar 26, 2024, 02:04 PM UTC
Monitoring - We continue observing an improvement on SMS outgoing traffic. Our engineers are still monitoring the issue. We will provide another update in 2 hours or as soon as more information is available.
- monitoring Mar 26, 2024, 02:43 PM UTC
We are continuing to monitor for any further issues.
- monitoring Mar 26, 2024, 02:44 PM UTC
Monitoring - We are continuing to monitor remaining queued messages. Overall, the system has reached a normal steady state; Now, most messages are sending without any additional queueing delay. However, individual customers may see messages remaining in queues due to large numbers of messages enqueued during this incident. We expect any messages delayed during this incident to have fully propagated through to a final status over the course of the next several hours. Note that we expect that some messages will eventually timeout and fail due to Validity Period Expired (error code 30036), others may eventually fail due to a Twilio Internal Error (error code 30500).
- monitoring Mar 26, 2024, 05:14 PM UTC
We are continuing to monitor for any further issues.
- monitoring Mar 26, 2024, 05:14 PM UTC
Monitoring - We continue monitoring remaining queued messages. Overall, the system has reached a normal steady state; Now, most messages are sending without any additional queueing delay. However, individual customers may see messages remaining in queues due to large numbers of messages enqueued during this incident. We expect any messages delayed during this incident to have fully propagated through to a final status over the course of the next several hours. Note that we expect that some messages will eventually timeout and fail due to Validity Period Expired (error code 30036), others may eventually fail due to a Twilio Internal Error (error code 30500). We will provide an update in 2 hours or as soon as more information becomes available.
- resolved Mar 27, 2024, 01:05 PM UTC
This incident has been resolved.