Uploadcare experienced a critical incident on July 19, 2019 affecting CDN, lasting 3h 35m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 19, 2019, 03:59 PM UTC
The issue has been identified and a fix is being implemented.
- identified Jul 19, 2019, 05:50 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Jul 19, 2019, 06:03 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 19, 2019, 07:35 PM UTC
This incident has been resolved.
- postmortem Aug 01, 2019, 02:10 PM UTC
# What happened On 19th July 2019 at 15:30 UTC certificate used to serve traffic from [ucarecdn.com](http://ucarecdn.com) and [www.ucarecdn.com](http://www.ucarecdn.com) has expired. All HTTPS traffic to these domains effectively stopped. We were able to quickly \(within minutes\) redirect [www.ucarecdn.com](http://www.ucarecdn.com) traffic to alternative CDN provider with proper certificate installed. It took us approximately 105 minutes since start of the incident to fix the issue with [ucarecdn.com](http://ucarecdn.com).  # Why that happened 1. Our CDN partner's certificate management system failed to renew the certificate in question and needed an input from our team. 2. It did sent an email 5 days prior to expiration requesting manual input. 3. This email wasn't noticed, because it was sent to a team member that was off the grid on vacation. # What we should do to improve 1. Change notification system settings, so any issues with certificates with our CDN partners are sent to a team, not a person \[done\] 2. Use 3rd party service to monitor certificate expiration and other settings \[done\] 3. Use CDN partner's APIs to actively and automatically monitor certificates \[in progress\] 4. Create an service that could be used to quickly redirect [ucarecdn.com](http://ucarecdn.com) traffic to backup CDNs in case of any issues with the main one \(not only due to certificates\)