Embed.ly incident

High Latency

Notice Resolved View vendor source →

Embed.ly experienced a notice incident on August 13, 2015, lasting 1h 46m. The incident has been resolved; the full update timeline is below.

Started
Aug 13, 2015, 11:41 PM UTC
Resolved
Aug 14, 2015, 01:27 AM UTC
Duration
1h 46m
Detected by Pingoru
Aug 13, 2015, 11:41 PM UTC

Update timeline

  1. investigating Aug 13, 2015, 11:41 PM UTC

    We are experiencing some increased latency, we are looking into this now.

  2. investigating Aug 14, 2015, 12:18 AM UTC

    Still on it, it's the engineer's birthday too. Poor @anrope.

  3. investigating Aug 14, 2015, 01:13 AM UTC

    Root cause found, latency is moving in the correct direction.

  4. resolved Aug 14, 2015, 01:27 AM UTC

    Everything has returned to normal. 2 bad things happened at once: 1. One of services that checks URLs against the safe browse index died. This is generally not a big deal as it's meant to quietly rebuild itself while traffic keeps humming along. 2. We saw a drastic change in traffic patterns. This made the first problem manifest into a big deal. We have corrected the first problem and almost double capacity for the night.