IT Portal incident

Cloud services affected by CloudFlare Outage

Resolved after 58 minutes, affecting US API and Portal KBs and 1 more, on May 27, 2022.

monitoring May 27, 2022, 04:12 AM UTC

We have worked with Cloudflare and made changes to our setup of DNS records and the issue now seems resolved. We speculate an issue on amazon's DNS servers. We switched our cluster DNS records from amazon CNAMEs to the IP of the cluster and the issue had resolved itself. BEFORE www.siportal.net is a CNAME pointing to us-active.siportal.net us-active.siportal.net is a CNAME pointing to wc04.siportal.net wc04.siportal.net was a CNAME pointing to ec2-34-232-208-181.compute-1.amazonaws.com AFTER www.siportal.net is a CNAME pointing to us-active.siportal.net us-active.siportal.net is a CNAME pointing to wc04.siportal.net wc04.siportal.net is now pointing to A 34.232.208.181

Started
May 27, 2022, 03:15 AM UTC
Resolved
May 27, 2022, 04:14 AM UTC
Duration
58m
Detected by Pingoru
May 27, 2022, 03:15 AM UTC

Affected components

US APIPortal KBsUK CloudCA CloudEU CloudAU CloudUS CloudUK APICA APIEU API

Update timeline

  1. identified May 27, 2022, 03:15 AM UTC

    Access to multiple cloud regions is being affected by an Issue at Cloudflare. The issue seems to be sporadic and access is limited at times. See the video below for a detailed explanation. https://www.loom.com/share/9c7e6ff845034fbcae8786414b74cd54

  2. monitoring May 27, 2022, 04:09 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring May 27, 2022, 04:12 AM UTC

    We have worked with Cloudflare and made changes to our setup of DNS records and the issue now seems resolved. We speculate an issue on amazon's DNS servers. We switched our cluster DNS records from amazon CNAMEs to the IP of the cluster and the issue had resolved itself. BEFORE www.siportal.net is a CNAME pointing to us-active.siportal.net us-active.siportal.net is a CNAME pointing to wc04.siportal.net wc04.siportal.net was a CNAME pointing to ec2-34-232-208-181.compute-1.amazonaws.com AFTER www.siportal.net is a CNAME pointing to us-active.siportal.net us-active.siportal.net is a CNAME pointing to wc04.siportal.net wc04.siportal.net is now pointing to A 34.232.208.181

  4. resolved May 27, 2022, 04:14 AM UTC

    This incident has been resolved.

Your product runs on more than IT Portal

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →