12Twenty incident
Small percentage of requests resulting in 502 Bad Gateway error
12Twenty experienced a minor incident on January 7, 2022 affecting Web Application, lasting 17d 17h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 07, 2022, 01:45 AM UTC
We are currently seeing up to 1% of requests for brief periods of time during the day receiving a 502 Bad Gateway error. We are currently working with Cloudflare and Azure to determine the root cause of the issue. Refreshing the page, retrying the request, or using a different browser will often result in a successful request. If you are experiencing this behavior or if you are consistently unable to access the site, please email support with how often you are experiencing this issue and how quickly the error message appears after executing your request. Thank you and we apologize for any inconvenience this has caused.
- investigating Jan 07, 2022, 06:02 PM UTC
No errors in the past 12 hours. We are continuing to monitor the situation and are continuing to work with Azure to fully resolve.
- investigating Jan 12, 2022, 11:12 PM UTC
We are still seeing some users experience this issue and are still working with Azure to investigate. Please let our support team know if a refresh or retrying the request does not result in a successful response.
- investigating Jan 13, 2022, 05:05 PM UTC
We made some configuration changes on the evening of 1/12/22 that we believe may resolve this issue. We are continuing to monitor.
- investigating Jan 24, 2022, 06:50 PM UTC
This incident has been resolved.
- resolved Jan 24, 2022, 06:52 PM UTC
Summary of impact: between 1/5/22 and 1/14/22, a small percentage of users on a handful of occasions (about 7 during this period of time) were faced with 502 errors when trying to access the site. These errors were ephemeral, but experienced by .5%-1.5% of users at the moment in time when one of the web application's instances was being automatically replaced due to hardware issues. Root Cause: working with the Azure support team, we determined that there was an issue with one of the instances serving the web application. Moving the web application to new instances has resolved the issue. Next steps: the replacement of instances should be completely transparent to the user and we are continuing to work with the Azure team to identify why this was not the case in our situation.