SchemeServe incident
Intermittent errors across SchemeServe services
SchemeServe experienced a minor incident on September 13, 2023 affecting 🎩 SchemeServe, lasting 2h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 13, 2023, 06:50 AM UTC
We are seeing intermittent connectivity issues across SchemeServe services this morning. We are currently investigating and will update when more information is available
- monitoring Sep 13, 2023, 07:28 AM UTC
We have identified the issue and have applied mitigating action to resolve it. We will continue to monitor to ensure there are no reoccurrences.
- resolved Sep 13, 2023, 09:13 AM UTC
This issue is now resolved, a post mortem will follow shortly.
- postmortem Sep 13, 2023, 09:13 AM UTC
During a routine scale-up of SchemeServe services at 07:30 this morning the designated subnet used had an issue with assigning available IP addresses, and as such some of the scaled services could not be contacted correctly. We applied mitigating actions, and by 08:10 all services could be contacted correctly again. We have redistributed services across other subnets to ensure that this does not happen again.