RoseHosting experienced a minor incident on March 2, 2021 affecting Upstream, lasting 6d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 02, 2021, 09:42 PM UTC
One of our upstream network peers is down due to a fiber cut. All routing is going through alternate network peers and all services are up and running at this time. Some customers may notice some degraded network performance. The team is on site working on a fix.
- identified Mar 03, 2021, 10:08 AM UTC
Carrier crew is still drilling and pulling fiber for repairs. We do not have an ETA on this yet. The network is operating with degraded performance but all services are up at this time.
- identified Mar 03, 2021, 02:20 PM UTC
We have a provisional update from the splicing team that the issue is likely going to be fixed in about 2-3 hours from now, so at or sometime before 11:30 AM Central Time.
- monitoring Mar 03, 2021, 02:47 PM UTC
We are now 100% back up and operational. We still don't have confirmation from the fiber repair team that they are done working but our fiber is active and 100% operational. We are now closely monitoring the situation and will post another update as soon as possible. We are also reviewing our current setup and will be implementing additional redundancies to prevent this from happening in the future.
- resolved Mar 09, 2021, 09:25 PM UTC
The upstream confirmed that all work on the fiber cut has been completed. The network performance is still optimal so we are going to be resolving this issue now.