Nobl9 had 15 outages in the last 2 years totaling 20h 55m of downtime — averaging 0.6 incidents per month.
There were 15 Nobl9 outages since July 11, 2024 totaling 20h 55m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are experiencing partial service degradation affecting SLO processing. Alerting, reports, and SLO functionality may be impacted for a subset of users. The degradation was mitigated and resolved within 20min.
We experienced a temporary infrastructure issue affecting one of our Kubernetes worker nodes. This resulted in degraded Replay performance and intermittent chart issues.
We are currently investigating an incident where silenced alerts may still fire.
We are continuing to look into this issue.
We have identified the issue. Most SLOs and alert policies are not impacted and we are working on deploying a fix.
A fix was deployed to production.
This incident has been resolved.
Nobl9 is experiencing delays in sending alerts and data on the charts. Our engineering team is actively investigating the problem. If you have any questions please reach out to [email protected]
Charts are back, processing and alerting are recovering.
Charts are back, processing and alerting are recovering.
Charts, processing, and alerting systems are fully operational. However, if any alerts were generated during the incident, they may be delayed. If you have any questions, please feel free to reach out to support at [email protected].
This incident has been resolved.
Between 11:00 UTC and 14:10 UTC, data in all charts, dashboards, and reports may have been delayed. The delay began to grow around 11:00 UTC and was resolved by 14:10 UTC. All affected components have now caught up, and data is current.
We have identified an issue causing degraded performance of replay and delays in processing and data intake. Some replays will take longer to finish, some may need to be retried.
The issue has been resolved, we are monitoring the situation.
This incident has been resolved.
Some replays may take longer to finish. Some replays will need to be re-run.
Issue has been resolved
The bug affects the creation and display of RRR (Reliability Rollup Report) reports that use the “rolling” window “Last Month” and are created/viewed on March 31st and possibly March 30th. We are investigating the issue and possible solutions.
The issue has been identified and it will be automatically resolved once the 1st of April begins (UTC time zone), which will happen in around 6 hours. The issue is only affecting dates which do not have corresponding days in the previous month, i.e. 29th, 30th and 31st of March. The 28th of March is not affected as it has a corresponding date in the previous month, February 28th. Work on a permanent fix to the affected code is in progress.
The issue is resolved since 1st of April in (UTC time zone). Work on a permanent fix to the affected code is in progress, but the underlying issue no longer affects creating or viewing reports.
We are currently investigating an issue impacting US1 processing and the web application.
We are continuing to investigate this issue.
We've identified the cause and are working on a fix
We are continuing to work on a fix for this issue.
The fix has been implemented and services are now available. Data may take some time to recover.
Live data and alerts are fully functional, but historical chart data from during the incident is not yet available
The system is now fully operational.
Users of app.nobl9.com may have encountered errors in the UI while loading SLOs caused by post-release API unavailability. Our engineering team is actively working to improve API reliability during releases to minimize user impact. Additionally, we are reviewing and enhancing our error-handling processes to prevent similar issues in the future.
Nobl9 is currently experiencing API related issues. Processing and Alerting is not affected.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue with being able to view data in the web UI of Nobl9. Data collection, processing, and alerting is not impacted.
Data processing and alerts are now delayed as well. We are continuing to investigate the issue.
We are continuing to investigate this issue.
The platform is back up; charts are back, processing and alerting are recovering.
Charts, processing, and alerting systems are fully operational. However, if any alerts were generated during the incident, they may be delayed. If you have any questions, please feel free to reach out to support at [email protected].
This incident has been resolved.
Nobl9 is experiencing problems with displaying data on charts. Our engineering team is actively investigating the issue, we will follow up with more details as the situation unfolds. If you have any questions, please reach out to [email protected].
We have identified the issue, and our team is working on a fix. If you have any questions, please contact us at [email protected].
A fix has been implemented and we are monitoring the results.
This incident has been resolved, and charts are now working correctly. If you have any questions, please contact us at [email protected].
We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Alerts have experienced delays of 15-30min.