Tuff Shed experienced a critical incident on July 9, 2025 affecting Business Objects (BOBJ), lasting 25d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 09, 2025, 01:56 PM UTC
We’re aware the BOBJ Daily Revenue Projection Report has not ran since 7/7. Our team is investigating this now.
- investigating Jul 09, 2025, 05:29 PM UTC
Our team is continuing to investigate this issue.
- monitoring Jul 09, 2025, 06:24 PM UTC
We've implemented a fix and will continue to monitor the results. Please contact IT Support if you have further issues.
- investigating Jul 10, 2025, 02:40 PM UTC
Our team is aware that BOBJ did not run overnight, we're investigating this now.
- monitoring Jul 10, 2025, 07:34 PM UTC
We've restarted the server and can see the missing data for the reports have populated. Our team is continuing to investigate the instability of BOBJ over the last week.
- monitoring Jul 11, 2025, 02:32 PM UTC
We're aware BOBJ Reports did not run overnight. This is related to the ongoing stability issues our team is investigating. We're working to restart the server to update the daily reports and continuing to investigate the root cause of instability.
- investigating Jul 18, 2025, 04:37 PM UTC
Our team is actively troubleshooting the reporting issues in BOBJ. Multiple reports aren't functional right now and may be missing data. We are working to resolve this as quickly as possible.
- monitoring Jul 20, 2025, 07:36 PM UTC
After restoring systems the web server is now stable. All tested reports have run successfully. Please open a ticket with any reported problems. The team will continue to monitor for issues.
- identified Jul 21, 2025, 12:53 PM UTC
Some of the overnight jobs to move data from JDE to BOBJ failed which has caused some data in BOBJ to be inaccurate. We are investigating and re-running the batch jobs.
- monitoring Jul 22, 2025, 02:55 PM UTC
The overnight jobs completed and the BOBJ data appears to be accurate at this time. We will continue to monitor the stability of the systems, please enter a support ticket if you encounter problems.
- monitoring Jul 23, 2025, 01:28 PM UTC
One of the overnight jobs did not complete, we are re-running it. Data in BOBJ may not be accurate.
- identified Jul 25, 2025, 09:30 PM UTC
BOBJ is currently down, we are working on restoring it.
- identified Jul 26, 2025, 01:18 AM UTC
BOBJ is currently functional, we will continue to monitor.
- identified Aug 01, 2025, 01:40 PM UTC
The BOBJ server crashed this morning and is unresponsive. We are investigating.
- monitoring Aug 01, 2025, 11:41 PM UTC
The team has implemented a fix for the BOBJ issues we were seeing this morning and the system has been stable. We will be monitoring over the weekend and into Monday for additional issues. Please enter a ticket if additional issues come up.
- resolved Aug 04, 2025, 01:20 PM UTC
All overnight jobs ran as expected over the weekend and the system has been stable since Friday evening. We are marking this as resolved, please enter a ticket for any new problems with BOBJ.