Orbee had 20 outages in the last 2 years totaling 1753h 17m of downtime — averaging 0.8 incidents per month.
There were 20 Orbee outages since June 19, 2024 totaling 1753h 17m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are experiencing a partial outage affecting Data Studio. We are actively working to resolve the issue and will provide regular updates until full functionality is restored.
A fix has been implemented and we are monitoring the results
This incident has been resolved.
THIS IS A SCHEDULED EVENT Apr 21, 21:00 - 23:00 PDT Apr 16, 12:49 PDT Scheduled - We will be undergoing scheduled maintenance during this time.
Feb 22, 01:00 PST Completed - The scheduled maintenance has been completed. Feb 21, 19:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan 13, 09:52 PST Scheduled - We will be undergoing scheduled maintenance during this time
Jan 17, 11:46 PST Resolved - This incident has been resolved. Jan 17, 11:30 PST Monitoring - A fix has been implemented and the Platform has shown a recovery in performance. We'll continue to monitor the fix to make sure things remain performant. Jan 17, 11:20 PST Identified - We have identified an issue with the Orbee Platform's Reporting Dashboards and Views, related to a read replica with degraded performance from our cloud provider. We're isolating the usage of the instance and working on getting database reads working with acceptable performance again. During this time, you may see reporting dashboards show poor load times, or fail to load.
Dec 16, 15:26 PST Resolved - This incident has been resolved. Dec 11, 17:59 PST Monitoring - We have implemented a fix and we are monitoring the results. Dec 11, 10:32 PST Investigating - Our data processing infrastructure is running behind which is causing delays in our audience building tool and by extension is impacting email marketing tools. Uploading third-party lists is still available as an option to add create email marketing audience.
Nov 25, 11:16 PST Resolved - We have resolved the issue affecting legacy VinSolutions data ingestion. We completed a manual synchronization and migration to ensure all data tables are now current. We verified that automated email workflows and audiences are querying up-to-date data. We also determined that other data sources were not impacted and remain fully operational. Nov 20, 15:21 PST Investigating - We’re currently investigating an issue affecting performance for customers using the legacy VinSolutions integration. This may impact data availability for select audiences and automated email workflows. Our VinSolutions v2 integration remains fully operational. Our engineering team is actively working to resolve the issue. We’ll provide updates as more information becomes available.
Oct 21, 19:22 PDT Completed - The scheduled maintenance has been completed. Oct 21, 19:00 PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct 20, 13:08 PDT Scheduled - We will be undergoing maintenance during this time.
Oct 20, 17:27 PDT Resolved - AWS has reported the outage has been resolved as of 4 PM PST. Oct 20, 12:16 PDT Monitoring - Orbee is monitoring the current AWS Outage and maintaining our service uptime. As of 12 PM PST, no services have had any issues, outages or downtime at Orbee. We will continue to monitor the AWS outage and address any issues we do see come up.
Jul 24, 21:41 PDT Completed - The scheduled maintenance has been completed. Jul 24, 21:00 PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul 24, 12:22 PDT Scheduled - We will be undergoing maintenance during this time.
For about an 90 minutes from 4:35 PM UTC to 6:10PM UTC, users experienced slower administration API and UI performance (listing accounts and navigating the platform). This was caused by a reoccurring background job taking more resources than normal this morning. We have since rectified the issue and the platform and APIs are performing normally again.
We are aware of a widespread outage affecting data centers globally. Orbee core services and infrastructure are NOT directly impacted by this incident. We are actively monitoring the situation and will provide further updates here if there are any changes or if the outage affects our core services.
The data centers impacted by the global outage have recovered. Orbee core services were not affected, and we continue to monitor for any residual issues.
We are currently investigating an issue experienced by some customers intermittently, where automated emails going out can sometimes be missing vehicle images when referencing their ingested inventory. We will provide an update once a cause for this issue has been identified.
We have identified that the inventory image download queue is currently lagging. Therefore, this issue only affects vehicles made available in the dealer's inventory feed within the past 72 hours that are already being marketed to customers in automated emails. We have already increased the amount of processing we support for this queue, and will continue to monitor the situation and identify why the queue fell behind. We'll provide another update on the progress of getting this image download queue back under control.
A hotfix has been implemented in the meantime and we do see images showing up again for many vehicles. We will continue to monitor the situation as the rest of the image processing queue finishes processing.
We are continuing to monitor the image queue as it gets back to normal. Vehicle images are largely back up and working but we will continue to monitor while new inventory vehicles come through, to make sure the queue maintains its throughput and stays up-to-date.
We are continuing to monitor the image queue as it catches up to real-time ingestion. Given the fact that images are hosted on third-party systems before being downloaded to Orbee's CDN, we need to pull these "politely" with proper timing between requests and have pushed the speed at which images can be processed to close to those third-party systems' limits. Therefore we expect the queue to be fully caught up in the next 24 hours. We'll provide an update tomorrow regarding the wrap-up of the image queue. Currently, images are already being processed same-day for new vehicles ingested into Orbee's Platform.
We have verified the queue has caught up yesterday, and continues to not have issues processing images. We will continue monitoring the next inventory processing cycle to confirm before closing this incident out. Images should be rendering for all vehicle content in marketing and advertising material.
This incident has been resolved. Multiple inventory ingestion passes have completed without the queue falling behind or raising errors anymore.
We are aware that our Email Service Provider (ESP) may be experiencing intermittent issues. Although we are not seeing any direct problems on our end at this time, we are taking a proactive approach and closely monitoring the situation. We understand the importance of reliable email delivery and will provide immediate updates if we detect any impact on our services.
We are continuing to monitor for any further issues.
Our team has confirmed that the recent service issues have been resolved. All services are now operating as expected.
The temporary outage affecting our email marketing services has been resolved. The server issue that caused the disruption has been addressed, and all email marketing pages should now be functioning normally. During the outage, you may have experienced issues accessing or using the email marketing pages. We apologize for any inconvenience this may have caused. Thank you for your patience.
We are investigating an issue impacting our automated email service. We are actively working to identify and resolve the issue. Our initial investigation indicates that this may result in a lower volume of emails going out until the issue fix goes in place. Once the issue is resolved, we expect automated emails to send as expected. The following services may be impacted: Automated Email Marketing We will continue to provide updates on this page as more information becomes available.
A fix has been implemented and we are monitoring the performance.
We are pleased to report that the issue impacting our automated email service has been resolved. The fix has been successfully implemented and automated emails are sending as expected.
We are currently investigating degraded performance of the Reporting Data Pipeline. This will lead to dashboards for website analytics to lag before showing accurate traffic counts. The pipeline is currently about 12-hours behind and is catching up. We will share more information as soon as we've identified the cause of the degregation.
We have identified a fix and applied it to the pipeline, increasing performance when dealing with inserting data. We will continue to monitor the performance today, but all data should already be up-to-date and caught up due to this degraded performance window.
This incident has been resolved, and the fixes applied have continued to keep our reporting/analytics pipeline processing times consistent.
We are aware of an issue affecting some users' ability to log in to the Orbee platform. We are investigating the cause and working to restore access as quickly as possible.
The login issue has been resolved. All users should now be able to access the platform. Thank you for your patience as we resolved this issue.
We are investigating an issue impacting our inventory service, which may affect advertising campaigns and automated email marketing that rely on our Inventory API. The following advertising platforms may be impacted: Meta - Disruptions to Dynamic Inventory/Carousel Ads. Google Ads - Disruptions to Vehicle Listing Ads and potentially inventory-reliant Responsive Search Ads. Automated Email Marketing - Emails triggered by inventory events may not be sent due to missing or outdated data. We are actively working to identify and resolve the issue. Our initial investigation indicates that while we are receiving inventory feeds from our third-party providers, our system is not processing and saving these updates. Once the issue is resolved, we expect to be able to reprocess and correct all affected inventory data. We will continue to provide updates on this page as more information becomes available. We apologize for any disruption this may cause and appreciate your patience.
The issue has been narrowed down to the processing of inventory vehicle sets and inventory price drop alerts. Vehicle Sets and Price Drop information will remain in their current state without updates until performance issues are addressed. We are expecting other inventory-related functionality to return to normal soon, and will provide another update once we reach some stability.
We are continuing to monitor the performance of the Inventory systems, but feeds are now being processed again and inventory sets are functional again. The UI and Advertising integrations are all functioning correctly. Price Drop fixes are still being performed and we will provide another update as soon as we have those fixes deployed.
A fix has been implemented for price drops and will be monitored overnight. Feeds are continuing to run and catch up to their latest changes. We'll provide another update in 12 hours or sooner if we have more details to share.
Overnight all inventory feeds have caught up and all price changes were recorded correctly, without any performance impact to systems integrating with inventory systems. Customers should see all systems related to inventory working properly now with no gaps. The updates made caused a large increase in the performance of inventory-related data services and helped scale the system for spikes moving forward. All inventory feeds that were falling behind are now up-to-date and are no longer storing invalid states for inventory vehicles.
Orbee is monitoring and responding to the CDK outages nationwide. We value our relationship with CDK and we are here to support our dealerships in any way we can while these services are restored. Orbee data and application middleware processes from CDK services including Modern Retail CRM, (formerly eLead) CDK DRIVE, Service Scheduler and other CDK services, will experience downtime as we have intentionally paused and disconnected our services to CDK to ensure the safety and integrity of our systems and our clients' customer data. The processes affected will include APIs, file transfers, third-party feed providers, and dealer-provided database feeds, which are expected to be affected by this. We are committed to working with our clients to reconcile datasets and ensure all data management services can resume once CDK services stabilize. Dealership and enterprise customers can reach out to their point of contact at Orbee for further assistance.
Orbee is continuing to monitor and respond to the CDK outages nationwide. Customers will continue to experience downtime as we continue to keep our services to CDK disconnected to ensure the safety and integrity of our systems and our clients’ customer data. The processes affected will include APIs, file transfers, third-party feed providers, and dealer-provided database feeds, which are expected to be affected by this. We are committed to working with our clients to reconcile datasets and ensure all data management services can resume once CDK services stabilize. Dealership and enterprise customers can reach out to their point of contact at Orbee for further assistance.
Over the last couple of weeks, Orbee rolled out CDK integrations again with our customers. As of this week, Orbee has completed the review of the final enabled integrations between our customers and CDK and has approved the final set of checks. We have also worked on filling in any gaps caused by the incident that the data provided by the integrations could provide. We are committed to working with our clients to reconcile datasets and ensure all data management services can resume since CDK services stabilized. Dealership and enterprise customers can reach out to their point of contact at Orbee for further assistance.