Orbee Outage History

A fix has been implemented and we are monitoring the results

THIS IS A SCHEDULED EVENT Apr 21, 21:00 - 23:00 PDT Apr 16, 12:49 PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Feb 22, 01:00 PST Completed - The scheduled maintenance has been completed. Feb 21, 19:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan 13, 09:52 PST Scheduled - We will be undergoing scheduled maintenance during this time

Jan 17, 11:46 PST Resolved - This incident has been resolved. Jan 17, 11:30 PST Monitoring - A fix has been implemented and the Platform has shown a recovery in performance. We'll continue to monitor the fix to make sure things remain performant. Jan 17, 11:20 PST Identified - We have identified an issue with the Orbee Platform's Reporting Dashboards and Views, related to a read replica with degraded performance from our cloud provider. We're isolating the usage of the instance and working on getting database reads working with acceptable performance again. During this time, you may see reporting dashboards show poor load times, or fail to load.

Dec 16, 15:26 PST Resolved - This incident has been resolved. Dec 11, 17:59 PST Monitoring - We have implemented a fix and we are monitoring the results. Dec 11, 10:32 PST Investigating - Our data processing infrastructure is running behind which is causing delays in our audience building tool and by extension is impacting email marketing tools. Uploading third-party lists is still available as an option to add create email marketing audience.

Nov 25, 11:16 PST Resolved - We have resolved the issue affecting legacy VinSolutions data ingestion. We completed a manual synchronization and migration to ensure all data tables are now current. We verified that automated email workflows and audiences are querying up-to-date data. We also determined that other data sources were not impacted and remain fully operational. Nov 20, 15:21 PST Investigating - We’re currently investigating an issue affecting performance for customers using the legacy VinSolutions integration. This may impact data availability for select audiences and automated email workflows. Our VinSolutions v2 integration remains fully operational. Our engineering team is actively working to resolve the issue. We’ll provide updates as more information becomes available.

Oct 21, 19:22 PDT Completed - The scheduled maintenance has been completed. Oct 21, 19:00 PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct 20, 13:08 PDT Scheduled - We will be undergoing maintenance during this time.

Oct 20, 17:27 PDT Resolved - AWS has reported the outage has been resolved as of 4 PM PST. Oct 20, 12:16 PDT Monitoring - Orbee is monitoring the current AWS Outage and maintaining our service uptime. As of 12 PM PST, no services have had any issues, outages or downtime at Orbee. We will continue to monitor the AWS outage and address any issues we do see come up.

We are currently monitoring an incident originating from Google Ads which has impacted multiple advertising-related services. These disruptions include delays or inconsistencies in reporting, audience syncing, campaign delivery, and budget updates. Google has identified and resolved the root issue, as detailed on their status dashboard. The incident spanned approximately 11 hours and affected Google Ads and Search Ads 360. While Google services have been marked as recovered, Orbee systems that rely on these integrations may still show delayed data propagation or partial functionality in the short term. Our engineering team is validating system stability and ensuring that all downstream effects are fully resolved. We will provide an update once all affected components are confirmed to be operating normally.

We are aware of a widespread outage affecting data centers globally. Orbee core services and infrastructure are NOT directly impacted by this incident. We are actively monitoring the situation and will provide further updates here if there are any changes or if the outage affects our core services.

A hotfix has been implemented in the meantime and we do see images showing up again for many vehicles. We will continue to monitor the situation as the rest of the image processing queue finishes processing.

We are continuing to monitor the image queue as it gets back to normal. Vehicle images are largely back up and working but we will continue to monitor while new inventory vehicles come through, to make sure the queue maintains its throughput and stays up-to-date.

We are continuing to monitor the image queue as it catches up to real-time ingestion. Given the fact that images are hosted on third-party systems before being downloaded to Orbee's CDN, we need to pull these "politely" with proper timing between requests and have pushed the speed at which images can be processed to close to those third-party systems' limits. Therefore we expect the queue to be fully caught up in the next 24 hours. We'll provide an update tomorrow regarding the wrap-up of the image queue. Currently, images are already being processed same-day for new vehicles ingested into Orbee's Platform.

We have verified the queue has caught up yesterday, and continues to not have issues processing images. We will continue monitoring the next inventory processing cycle to confirm before closing this incident out. Images should be rendering for all vehicle content in marketing and advertising material.

We are aware that our Email Service Provider (ESP) may be experiencing intermittent issues. Although we are not seeing any direct problems on our end at this time, we are taking a proactive approach and closely monitoring the situation. We understand the importance of reliable email delivery and will provide immediate updates if we detect any impact on our services.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the performance.

We have identified a fix and applied it to the pipeline, increasing performance when dealing with inserting data. We will continue to monitor the performance today, but all data should already be up-to-date and caught up due to this degraded performance window.