DealerSocket experienced a major incident on September 16, 2020 affecting CRM, lasting 1d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 16, 2020, 02:13 PM UTC
We are investigating reports of delayed lead processing. Teams will provide an update shortly.
- investigating Sep 16, 2020, 02:39 PM UTC
Teams are still working to identify cause of the delayed leads. At this point, Direct Post Leads are processing normally however Pop3/Email leads are delayed.
- investigating Sep 16, 2020, 03:20 PM UTC
Our Teams are still working to resolve this issue.
- investigating Sep 16, 2020, 04:07 PM UTC
Teams are still investigating to find the full cause of this problem. We will post an update at 11:35 AM Central time or sooner.
- investigating Sep 16, 2020, 04:31 PM UTC
Our Engineering teams are working through all avenues to resolve this issue. We are still working on the best pathway for recovery. The next update will come at 12:05 PM Central time.
- investigating Sep 16, 2020, 05:05 PM UTC
Our Engineering teams are working diligently to resolve this issue, however we do not have an update at this time. Next status update will be in approximately 30 minutes.
- identified Sep 16, 2020, 05:30 PM UTC
Teams have identified what the problem is and are working on steps to resolve the issue. Next status update will be in approximately 30 minutes.
- identified Sep 16, 2020, 06:07 PM UTC
We are continuing to work on a fix for this issue.
- identified Sep 16, 2020, 06:32 PM UTC
Teams have confirmed the issue and are working on the issue. Teams are working on validating their fix before it is pushed out. Next status update will be in approximately 30 minutes.
- identified Sep 16, 2020, 07:05 PM UTC
Teams are still validating their fix. We are hopeful to have an ETA at our next update.
- identified Sep 16, 2020, 07:39 PM UTC
Our teams have validated their fix and making the final preparations to push the changes into production. We are working on ensure that all leads are processed after the fix is in place. Final testing is underway and we will update this page once we have a more firm deployment time frame. Our next update will be at 3:35 PM Central time or sooner if conditions change.
- identified Sep 16, 2020, 08:48 PM UTC
Our Teams are running the final checks against our stage/test platforms to ensure it works as expected. Teams are making progress. We do apologize for the delay and appreciate your patience. Our next update will be around 4:40 PM Central.
- identified Sep 16, 2020, 09:31 PM UTC
Teams are tweaking the process based on the testing results. We do not have a firm ETA to report at this point in time for deployment into production. We will update at 5:30 PM Central time.
- identified Sep 16, 2020, 10:46 PM UTC
Thank you for your patience, Out of an abundance of caution, we've determined the best path forward is for us to revert part of our software release last night that is at the heart of this issue. After this change is rolled back, our team will run a script to reprocess any leads that were not correctly inserted into the CRM today. Due to the nature of this code roll back, this change must be completed during our maintenance window over night. With the change of plans and our next action happening overnight, we will update this page as information changes and not on a set schedule.
- identified Sep 17, 2020, 11:51 AM UTC
Our teams have released the fix to allow new email leads to process correctly. We are currently working on reprocessing leads what we’re not processed correctly yesterday. We will provide an update as teams make progress.
- identified Sep 17, 2020, 02:59 PM UTC
Recapping our current status: We have released a code fix which allows email leads to process normally. Teams are still working on the pathway to process the leads which came in yesterday and were not processed correctly. As the team makes progress on the leads which were not processed, we will update with an ETA.
- identified Sep 17, 2020, 05:56 PM UTC
We are in the process of running the last script needed to populate the leads that were delivered yesterday into the CRM. We will update once processing has finished.
- monitoring Sep 17, 2020, 06:38 PM UTC
We have finished processing all the leads which were not processed into the CRM. Teams are monitoring the environment for stability.
- resolved Sep 17, 2020, 06:52 PM UTC
This incident has been resolved. If you are still having and problems with leads, please contact the support department for troubleshooting.