MadKudu had 16 outages in the last 2 years totaling 187h 50m of downtime — averaging 0.7 incidents per month.
There were 16 MadKudu outages since July 29, 2024 totaling 187h 50m of downtime. Each is summarised below — incident details, duration, and resolution information.
Apr 20, 15:56 PDT Identified - We’ve identified an issue affecting Sales Copilot and are actively working on a fix. We expect full recovery within approximately 90 minutes.
Feb 27, 04:01 PST Resolved - This incident has been resolved. Feb 26, 03:51 PST Monitoring - A fix has been implemented and we are monitoring the results. Feb 24, 05:48 PST Identified - We’re currently investigating an issue affecting our ability to load new data from a subset of customers. Updates to follow.
The service used to provide transformation and updating of extracted data in the platform ("Data Workers") was degraded causing instability on the service and delays in refreshing data loaded from external integrations. This has been fixed and services are resuming although some delays in seeing updated data in copilot and scores in linked CRM tools may be experienced still.
Oct 21, 00:52 PDT Resolved - This incident has been resolved. Oct 21, 00:51 PDT Monitoring - Service functionality has been restored. We are continuing to monitor it to ensure full stability. Oct 20, 18:52 PDT Update - We're still monitoring our dependent services. Oct 20, 18:39 PDT Identified - Sales Copilot is currently experiencing renewed service disruption as some of our dependent services are facing issues again Oct 20, 17:13 PDT Monitoring - Service functionality has been restored. We are continuing to monitor it to ensure full stability. Oct 20, 12:58 PDT Identified - Sales Copilot is currently experiencing service degradation due to widespread outages affecting multiple online services, including our infrastructure providers. Our team is actively monitoring the situation and implementing mitigations as providers restore functionality. We will provide updates as more information becomes available.
Oct 6, 10:30 PDT Resolved - This incident has been resolved. Oct 6, 10:08 PDT Monitoring - A fix has been released, and we are monitoring the results. Oct 6, 09:54 PDT Identified - The issue has been identified and a fix is currently being released. Oct 6, 09:53 PDT Investigating - An issue has appeared on app.madkudu.com preventing people to access any pages of it.
Jun 16, 07:41 PDT Resolved - This incident has been resolved. MadKudu is pulling data normally again. Expect any remaining scoring delay backlog to clear in the next 12 hours. Jun 16, 04:16 PDT Monitoring - A fix has been implemented and we are monitoring the results. Jun 16, 02:51 PDT Identified - The issue has been identified and a fix is being implemented. Jun 16, 02:23 PDT Investigating - Our infrastructure has been experiencing delays in pulling new data from our customers' systems over the weekend. Our team has been notified and is actively looking into the issue.
May 30, 00:46 PDT Resolved - This incident has been resolved. May 27, 08:52 PDT Monitoring - The LaunchDarkly outage triggered an overload on our production database by one of our systems, which in turn prevented the Copilot service from handling some requests. After restarting the database and with LaunchDarkly gradually recovering, the load has decreased and Copilot is now functioning normally. We are closely monitoring the situation and are developing a plan to improve our resilience to similar incidents, which will be implemented in the coming weeks. May 27, 07:21 PDT Investigating - One of our provider, LaunchDarkly, is experiencing a major outage: https://status.launchdarkly.com/ We use Launchdarkly for our feature flag infrastructure (i.e. control how features are released). We are looking for a way to fix the situation without having to wait for Launchdarkly to fix their service.
During April 19th 2025 07:00 AM - April 22nd 2025 our Push ECS agent experienced intermittent disconnections, causing push delivery to be unstable. This was due to the agent becoming unresponsive, likely from machine lock or CPU overload. Actions taken: • Identified and rebooted the affected instance • Doubled CPU and memory reservations for the push tasks • Push service is now stable and under close monitoring Next steps to prevent recurrence: • Continue monitoring agent status and container performance
The explore view is experiencing outage and the message "It seems that there are some problems loading your data. Please update the rules or contact support if the issue persists." is displayed on some request. We are currently working on solving it and the system should be back very soon.
This incident has been resolved.
The service in charge of executing explore queries is experiencing an outage due to an overload. We are restarting the service and scaling up the machines to bring it back online.
The service is back and we upscaled the machines to make sure you don't experience degraded performance. We will keep monitoring the situation for 24 hours.
This incident has been resolved.
Our AWS infrastructure is suffering partial outage that cause Intermittent 504 Errors on the public API and can prevent new attributes mapping from being published. We are actively investigating the issue but due to its complexity, we don't expect it to be completely fixed before 24h
We identified the issue and are actively working on fixing it. We expect the performances to go back to normal before 6:00 UTC
The issue has been identified and the fix has been released. We are actively monitoring the platform.
This incident has been resolved.
The AWS region where the public API is hosted is having issues. We are redeploying the instances.
We are continuing to work on a fix for this issue.
The fix has been deployed and the error rate dropped significantly. We are monitoring the systems to ensure stability.
This incident has been resolved.
A major database migration created disturbance in our explore views. As a result, these views are empty. The problem has been identified and we are working to fix the problem.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
This incident has been resolved.
Issue AWS on Saturday 27 degraded our ability to push scores into destination systems (e.g. SFDC). We identified the issues and push resumed. We expect every score to be up-to-date in a few hours.
This incident has been resolved.