MoEngage Outage History

This incident has been resolved.

There was a degradation identified on DC03 Data Ingestion pipeline since 9th Apr, 10:30 AM IST. This was due to an infra/instance type change which was deployed. This change seemed to be unstable and hence has been reverted now. It resulted in degraded data processing throughput resulting in lag. Duration of the incident: 10:30am IST - 01:45pm IST (~3.15 hours) Impact: The delay in data processing has impact on availability of user updates and events in segmentation. it also had an impact on Event triggered campaigns and flows due to the delay. Resolution: The lag has been recovered as of 01:45pm IST. There is no data loss. Team is currently monitoring the systems and doing a deeper post-mortem to avoid such occurrence in future.

Mar 31, 11:41 UTC Resolved - This incident has been resolved. The lag was observed from 06:45 UTC (12:15pm IST) to 11:30 UTC (05:00pm IST). Mar 31, 11:11 UTC Monitoring - We are currently experiencing degraded performance in our Trigger-Based Flows system. This is causing delays of up to 45 minutes in user entry and movement within Trigger-Based Flows. Please note that Event-Triggered Campaigns remain unaffected. Our team has fixed the issue and is actively monitoring the system. We apologize for the inconvenience and appreciate your patience as we work to resolve this.

The issue is resolved as of now and ingestion pipeline latency seems stable. However, the technical team is root causing the issue to deploy a permanent fix. This data ingestion delay had an impact on Smart trigger campaigns and Flows, as it consumes the data from the ingestion pipeline and apply trigger/flows conditions. Delay of around 45 min lasted between 24-March 13:00 UTC to 15:40 UTC. There were delay observed later as well on 24-March 20:30 UTC to 21:30 UTC.

Mar 13, 05:16 UTC Resolved - This incident has been resolved. Mar 13, 03:11 UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar 13, 02:32 UTC Identified - The issue has been identified and a fix is being implemented. Mar 13, 02:13 UTC Update - We are continuing to investigate this issue. Mar 13, 02:12 UTC Investigating - Team is currently working on investigating following impacted services in DC03 - Smart Trigger & Flow Campaigns We apologize for inconvenience this may have caused and appreciate your patience. For any queries or immediate concerns, please do not hesitate to contact our support team.

Mar 12, 19:18 UTC Resolved - This incident has been resolved. Mar 12, 17:05 UTC Identified - We are currently experiencing a delay in the data ingestion pipeline for following services: 1. Event Triggered Campaigns and Flows 2. Inbound Campaigns (In-App, OSM, WebP) 3. Recent Activity on User Profile We have identified the issue and mitigation has been deployed and lag is being monitored closely by teams.

Mar 6, 11:06 UTC Resolved - The incident has been resolved. The issue lasted for 30 min from 10:35 to 11:05 UTC. Mar 6, 11:05 UTC Update - We are continuing to monitor for any further issues. Mar 6, 10:56 UTC Monitoring - All Campaigns Page is not loading since 10:35 UTC on DC-02. A fix has been implemented and issue should be resolved in the next 10 minutes.

Mar 5, 17:15 UTC Resolved - This incident has been resolved. Mar 5, 16:29 UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar 5, 15:31 UTC Update - We are continuing to work on a fix for this issue. Mar 5, 15:31 UTC Identified - The issue has been identified and a fix is being implemented. Mar 5, 15:28 UTC Investigating - We are currently experiencing delay with the Stats update in Dashboard 03. Please note that campaign sending is not impacted; however, the latest statistics may be delayed on the dashboard at this time. Our team is treating this as a top priority and is working to resolve the issue as quickly as possible. Thank you for your patience.

Feb 25, 12:18 UTC Resolved - This incident has been resolved. Feb 25, 12:16 UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb 25, 12:05 UTC Investigating - We are currently investigating the issue.

Feb 19, 20:30 UTC Completed - The scheduled maintenance has been completed. Feb 19, 18:22 UTC Update - We have paused our data processing systems and initiated cluster resizing. Feb 19, 17:30 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb 17, 08:52 UTC Scheduled - Hello, To ensure the long-term stability and efficiency of our data pipelines, we will be resizing our Kafka and Data Warehouse clusters during a scheduled maintenance window on Feb 19-20. Schedule: Window: Feb 19, 17:30 PM UTC – Feb 19, 20:30 PM UTC Duration: 3 hours Service Impact: To facilitate the rebalancing of our data brokers, users may experience transient latency in data ingestion. This means that while user and events are being successfully received, they may take longer to reflect in downstream reports, segmentation, campaign stats and analytics. Technical Note: Once the cluster rebalancing is complete, the system will automatically clear any accumulated backlogs. We do not anticipate any data loss or service downtime. There will not be any impact on campaign sending schedules.

Feb 11, 17:50 UTC Resolved - This incident has been resolved. Feb 11, 16:16 UTC Update - We are continuing to monitor for any further issues. Feb 11, 16:12 UTC Monitoring - We are experiencing a delay of around 90 minutes in processing data imports & exports. A fix has been implemented and currently the system is recovering.

Feb 11, 03:31 UTC Resolved - This incident has been resolved. Feb 10, 22:56 UTC Update - We are continuing to monitor for any further issues. Feb 10, 22:52 UTC Monitoring - AWS has confirmed recovery of all their affected services. We are monitoring our services to ensure the services are back to normal. Feb 10, 22:35 UTC Update - We are continuing to investigate this issue with AWS as this may be affecting other regions (DCs/ Dashboards) due to issues in their global services like CloudFront. Feb 10, 22:10 UTC Investigating - We are currently experiencing a disruption affecting multiple services due to an ongoing outage with our cloud service provider, AWS. Our team is actively monitoring their status updates and collaborating with AWS to restore services as quickly as possible. For more details on AWS’s status, please visit: https://health.aws.amazon.com/health/status