Melo Technology experienced a notice incident on March 19, 2025 affecting MeloTel Customer Portal and Click2Call, lasting 1h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 19, 2025, 07:52 PM UTC
We have identified an issue with Click2Call due to an emergency maintenance with Melo Customer Portal. We are working on the situation with urgency. We will provide updates as soon as they become available.
- investigating Mar 19, 2025, 08:09 PM UTC
We are expecting an update from our support engineers on the situation shortly.
- identified Mar 19, 2025, 08:25 PM UTC
We have identified the issue with our customer portal. We are working as quickly as possible to restore services.
- identified Mar 19, 2025, 08:31 PM UTC
The Customer Portal and MeloSIP Click2Call services have been successfully restored. While maintenance is still ongoing, services are currently operational. We will provide further updates as soon as possible.
- resolved Mar 19, 2025, 08:59 PM UTC
All services have been restored. We have scheduled another maintenance session for the evening to correct an identified bug in our support ticketing system. However, this should not impact customer services. We apologize for any inconvenience caused during this event.
- postmortem Mar 19, 2025, 09:04 PM UTC
# **Postmortem Report: Service Interruption** ## **Summary** Today at 15:52 EDT, a service disruption affected the **MeloTel Customer Portal** and **MeloSIP Click2Call** services due to an unintended misconfiguration during email service updates. This incident resulted in an automatic email retrieval issue that required emergency maintenance, temporarily impacting customer access. ## **Incident Timeline** * **15:52 EDT – Issue Identified:** * An issue was detected affecting **Click2Call**, stemming from emergency maintenance on the **MeloTel Customer Portal**. * The team began investigating the root cause with urgency. * **16:09 EDT – Investigation Ongoing:** * Support engineers were engaged, and further analysis was conducted. * **16:25 EDT – Issue Identified:** * The cause of the customer portal outage was determined, and recovery efforts began. * **16:31 EDT – Partial Restoration:** * **Customer Portal** and **MeloSIP Click2Call** services were restored. * Maintenance continued to ensure full resolution. * **16:59 EDT – Full Resolution:** * All affected services were reported restored. * A secondary maintenance session was scheduled to address a **support ticketing system bug**, though this is not expected to impact customer services. ## **Root Cause** * A misconfiguration occurred while attempting to reconnect email services to our Customer Portal. * The incorrect email connection triggered **automated email retrieval**, resulting in the unintended deletion of emails and an impact on our Customer Portal. * To prevent further data loss, the **server was temporarily shut down**, leading to a brief service disruption for the **Customer Portal and MeloSIP Click2Call**. * Once the situation was secured, services were restarted, and minor post-recovery adjustments were made. ## **Impact** * **Customer Portal and MeloSIP Click2Call** were unavailable for approximately 40 minutes. * Several pending **support tickets were unintentionally closed**. ## **Next Steps & Preventative Actions** * **Email Service Reconfiguration:** Ensure correct reconnection of email services to Customer Portal. * **Process Improvements:** Implement additional verification steps for email service changes. * **Automated Monitoring Enhancements:** Strengthen monitoring to detect unintended email retrieval events before they escalate. * **Support Ticket Review:** Assess and restore any support tickets that were mistakenly closed. We sincerely apologize for the inconvenience caused by this incident and appreciate your patience. Our team remains committed to improving our systems and preventing similar occurrences in the future. For further inquiries, please contact our support team. **Melo Technology Team**