CallHub incident
The CallHub dashboard is throwing HTTP error 403 and experiencing a similar issue on the mobile app.
CallHub experienced a critical incident on February 15, 2024 affecting Main Dashboard, lasting 13h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Feb 15, 2024, 03:07 AM UTC
On Feb 14, 2024, CallHub users experienced disruption on the CallHub dashboard, encountering HTTP error 403. This issue also affected the performance of the mobile app during the period from 5:16 PM to 5:50 PM PST. Upon conducting an initial investigation, we identified the root cause related to auto-scaling processes. The issue has been promptly addressed, and as of now, all services have been restored to normal functionality. Our team actively monitors the services to ensure stability. We apologize for any inconvenience caused during this period.
- resolved Feb 15, 2024, 04:16 PM UTC
RCA - One of the database connection pooler system configuration settings was incorrectly set to lower limit when we upgraded the servers which resulted in failure of application to db server connections when the traffic was high. Impact - System was entirely down from 5:16pm PST to 5:35pm PST - 19min complete outage. System was available with degraded performance (slowness) from 5:35pm PST till about 5:50pm PST - 15min degraded performance.