IntelePeer experienced a major incident on September 12, 2023, lasting 8m. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Sep 15, 2023, 01:40 PM UTC
Audio File Path Impairment experienced by some customers.
- postmortem Sep 15, 2023, 01:43 PM UTC
**Impact:** Customer experienced intermittent calls with no audio or dead air on affected calls **Root Cause:** An IntelePeer engineer had staged a new audio configuration file for future testing and validation but inadvertently and accidentally deployed the configuration. The file still had to be tested and validated with a destination path still to be defined. The new configuration file was intended to streamline the audio path to continue to improve performance. Because the file was deployed inadvertently, the audio file paths were directed to an errant location which resulted in no audio available creating the dead air effect. **Resolution:** Upon immediate alerting of low memory on the storage area networks and customers beginning to report calling issues, Intelepeer engineers immediately worked to identify and re-synch all audio and data files. In certain cases, to facilitate more immediate resolution, audio files were manually synched which restored the service earlier. **Corrective Actions Going Forth:** 1. IntelePeer has implemented a process of checks and balances whereby the individual staging a code change cannot be the same person deploying the code change. The process requires full validation and approval by the QA team before code can be deployed. 1. IntelePeer is reviewing options to include a systematic addition whereby in the event of errant or no audio file available, the system defaults to a customer approved message/file.