HTTPS outage for sites using the ExaVault host key without a Custom Domain for users in non-US regions

Update timeline

1. investigating May 01, 2024, 06:42 PM UTC

   ExaVault is a service that was acquired by Files.com. A small percentage of former ExaVault customers are still using the ExaVault host key after their migration to Files.com. If your site was not migrated from ExaVault, or if you no longer use the ExaVault host key, this incident does not affect you. This incident only affects users connecting to Files.com from non-US regions. We are currently investigating an incident causing an HTTPS outage for sites that use the ExaVault host key without a Custom Domain for users in non-US regions. This incident does not affect other services like unencrypted HTTP, FTP or SFTP. At this time, we believe that all network services are up in our US region.

2. resolved May 01, 2024, 06:49 PM UTC

   ExaVault is a service that was acquired by Files.com. A small percentage of former ExaVault customers are still using the ExaVault host key after their migration to Files.com. If your site was not migrated from ExaVault, or if you no longer use the ExaVault host key, this incident did not affect you. As of 11:39am Pacific Time, we have resolved an incident causing an HTTPS outage for sites that use the ExaVault host key without a Custom Domain for users in non-US regions. This incident did not affect sites with Custom Domains, sites that do not use the ExaVault host key, or users connecting from our US region.

3. postmortem May 09, 2024, 06:18 PM UTC

   On May 1st, 2024, at 11:17 AM PST, [Files.com](http://Files.com) correlated multiple customer tickets indicating _‘HTTPS outage for sites using the ExaVault host key without a Custom Domain for users in non-US regions_, which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. The _‘HTTPS outage for sites using the ExaVault host key without a Custom Domain for users in non-US regions’_ issue was resolved on May 1st, 2024, at 11:39 AM PST, returning the platform to full functionality. [Files.com](http://Files.com) released an initial posting to the [Status Page](https://status.files.com/) on May 1st, 2024, at 11:42 AM PST stating: _‘ExaVault is a service that was acquired by_ [_Files.com_](http://Files.com)_. A small percentage of former ExaVault customers are still using the ExaVault host key after their migration to_ [_Files.com_](http://Files.com)_. If your site was not migrated from ExaVault, or if you no longer use the ExaVault host key, this incident does not affect you._ _This incident only affects users connecting to_ [_Files.com_](http://Files.com) _from non-US regions._ _We are currently investigating an incident causing an HTTPS outage for sites that use the ExaVault host key without a Custom Domain for users in non-US regions. This incident does not affect other services like unencrypted HTTP, FTP or SFTP. At this time, we believe that all network services are up in our US region.’_ [Files.com](http://Files.com) released a resolution posting to the [Status Page](https://status.files.com/) on May 1st, 2024, at 11:49 AM PST stating: ‌ _‘ExaVault is a service that was acquired by_ [_Files.com_](http://Files.com)_. A small percentage of former ExaVault customers are still using the ExaVault host key after their migration to_ [_Files.com_](http://Files.com)_. If your site was not migrated from ExaVault, or if you no longer use the ExaVault host key, this incident did not affect you._ _As of 11:39am Pacific Time, we have resolved an incident causing an HTTPS outage for sites that use the ExaVault host key without a Custom Domain for users in non-US regions._ _This incident did not affect sites with Custom Domains, sites that do not use the ExaVault host key, or users connecting from our US region.’_ [Files.com](http://Files.com) acquired ExaVault, another MFT service, in 2022. Although all former ExaVault customers have been migrated to the mainline [Files.com](http://Files.com) platform, we do still support and maintain connectivity options which use legacy former ExaVault SFTP Host Key. This allows former ExaVault customers to continue their connections which were established using that host key. We recently deployed a change to expand our support for the ExaVault host key from beyond just our USA region service to all 7 of our global [Files.com](http://Files.com) service regions. This was done at customer request and to improve performance for non-US based customers. As part of this deployment, we replaced several customer-facing systems, and we updated our automatic DNS management system to use regional “latency-based” routing for the ExaVault Host Key domain. There was [another incident](https://status.files.com/incidents/dknqmws6cvr4) which occurred as part of that process. In addition to that incident, we also discovered that in some regions, customers were not made aware of the new IP addresses that we intended to use for this connectivity. We generally promise to alert customers to new IP addresses in advance of their use so they can update their firewalls. While we did publish these IP addresses to our documentation and APIs, we didn’t send an email notification of the new IPs. This was caused by the fact that we maintain a separate list of IPs for ExaVault, and that list was not fully integrated into our operational procedures. We have since integrated that list and updated our documentation to make very clear that there is a separate list for ExaVault IPs. We have reverted the global support for the ExaVault host key until we can provide adequate notice of the IP addresses which will be used. We will be making such announcement soon. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.