Trend Micro incident
Internal Service Error on Multiple Vision One Apps
Trend Micro experienced a major incident on February 8, 2023 affecting TrendAI Vision One™, lasting 1d 12h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 08, 2023, 02:45 AM UTC
Trend Micro service engineers are currently investigating a potential issue that may be affecting our customers. Some services may be severely degraded or completely unavailable. We are working to identify the specific impact and resolve the issue as quickly as possible.We apologize for the inconvenience and greatly appreciate your patience as we work to a resolution.
- investigating Feb 08, 2023, 05:30 AM UTC
Trend Micro service engineers believe that the cause of the incident has been identified. A solution to mitigate the impact is being worked on. More information will be posted as available, and we greatly appreciate your patience as we work to a resolution. l Assessment App: Internal Service Error when accessing Assessment App. (Caused by Azure Datacenter Cooling Event - Southeast Asia, still ongoing). Affected Site: SG l Observed Attack Techniques (OAT): Internal errors when accessing OAT. Workaround applied and is currently working properly. Affected Site: SG l Search App: Data ingestion issue. Affected sites: SG, IN and AU (Caused by Azure Datacenter Cooling Event - Southeast Asia, still ongoing).
- investigating Feb 08, 2023, 06:30 AM UTC
Trend Micro service engineers are still addressing an ongoing issue. We are continuing to work towards a resolution and restoring services as soon as possible. More information will posted as available, and we greatly appreciate your patience as we work to a resolution. l Assessment App: Internal Service Error when accessing Assessment App. (Caused by Azure Datacenter Cooling Event - Southeast Asia, still ongoing). Affected Site: SG l Observed Attack Techniques (OAT): Internal errors when accessing OAT. Workaround applied and is currently working properly. Affected Site: SG l Search App: Data ingestion issue. Affected site: SG. (Caused by Azure Datacenter Cooling Event - Southeast Asia, still ongoing). IN and AU sites have recovered
- investigating Feb 08, 2023, 09:30 AM UTC
Trend Micro service engineers are still addressing an ongoing issue. We are continuing to work towards a resolution and restoring services as soon as possible. More information will posted as available, and we greatly appreciate your patience as we work to a resolution. l Assessment App: Internal Service Error when accessing Assessment App. (Caused by Azure Datacenter Cooling Event - Southeast Asia, still ongoing). The Security Assessment page has been recovered. Work is ongoing on the remaining affected functions. l Observed Attack Techniques (OAT): Internal errors when accessing OAT. Status: Recovered l Search App: Data ingestion issue. IN and AU sites have recovered. Affected site: SG
- investigating Feb 08, 2023, 01:10 PM UTC
Trend Micro service engineers are still addressing an ongoing issue. We are continuing to work towards a resolution and restoring services as soon as possible. More information will posted as available, and we greatly appreciate your patience as we work to a resolution. l Assessment App: Internal Service Error when accessing Assessment App. (Caused by Azure Datacenter Cooling Event - Southeast Asia). The Security Assessment page has been recovered. Work is ongoing on the remaining affected functions. l Observed Attack Techniques (OAT): Internal errors when accessing OAT. Status: Recovered l Search App: Data ingestion issue. IN and AU sites have been recovered. Affected site: SG - currently waiting for Microsoft update
- investigating Feb 08, 2023, 03:10 PM UTC
Trend Micro service engineers are still addressing an ongoing issue. We are continuing to work towards a resolution and restoring services as soon as possible. More information will posted as available, and we greatly appreciate your patience as we work to a resolution. l Assessment App: Internal Service Error when accessing Assessment App. (Caused by Microsoft Azure Datacenter Cooling Event - Southeast Asia). Status: Recovered l Observed Attack Techniques (OAT): Internal errors when accessing OAT. Status: Recovered l Search App: Data ingestion issue. Status: IN and AU sites have been recovered. Affected site: SG - currently awaiting Microsoft Azure update
- investigating Feb 09, 2023, 05:45 AM UTC
According to Microsoft Azure status, the Southeast Asia Datacenter Cooling Event has been fully restored. Trend Micro is now processing stacked logs to restore full functionality. Log processing may take up to 7 hours (ETA: February 9, 12:45 GMT)
- resolved Feb 09, 2023, 03:15 PM UTC
Trend Micro service engineers have confirmed that the implemented solution has resolved the issue. All affected services are considered back to normal and the issue has been resolved.