Mercatus incident
Financial Modeling instability for some operations in US
Mercatus experienced a minor incident on June 17, 2016, lasting 22h 37m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Jun 17, 2016, 02:19 AM UTC
Currently there are some customers reporting issues with the ability to run Get Investment Memo functionality on US production environments. The team is currently working to stabilize and restore this capability. Currently other functions are operational, however users may experience degraded performance and responsiveness while we work to resolve the issues.
- identified Jun 17, 2016, 04:18 AM UTC
We have identified the issue impacting some customers ability to run investment memo functions on the Mercatus Financial Modeling service. We have found a scenario in which the service will attempt to generate a report for information that is hidden and not accessible, causing the service to time out and reset. The team is working to provide a resolution for this scenario as soon as possible.
- investigating Jun 17, 2016, 03:53 PM UTC
Teams are continuing to investigate issues related to financial model calculations and investment memo generation. These instances are still limited and affecting only a limited group of customers. We have identified remediation plans and are working to ensure that our approach to resolution encompasses all of the existing outage. We are still investigating issues related to Get Investment Memo and have not yet determined a root cause, but are diligently investigating.
- monitoring Jun 17, 2016, 06:12 PM UTC
The engineering teams have discovered the cause and provided a fix for the issues impacting users ability to calculate projects. The issue was related to sequencing of calculation and macros run upon request execution. All services are operational, and we will continue to monitor the situation to ensure that service levels are maintained.
- resolved Jun 18, 2016, 12:57 AM UTC
We have been monitoring all FMS operations and the service remains stable. Issues relating to failure of get investment memo functionality have been resolved. The root cause was related to invalid permissioning, which has now been corrected.