Newforma experienced a critical incident on September 9, 2022 affecting AWS us-east-1 and Projects (N. America) and 1 more component, lasting 2h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 09, 2022, 04:25 PM UTC
We are currently experiencing issues with loading projects in our Newforma Cloud applications and any of our Project Center Connectors due to an issue with an AWS Service we rely on, the Parameter Store. We will provide updates periodically, but also see the AWS status page if interested: https://health.aws.amazon.com/health/status Service: AWS Systems Manager Severity: Degradation Increased API Error Rates and Latency 8:36 AM PDT We are investigating increased API error rates and latency in the US-EAST-1 Region. This will impact customers' ability to make calls to Parameter Store.
- identified Sep 09, 2022, 04:26 PM UTC
Service: AWS Systems Manager Severity: Degradation Increased API Error Rates and Latency 9:29 AM PDT We can confirm increased API error rates and latencies for the Systems Manager Parameter Store in the US-EAST-1 Region. Customers using the Parameter Store for the storage of data may be affected by the event as they are unable to retrieve the data. We have identified the subsystem where the errors are occurring and are working to resolve the issue. At this stage, we expect recovery to take more than an hour but will keep you updated on our progress.
- identified Sep 09, 2022, 05:50 PM UTC
AWS continues to work on the issue. Our functionality has started to recover, but not fully available. We will continue to provide updates as we get more from AWS, and validations from our internal testing. Thank you for your patience. Increased API Error Rates and Latency 10:33 AM PDT We continue to work on the issue causing increased API error rates and latencies for the Systems Manager Parameter Store in the US-EAST-1 Region. Customers using the Parameter Store for the storage of configuration may be affected by the event as they are unable to retrieve the data. We have confirmed that customers using Lambda Environment Variables are not affected by this event, but rather customers that are attempting to access the Parameter Store directly from a Lambda function.
- resolved Sep 09, 2022, 06:36 PM UTC
AWS has confirmed the issue is resolved. All services should be functioning normally again. Please inform us if you continue to experience anything more. Again, thank you for your patience. [RESOLVED] Increased API Error Rates and Latency 11:21 AM PDT Starting at 7:34 AM PDT, we experienced increased error rates and latencies for the Systems Manager Parameter Store in the US-EAST-1 Region. Customers using the Parameter Store for the storage of configuration were affected by the event as they were unable to retrieve the data. Customers using Parameter Store from Lambda functions also experienced increased failure rates. The issue was resolved at 10:45 AM PDT, when error rates and latencies returned to normal levels. The issue has been resolved and the service is operating normally.