Alkira incident
Provision API is returning errors intermittently
Alkira experienced a minor incident on October 9, 2023 affecting Network Provisioning Service, lasting 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 09, 2023, 04:12 PM UTC
We are currently investigating an issue with provision API. Issue started occurring at 15:50 UTC. We are actively investigating the issue.
- investigating Oct 09, 2023, 04:16 PM UTC
We also noticed that saving changes on the customer portal is returning errors.
- identified Oct 09, 2023, 04:32 PM UTC
The service seems to be recovering now. Provision API and connector save APIs errors should have been significantly reduced.
- monitoring Oct 09, 2023, 05:01 PM UTC
We believe the errors should have been significantly reduced. We continue to monitor the service and will post an RCA here soon.
- resolved Oct 09, 2023, 06:13 PM UTC
We have now resolved the issue. The issue was with provisioning requests taking much longer than anticipated. It appears that one of the backend databases was in a maintenance state, limiting the number of requests that can be made, resulting in intermittent errors while saving the connectors or provisioning. We will continue to monitor the databases and provision service for any errors. We apologize for any inconvenience this might have caused. Please reach out to Alkira support if you have any questions.