Aptoma experienced a critical incident on May 29, 2023 affecting Smooth Storage API (images, graphics and files), lasting 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 29, 2023, 03:16 PM UTC
We are investigating an image with uploading and/or transforming images.
- identified May 29, 2023, 03:20 PM UTC
We have identified the issue and are deploying an update.
- monitoring May 29, 2023, 03:34 PM UTC
Image systems are working again in general. However some 900 images uploaded during the incident may be stuck. Re-uploading them again does not solve the issue. A workaround exists: You have to edit the image locally (resize, crop a bit, or any way modify the original image a bit) before re-uploading it. This will solve the issue for this particular image.
- resolved May 29, 2023, 03:47 PM UTC
This incident has been resolved, including the 900 stuck images. If you still have issues, please contact support. We are sorry for the problems caused.
- postmortem May 31, 2023, 09:52 AM UTC
The cause of this incident was a planned and proactive maintenance. The maintenance caused a performance issue on database-level on Aptoma Smooth Storage services. We have since rolled back this maintenance change, and we have found other methods to accomplish this form of maintenance. This means we will not experience this kind of issue by the same causes in the future.