Resolved after 1 day 11 hours, affecting Dashboard Developer APIs and Data Pipeline and 1 more, on October 27, 2020.
Awair’s mission is to provide reliable \(always-on\) data service. Over the past couple of days we failed to deliver on this promise. We want to take ownership of our mistake, and sincerely apologize for any inconvenience caused by disruptions to the Awair service. We further want to communicate as clearly and quickly as possible what happened, how to resolve any remaining issues with Awair Omni, and what we are doing to prevent this from happening in the future. _**What caused the disruption:**_ We had a web certificate expire that impacted our service on October 26th, 2020 at approximately 17:35 PT. We renewed the certificate on October 28th at approximately 02:35 PT, which we thought resolved the issue. However, two hours later, at 04:45 PT on October 28th, we noticed a further incident the outage had created for Awair Omni devices: during the period of time the certificate was expired, Transport Layer Security \(TLS\) communications between Awair Omni devices and our servers failed. When Omni devices could not synchronize their clock with our servers \(due to the expired certificate\), these repeated failures caused devices to crash and remain disconnected. _**How to resolve remaining issues with Omni:**_ NOTE: Your Awair Omni device does not require a factory reset. It may, however, require a manual power cycle. To power cycle your Awair Omni: * Hold down the right side button for ~3-5 seconds * Let go for a second * Push the button again until the Awair logo appears on the front of the device. The service disruption for some Omni devices began on October 27th at 07:30 PT. You may see gaps in data from then until October 28th at 04:45 PT and/or until you power cycle your device. _**How we will prevent this from happening in the future:**_ Awair Omni’s response to the server disconnection was an anomaly, one that we have addressed and debugged to make sure that it does not occur in the future. Additionally, we have internally set up cross-checks within the company regarding the ownership of the certification renewal to guarantee that a lapse in renewal time will not be a repeated incident. We are deeply apologetic for the inconvenience caused by this disruption in service. We are immensely appreciative of your patience and support as we grow as a global company. If you find that your team or your Omni devices continue to have any issues following a power cycle, please reach out to [[email protected]](mailto:[email protected]).
Affected components
Update timeline
- investigating Oct 27, 2020, 02:23 AM UTC
We are currently investigating this issue.
- identified Oct 27, 2020, 02:25 AM UTC
The issue has been identified and a fix is being implemented.
- identified Oct 27, 2020, 06:57 AM UTC
Awaiting paperwork from Digicert
- monitoring Oct 27, 2020, 03:03 PM UTC
A fix by replacing CA with letsencrypt and we are monitoring the results.
- monitoring Oct 27, 2020, 05:14 PM UTC
Dashboard, web, API, and most app features are working. We are working on resolving new (or factory reset) device onboarding. If you are experience issues, please avoid factory resetting your device until we can confirm that it is working.
- monitoring Oct 27, 2020, 05:42 PM UTC
Temporary fix applied to address onboarding issues. Let us know if you see any unintended behaviors ([email protected]) while we work on a permanent fix.
- identified Oct 28, 2020, 02:20 AM UTC
Devices are refreshing their TLS handshakes with our backend services. Data loss is expected. We are working with the certificate authority to renew the certificate as soon as possible.
- identified Oct 28, 2020, 06:17 AM UTC
A certificate has been issued and we are working hard to apply it as soon as possible.
- monitoring Oct 28, 2020, 06:53 AM UTC
New certificate applied and monitoring for device reconnection.
- resolved Oct 28, 2020, 11:45 AM UTC
This incident has been resolved.
- postmortem Oct 29, 2020, 02:45 AM UTC
Awair’s mission is to provide reliable \(always-on\) data service. Over the past couple of days we failed to deliver on this promise. We want to take ownership of our mistake, and sincerely apologize for any inconvenience caused by disruptions to the Awair service. We further want to communicate as clearly and quickly as possible what happened, how to resolve any remaining issues with Awair Omni, and what we are doing to prevent this from happening in the future. _**What caused the disruption:**_ We had a web certificate expire that impacted our service on October 26th, 2020 at approximately 17:35 PT. We renewed the certificate on October 28th at approximately 02:35 PT, which we thought resolved the issue. However, two hours later, at 04:45 PT on October 28th, we noticed a further incident the outage had created for Awair Omni devices: during the period of time the certificate was expired, Transport Layer Security \(TLS\) communications between Awair Omni devices and our servers failed. When Omni devices could not synchronize their clock with our servers \(due to the expired certificate\), these repeated failures caused devices to crash and remain disconnected. _**How to resolve remaining issues with Omni:**_ NOTE: Your Awair Omni device does not require a factory reset. It may, however, require a manual power cycle. To power cycle your Awair Omni: * Hold down the right side button for ~3-5 seconds * Let go for a second * Push the button again until the Awair logo appears on the front of the device. The service disruption for some Omni devices began on October 27th at 07:30 PT. You may see gaps in data from then until October 28th at 04:45 PT and/or until you power cycle your device. _**How we will prevent this from happening in the future:**_ Awair Omni’s response to the server disconnection was an anomaly, one that we have addressed and debugged to make sure that it does not occur in the future. Additionally, we have internally set up cross-checks within the company regarding the ownership of the certification renewal to guarantee that a lapse in renewal time will not be a repeated incident. We are deeply apologetic for the inconvenience caused by this disruption in service. We are immensely appreciative of your patience and support as we grow as a global company. If you find that your team or your Omni devices continue to have any issues following a power cycle, please reach out to [[email protected]](mailto:[email protected]).
Your product runs on more than AWAIR
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services