ServiceMax incident
Google Maps SDK that is impacting customers using ServiceMax GO application
ServiceMax experienced a notice incident on April 24, 2020 affecting Critical Announcements, lasting 6d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Apr 24, 2020, 12:16 AM UTC
Currently there is a global issue with Google Maps SDK that is impacting customers using ServiceMax GO application. Recommendation from Google is to clear data for the applications, not just the cache. For ServiceMax GO users this means uninstalling and reinstalling the application. This issue is reported in the Google Issue Tracker - https://issuetracker.google.com/issues/154855417. Google is actively working on a resolution, with full resolution expected to be complete by Thursday, 2020-04-23 19:45 US/Pacific (PST).
- monitoring Apr 24, 2020, 04:03 AM UTC
Google has posted the following on resolution times: - As for Maps SDK for iOS, full resolution was expected to complete by Thursday, 2020-04-23 19:45 PDT. - As for Maps SDK for Android, full resolution is expected to complete by Thursday, 2020-04-23 22:45 PDT.
- monitoring Apr 24, 2020, 07:31 AM UTC
Google has posted the following: Summary: Google Maps SDK is crashing; partially resolved Description: We believe the crashes in the Google Maps SDKs should be fixed for iOS. In our previous updates we mentioned that we expected crashes in Maps SDK for Android to disappear by the latest by Thursday, 2020-04-23 22:45 PDT. However, based on internal monitoring and on external developer reports, it appears there are crashes remaining in a number of applications. We apologize for setting incorrect expectations, and we'll continue our investigations. Maps SDK customers for whom clearing application data is safe can recommend their users clear data for the applications (not just the cache), to speed up recovery. We will provide an update by Friday, 2020-04-24 00:15 US/Pacific with current details. Workaround: Clear application data (not just the cache).
- monitoring Apr 24, 2020, 07:37 AM UTC
Google has posted the following: Summary: Google Maps SDK is crashing; partially resolved Description: Based on additional investigations, it appears the remaining crashes might be due to some race condition, where the updated data doesn't get fetched early enough, leading to a crash loop. We are working with highest priority on a new fix, but current ETA for deployment is around 48 hours. At this point, the recommended way to fix the issue is to "Clear data" of the application, or delete + reinstall it. We will provide an update by Friday, 2020-04-24 02:15 US/Pacific with the latest details. Workaround: Clear the application data (not just the cache).
- monitoring Apr 24, 2020, 11:37 AM UTC
Google have posted the following: Summary: Google Maps SDK is crashing; partially resolved Description: Here's an update on the current mitigation and resolution steps: - The backend service that caused this crash has been hot-fixed on 2020-04-23 15:00 PT (13 hours ago), and crashes have steadily reduced since. Remaining crashes are due to client-side caching, which is why we recommend the workarounds mentioned. - For Android users, we are currently working on a release of the Google Play Services (that contains the Maps SDK runtime) that will also fix this issue client-side. The previous ETA of 48 hours as communicated previously still applies. Thank you to everyone for your continued patience. We will provide another update here by Friday, 2020-04-24 06:00 US/Pacific. (~ 2 hours from now) Diagnosis: Crash of the Google Maps Platform mobile SDKs (iOS + Android) at load Workaround: Clear the affected app's data (not just the cache), or uninstall-reinstall the affected app/s.
- monitoring Apr 24, 2020, 04:14 PM UTC
Google have posted the following: Summary: Google Maps SDK is crashing; partially resolved Description: Here's an update on the current mitigation and resolution steps: - The backend service that caused this crash has been hot-fixed on 2020-04-23 15:00 PT (17 hours ago), and we have seen a significant decrease in crashes since then. The remaining crashes are due to client-side caching, which is why we recommend the workarounds mentioned. - For Android users, although we are still expediting a new release of the Google Play Services (that contains the Maps SDK runtime) that will also fix this issue client-side, it appears our initial 48 hour estimate for rolling out the fix was overly optimistic. We are currently reassessing the release timeline. If Android developers wish to roll out their own client-side workarounds via an update to their apps in Google Play Store, but experience Play Store review approvals delays, please file a support case via https://developers.google.com/maps/documentation/android-sdk/support#contact-maps-support, and we can escalate the request internally. We are also aware of a few customer reports where clearing the application data or uninstalling and reinstalling the application has not fixed the issue, and we are continuing to investigate the issue. Thank you to everyone for your continued patience. We will provide another update here by Friday, 2020-04-24 11:00 US/Pacific. (~ 2 hour from now) Diagnosis: Crash of the Google Maps Platform mobile SDKs (iOS + Android) at load Workaround: Clear the affected app's data (not just the cache), or uninstall-reinstall the affected app/s.
- monitoring Apr 25, 2020, 12:54 AM UTC
Google posted the following Apr 24, 2020 05:17PM PST Summary: Google Maps Android and iOS SDK is crashing Description: Here's an update on the current mitigation and resolution steps: The backend service that caused this crash has been hot-fixed on 2020-04-23 15:00 PT, and we have seen a significant decrease in crashes since then. The remaining crashes are due to client-side caching, which is why we recommend the workarounds mentioned. For Android: We are expediting a new release of the Google Play Services (that contains the Maps SDK runtime) which will fix this issue client-side. Our initial 48 hours estimate for rolling out this fix was overly optimistic, and we are currently reassessing the release timeline. If Android developers wish to roll out their own client-side workarounds via an update to their apps in Google Play Store, but experience Play Store review approvals delays, please file a support case with your app’s Package ID: https://developers.google.com/maps/documentation/android-sdk/support#contact-maps-support. Our Support Team will internally escalate your request and expedite the approval. We are also aware of customer reports where clearing the application data or uninstalling and reinstalling the application has not fixed the issue, and we are continuing to investigate the issue. Thank you to everyone for your continued patience.
- monitoring Apr 25, 2020, 07:49 AM UTC
Google posted the following Apr 25, 2020 03:49AM GMT Summary: Google Maps Android and iOS SDK is crashing -- Mitigation Offered Description: The backend service that caused this crash has been hot-fixed on 2020-04-23 15:00 PT, and we have seen a significant decrease in crashes since then. The remaining crashes are due to client-side caching, which is why we recommend the workarounds mentioned. For Android: We are expediting a new release of the Google Play Services (that contains the Maps SDK runtime) which will fix this issue client-side. Our initial 48 hours estimate for rolling out this fix was overly optimistic, and we are currently reassessing the release timeline. For developers rolling-out their own code mitigation, Support can be found at https://developers.google.com/maps/documentation/android-sdk/support#contact-maps-support and https://developers.google.com/maps/documentation/ios-sdk/support#contact-maps-support We are also aware of customer reports where clearing the application data or uninstalling and reinstalling the application has not fixed the issue, and we are continuing to investigate the issue. Thank you to everyone for your continued patience.
- monitoring Apr 26, 2020, 09:48 AM UTC
Google have posted the following at Apr 25, 2020 4:00PM GMT Summary: Google Maps Android and iOS SDK is crashing its own thread, causing app crashes - Mitigation Offered Description: Here's an update on the current mitigation and resolution steps: Please note that we are shifting to longer intervals between updates and will provide the next update by Sunday, 2020-04-26 18:00 US/Pacific. The backend service that caused this crash has been hot-fixed on 2020-04-23 15:00 PT, and we have seen a significant decrease in crashes since then. The remaining crashes are due to client-side caching, which is why we recommend the workarounds mentioned. For Android: We are expediting a new release of the Google Play Services (that contains the Maps SDK runtime) which will fix this issue client-side. Our initial 48 hours estimate for rolling out this fix was overly optimistic, and we are currently reassessing the release timeline. Currently the best course of action is for developers to roll out their own client-side workarounds via an update to their apps. If Android developers wish to roll out their own client-side workarounds via an update to their apps in Google Play Store, but experience Play Store review approvals delays, please file a support case with your app’s Package ID: https://developers.google.com/maps/documentation/android-sdk/support#contact-maps-support. Our Support Team will internally escalate your request and expedite the approval. If iOS developers wish to roll out their own client-side workarounds via an update to their apps, but experience review approvals delays, please file a support case with your app’s ID, Bundle ID, name, and the version number that needs to be reviewed: https://developers.google.com/maps/documentation/ios-sdk/support#contact-maps-support. Our Support Team will help expedite the approval. We are also aware of customer reports where clearing the application data or uninstalling and reinstalling the application has not fixed the issue, and we are continuing to investigate the issue. Thank you to everyone for your continued patience. Diagnosis: Crash of the Google Maps Platform mobile SDKs (iOS + Android) at load. Workaround: * Clear the affected app's data (not just the cache), or uninstall then reinstall the affected app(s).
- monitoring Apr 27, 2020, 03:03 PM UTC
Google are continuing to work on a resolution for this issue, but no ETA is currently available for this to be fixed. The latest update and information can be found on the Google Issue Tracker - https://issuetracker.google.com/issues/154855417
- monitoring Apr 28, 2020, 08:55 AM UTC
Google have advised the following: We started rolling out updated versions of the Google Play Services on Android, at this stage at a small scale. We are monitoring how the roll-out is progressing throughout the day and plan to target a larger scale roll-out this evening PT based on our findings. We are continuing to monitor this tracker for updates from the community and will share more details about the progress of the roll-out according to the next update time communicated below (sooner if there are significant updates to be shared in the interim). Further updates can be found here - https://issuetracker.google.com/issues/154855417
- monitoring Apr 29, 2020, 12:58 PM UTC
Google have advised the following: The updates to Google Play Services to fix the crash is now available to all devices with Google Play Services version 17.4.55 and newer. End users' devices need to download and apply the updates. There will be no change to the version number of Google Play Services on the device after the update is installed. We continue to see crash rates of Android Maps SDK v2, which is included in Google Play Services, decrease significantly. They have also advised that an Incident Report will be made available once fully resolved. Further updates can be found here - https://issuetracker.google.com/issues/154855417
- resolved Apr 30, 2020, 09:56 AM UTC
Google have advised that this issue is now resolved. Google have advised that an incident report will be made available once they have completed internal discussions, in the meantime full details can be found on the Google Incident Tracker - https://issuetracker.google.com/issues/154855417#comment509