Google says the disruption to the global authentication system that affected most consumer-oriented series on Monday was caused by an error in the automated quota management system that affected the Google User ID service.
This global failure has prevented users from logging in to their accounts and logging in to all Cloud services.
As a direct result, users were unable to access Gmail, YouTube, Google Drive, Google Maps, Google Calendar and a few other Google services on Monday, December 14th.
During the break, users were unable to send emails via Gmail mobile apps or receive POP3 emails for desktop clients, while YouTube visitors saw error messages stating that “There was a server problem (503 ) – Touch to try again ”.
The impact of the interruption and the main cause
“On Monday, December 14, 2020, from 03:46 to 04:33 US / Pacific, the issuance of credentials and search of account metadata for all Google user accounts failed,” Google explained. “Therefore, we were unable to verify that user requests were authenticated and transmitted 5xx errors on almost all authenticated traffic.
“Most authenticated services had a similar impact on the control plan: high error rates across all Google Cloud Platform and Google Workspace APIs and consoles.”
The main cause behind the outage was a decrease in the capacity of Google’s central identity management system, due to an error that affected the automatic quota management system.
This led to problems verifying that Google user requests were authenticated, resulting in errors displayed on all authentication attempts.
Global identity management system
Google’s user ID service, which has been at the root of Google’s major outage since Monday, stores unique identifiers for all Google accounts and manages authentication credentials for both OAuth tokens and cookies.
It also stores user account data in a distributed database, which uses Paxos protocols to coordinate updates during authentication.
Because the User ID Service will reject requests when it detects outdated data for security reasons, all Google customer-oriented services that require Google OAuth access became unavailable immediately after the service began to experience problems and issued outdated identifiers.
“Google uses an evolving suite of automation tools to manage the share of various resources allocated to services,” the company said in a summary report released today.
“As part of an ongoing migration of the User ID Service to a new quota system, a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system have been left in place. and incorrectly reported the use of the user ID service as 0.
“An existing grace period on the application of quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to reduce the quota allowed for the User ID service and triggering this incident.”
Although safety controls are in place to prevent unplanned quota changes, they have failed to respond appropriately to the scenario of a single zero-reported loading service.
“As a result, the quota for the account database was reduced, which prevented the Paxos leader from writing,” Google added. “In a short time, most reading operations became obsolete, which led to errors in authentication searches.”
Google said the major outage also affected the company’s internal users and tools, causing delays during the outage investigation and reporting status updates.
Gmail affected by a second outage in one day
Gmail was affected by a second outage for a combined total of about 7 hours after the authentication issues were resolved on Monday, an outage that affected a subset of Gmail users who experienced email delivery issues.
“The error message indicates that the email address did not exist and, as a result, the affected emails were never delivered,” Google said in another report released today. “Affected senders may have received a rejection email generated by an intermediate SMTP service.”
“In some cases, the full SMTP error message was quoted in the rejection email. The behavior of these messages depended on external SMTP clients connecting to the Google SMTP service.”
The cause of this second outage was a continuous migration to update the underlying Gmail SMTP inbound configuration system.
“A configuration change during this migration moved the formatting behavior of a service option to incorrectly provide an invalid domain name, instead of the intended” gmail.com “domain name, to the Google Inbox SMTP, ”Google said.
“As a result, the service incorrectly turned searches for certain email addresses ending in ‘@ gmail.com’ into non-existent email addresses.
“When the Gmail user account service verified each of these non-existent email addresses, the service could not detect a valid user, resulting in SMTP error code 550.”