We're running an application that caches some details about users internally.
Last month (December 6th) this has stopped working intermittently across the day, and stopped working entirely late in the evening CET.
The issue appeared simultaneously across all four projects in the DTAP street as well as on all the domains it's authorized on. No recent deploys have been done prior to this and this code has been running without issues for about two years.
The app is running on the App Engine Standard Java 8 environment, we keep a watch on the Users resource via the Admin Directory API. The call is authorized with a Service Account impersonating an admin user.
The exception thrown:
com.google.api.client.auth.oauth2.TokenResponseException: 401 Unauthorized
at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:394)
at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:493)
Unlike most errors from this API this one doesn't contain any details.
The service account still exists, the key files are present and the scopes are set on each relevant domain.
The error is identical when using both scopes listed in the docs (https://developers.google.com/admin-sdk/directory/v1/reference/users/watch) https://www.googleapis.com/auth/admin.directory.user.readonly and https://www.googleapis.com/auth/admin.directory.user
The admin user has access to the directory as well, depending on the domain it's either a super admin or has been granted more granular access.
Depending on the version currently active the application uses the "com.google.apis:google-api-services-admin-directory" java client library at version directory_v1-rev105-1.25.0 or directory_v1-rev93-1.23.0
This watch has an expiration of an hour and is refreshed hourly. The projection is set to "full" the view type to "admin_view" and the channel type is "web_hook" pointed at a servlet within the same project.
Calls to users.get within the application using the same (cached) credentials, service account, admin user, and api client have continued working so far.
Since the error has appeared across multiple projects and domains simultaneously we're wondering if anything has changed on the API that could explain this issue.