Application default credentials aren't reloaded on expiration

[imathews]

Describe the issue
For security purposes, our application default credentials are set to expire every 16 hrs (in local dev environments). When a developer refreshes their credentials, GCS fuse doesn't seem to reload the credential file, and continues to get invalid_grant errors since it is using the old, expired credentials. This requires the additional step of restarting GCSFuse — not the end of the world, but an extra layer of complexity that would ideally be handled by fuse.

To Collect more Debug logs
Steps to reproduce the behavior:

1. Create application default creations: gcloud auth application-default login
2. Start GCS fuse, which will use these credentials
3. Revoke / expire the credentials. GCS fuse will start logging errors.
4. Refresh the credentials: gcloud auth application-default login
5. Within a reasonable period of time, we would expect fuse to reload the credentials file and start working again, but it currently doesn't.

System (please complete the following information):

• OS: Debian 11 (bullseye)
• Platform: local kubernetes
• Version: 1.4.0

[ashmeenkaur]

Hi @imathews,

Thanks for bringing this to our attention. We're working to determine if we can fix this directly in GCSFuse. We have raised a question for Oauth2 team here. In the meantime, I was wondering if using GCSFuse --token-url flag be a possible work around for you?

Thanks,
Ashmeen

[imathews]

Thanks @ashmeenkaur. Right now our workaround is to just to restart GCSFuse on token expiration, which is a bit easier (given our dev setup) than passing the --token-url flag. Though that would likely work too.

FWIW, I believe that other google cloud libraries are handling this properly. Specifically, the various GCP node clients (which I believe all rely on https://github.com/googleapis/google-auth-library-nodejs).