Skip to content

Fix for authentication expiry - Automatically attempt to refresh auth tokens before they expire #317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 7, 2021

Conversation

drobison00
Copy link
Contributor

Resolves #286

@jacobtomlinson Putting this up to get feedback on the approach. If it seems reasonable, I can extend it to other authentication methods.

Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

From what I can see this will handle updating your local k8s config with tokens from GKE periodically. I'm not sure I see where the auth gets refreshed though. I expect the client objects will pick up the config once and then also need refreshing.

What do you think?

@drobison00
Copy link
Contributor Author

@jacobtomlinson

This should actually be handled by this chunk:

class AutoRefreshConfiguration(Configuration):
    """
    Extends kubernetes_async Configuration to support automatic token refresh.
    Lets us keep track of the loader object
    """
    def __init__(self, loader, *args, **kwargs):
        super(AutoRefreshConfiguration, self).__init__(*args, **kwargs)
        self.refresh_api_key_hook = AutoRefreshConfiguration.refresh_api_key
        self.loader = loader

Every time you have a cluster interaction via __call_api, it retrieves the latest credentials for Configuration in client.api_client.py with a call to 'update_params_for_auth', which eventually calls 'get_api_key_with_prefix' on the Configuration object. At this point, if you've set 'refresh_api_key_hook', it will be called before the auth token is retrieved from the Configuration.

I wanted to trigger the auth update here, but Configuration isn't written to be async, so I don't think we can await here. Instead, I have the periodic credential refresh on the loader, and then in 'refresh_api_key_hook' update to the latest token retrieved by the loader.

@jacobtomlinson
Copy link
Member

Ah ok great!

@drobison00 drobison00 changed the title [WIP] Proof of concept fix for authentication expiry. Fix for authentication expiry - Automatically attempt to refresh auth tokens before they expire Apr 6, 2021
Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great thanks @drobison00

@jacobtomlinson jacobtomlinson merged commit ca70697 into dask:main Apr 7, 2021
@orcahmlee
Copy link

Hi, I currently hit the same issue about #286 and seems this PR would fix that.
Do yo have any plan to update new version including this feature on the PyPI?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KubeCluster loses the ability to run commands against a deployed cluster after periods of inactivity.
3 participants