"Unable to decode id token" on singleuser pod spawn with short-lived id tokens

[Aleksei-Poliakov]

Bug description

In our Jupyterhub setup with get a Refresh Token from OIDC that is long-lived, e.g. 24 hours. That can be exchanged for a short-lived (5 minutes) ID token.

We upgraded from OAuthenticator from 16.3.1 to 17.3.0 and started seeing "Unable to decode id token" pretty often.

Digging into the code it seems like the problem comes from the fact that OAuthenticator now implements refresh_user API from Jupyterhub's Authenticator class. The base implementation (that was used in 16.3.1) just returns true; new implementation (in 17.3.0) calls into _token_to_auth_model, which tries to parse out id token inside token_to_user. Since ID token is likely expired at this point - the singleuser pod spin up fails.

To work around this I just went back to the original behavior of returning True all the time from refresh_user.

How to reproduce

1. Go to hub, since you haven't been authenticated yet - it will redirect you to the OIDC provider, after signing on there you will be redirected to the hub and establish a session.
2. Spin up a singleuser pod, wait long enough for the id token to expire (but NOT refresh token) and shut down your singleuser pod.
3. Go back to the hub, try to spin up your singleuser pod again.
4. See error

Expected behaviour

During the refresh_user workflow the fact that id token expired shouldn't cause the refresh to halt. The id token isn't used for anything in the refresh_user flow anyway, e.g. the actual refreshing happens using refresh token.

Actual behaviour

Because the code is structured in a way that ends up parsing id token and the token is expired, this causes the entre reresh_user method to fail.

Your personal set up

• OS: Linux jupyterhub-apol-hub-7ddbc55c6-tlbrz 6.1.129
• Version(s): jupyterhub 5.2.1, Python 3.11.9

# paste output of `pip freeze` or `conda list` here

Configuration

jupyter-events==0.12.0
jupyter-telemetry==0.1.0
jupyterhub==5.2.1
jupyterhub-firstuseauthenticator==1.0.0
jupyterhub-hmacauthenticator==1.0
jupyterhub-idle-culler==1.3.1
jupyterhub-kubespawner==6.2.0
jupyterhub-ldapauthenticator==1.3.2
jupyterhub-ltiauthenticator==1.6.2
jupyterhub-nativeauthenticator==1.2.0
jupyterhub-tmpauthenticator==1.0.0
jupyterhub-traefik-proxy==2.1.0
oauthenticator==17.3.0
oauthlib==3.2.2

[Aleksei-Poliakov]

Reading the code more closely - I think this

oauthenticator/oauthenticator/oauth2.py

Lines 1384 to 1392 in ba6819e

	except HTTPClientError as e:
	# assume any client error means an expired token
	# most likely 401 or 403 for well-behaved providers
	if 400 <= e.code < 500:
	self.log.info(
	f"Error refreshing auth with current access_token for {user.name}: {e}. Will try to refresh, if possible."
	)
	else:
	raise

was supposed to handle expired tokens, but it expects a HTTPClientError to be thrown whereas with an expired ID token it's jwt.decode that is failing (and then

oauthenticator/oauthenticator/oauth2.py

Lines 1148 to 1150 in 109018e

	raise web.HTTPError(
	500, f"Unable to decode id token: {id_token}\n{err}"
	)

throws an HTTPError with code 500, which does NOT pass the < 500 check)

[consideRatio]

Beautiful debugging! I'd be happy to review a PR about this!

[Aleksei-Poliakov]

@consideRatio - what do you think is the right way to fix this? I am still looking through the code and for example for Entra (formerly known as AzureAD)

oauthenticator/oauthenticator/azuread.py

Lines 69 to 77 in 109018e

	async def token_to_user(self, token_info):
	id_token = token_info['id_token']
	decoded = jwt.decode(
	id_token,
	options={"verify_signature": False},
	audience=self.client_id,
	)
	return decoded

the settings for jwt.decode call are different, e.g. compare this with:

oauthenticator/oauthenticator/oauth2.py

Lines 1135 to 1145 in 109018e

	# Here we parse the id token. Note that per OIDC spec (core v1.0 sect. 3.1.3.7.6) we can skip
	# signature validation as the hub has obtained the tokens from the id provider directly (using
	# https). Google suggests all token validation may be skipped assuming the provider is trusted.
	# https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation
	# https://developers.google.com/identity/openid-connect/openid-connect#obtainuserinfo
	return jwt.decode(
	id_token,
	audience=self.client_id,
	options=dict(
	verify_signature=False, verify_aud=True, verify_exp=True
	),

For the purposes of retrieving data from the token the less restrictive list of validations makes more sense (and again, there is a comment seemingly saying that all validation may be skipped, yet this isn't what actually happens in the code), but making such a change of OAuthenticator would be a breaking change.

---

Note - this is how the defaults work in PyJWT:
https://github.com/jpadilla/pyjwt/blob/1f49f41d4c422105c45ebcb9541f7cc13cbf855e/jwt/api_jwt.py#L64-L77

    def _merge_options(self, options: Options | None = None) -> FullOptions:
        if options is None:
            return self.options

        # (defensive) set defaults for verify_x to False if verify_signature is False
        if not options.get("verify_signature", True):
            options["verify_exp"] = options.get("verify_exp", False)
            options["verify_nbf"] = options.get("verify_nbf", False)
            options["verify_iat"] = options.get("verify_iat", False)
            options["verify_aud"] = options.get("verify_aud", False)
            options["verify_iss"] = options.get("verify_iss", False)
            options["verify_sub"] = options.get("verify_sub", False)
            options["verify_jti"] = options.get("verify_jti", False)
        return {**self.options, **options}

[consideRatio]

@Aleksei-Poliakov I find this to be such a beautiful investigation and writeup! Thank you!!!

@consideRatio - what do you think is the right way to fix this?

I'll have a look!

[consideRatio]

I've taken a look but have so far not found a suitable solution. I hope to find more time, but I think it may take until Monday before I do.

[consideRatio]

@Aleksei-Poliakov i opened #795 with some changes, but I think there may be a higher level issue to address that I mentioned in the PR. I'm low on time, if you are able to push onwards please do, I'd be happy to priotize review effort!