CRD API No Longer Supports Watch Bookmarks

[paxbit]

Hi,

#185 seems to have removed support for watch bookmarks for custom objects, which currently breaks our application.

See:
2d0452d#diff-7ca22d542e5e2fc5a31472bd4643aa4db732e8666e7b4bba4acd0a123aeb389b

I believe this is a bug. However as I'm not familiar with the details of the client generation process I cannot say if it is due to the new 5.4.0 generator itself, the pre-processing or the actual kubernetes openapi spec. I checked all three but quickly found them to be rabbit holes. So hopefully you @tomplus can more easily say what's wrong here.

Thanks!

paxbit

[tomplus]

Sorry to hear that. AFAIK, bookmarks have never supported by the library so maybe the spec forces it somehow...

Could you share more information? What exception do you get? How do you use Watch()? Which version of K8s do you use?

[paxbit]

Hi @tomplus,

Thank you for you quick reply!

We're on kubernetes 1.23.2.

To reproduce it yourself you could first create the following CRD manually, then run the script below:
Note: As an aside and probably another issue, which so far I have not created, I could not find a way to create the CRD itself using kubernetes_asyncio. Is there a way, which does not involve calling call_api directly?

DemoCrd Manifest

Create this first using kubectl create -f ...

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: democrds.foo.bar
spec:
  scope: Namespaced
  group: foo.bar
  names:
    kind: DemoCrd
    plural: democrds
    singular: democrd
  conversion:
    strategy: None
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true

Afterwards run this script:

Code

import asyncio
from kubernetes_asyncio import client, config, watch
from kubernetes_asyncio.client.exceptions import ApiException, ApiTypeError


def main():
    loop = asyncio.get_event_loop()
    loop.run_until_complete(config.load_kube_config())

    api_client = client.ApiClient()
    v1_api = client.CoreV1Api(api_client)
    crd_api = client.CustomObjectsApi(api_client)

    ns = "default"

    cro = {
        "apiVersion": "foo.bar/v1",
        "kind": "DemoCrd",
        "metadata": {"name": "the-cro-1", "namespace": ns},
        "spec": {"list": ["one", "two"]},
    }

    async def _task():

        try:
            try:
                await crd_api.delete_namespaced_custom_object(
                    "foo.bar", "v1", ns, "democrds", "the-cro-1"
                )
            except ApiException as e:
                if e.status == 404:
                    pass
                else:
                    raise

            await crd_api.create_namespaced_custom_object(
                "foo.bar", "v1", ns, "democrds", cro
            )

            # here it works...
            pod_list = await v1_api.list_namespaced_pod(
                ns, allow_watch_bookmarks=True
            )

            try:
                # here it does not...
                cro_list = await crd_api.list_namespaced_custom_object(
                    "foo.bar", "v1", ns, "democrds", allow_watch_bookmarks=True
                )
            except ApiTypeError:
                # ApiTypeError: Got an unexpected keyword argument 'allow_watch_bookmarks'
                # to method list_namespaced_custom_object
                pass

        finally:
            await api_client.close()

    loop.run_until_complete(asyncio.wait([_task()]))


if __name__ == "__main__":
    main()

Still works with 21.7.1, broken in 22.6.0

I use the watch in an Informer style implementation inspired by the following two implementations:

• https://github.com/kubernetes/client-go/blob/af4295f5013b55c45b05f1e7850f7fa9c4dbfdb4/tools/cache/shared_informer.go#L134
• https://github.com/JacobHenner/kubernetes_asyncio_informer/blob/main/kubernetes_asyncio_informer/informers.py

I need the bookmarks to get around implementing already seen event re-emission handling.

[tomplus]

Thanks for detailed information, it helps a lot.

I can confirm that the latest version of the library was generated using the old CRD spec and the attribute "allow_watch_bookmarks", which you use, is missing. It's my mistake, I used a stale branch 🤦‍♂️ when I was working on switching to the new generator.

[tomplus]

Fixed in v22.6.4.

[paxbit]

Awesome, Thanks for the timely fix :) It's working now.