Filtering and mapping nested outputs

[nicoonoclaste]

Hi!

I'm fairly new to Pulumi, and trying it out to bring up and manage a Kubernetes cluster, as a demo to show at the local Python user group.

Something I'm currently having trouble with, is filtering an Output[List[k8s.core.v1.Service]] based on metadata and extract status information. To be precise, given a chart, I need to compute (and export as a stack output) something like

chart.resources.apply(lambda resources: [
    ingress.ip
    for svc in resources
    if isinstance(svc, Service) and svc.metadata.name == "cilium-ingress"
    for ingress in svc.status.load_balancer.ingress
])

However, svc.metadata and svc.status are both Outputs of their own, so this doesn't work

• svc.status.load_balancer.ingress cannot directly be iterated on
• svc.metadata.name == "cilium-ingress" is always false, as the left operand is an Output object.

The closest I got so far is

chart.resources.apply(lambda resources: [
    svc.status.load_balancer.apply(lambda lb: [ ingress.ip for ingress in lb.ingress or [] ])
    for svc in resources
    if isinstance(svc, Service)
])

which produces a list of lists (instead of a flat list) and isn't filtering based on metadata

[Frassle]

Yeh this is a tricky one because of the different layers. I think you want something like:

chart.resources.apply(lambda resources: pulumi.Output.all(*[
    svc.metadata.name.apply(lambda name: [name == "cilium-ingress", svc])
    for svc in resources
])).apply(lambda resources: pulumi.Output.all(*[
    svc.status.load_balancer.ingress    
    for [filter, svc] in resources
    if filter
]))

That is first iterate the resources list and apply on the name so that you can return an output<[bool, Service]>, then use all to turn that list<output<[bool, Service]>> into an output<list<[bool, Service]>>. Then apply on that output so you can filter out the false bools and just access the ingress property, then use all again to get back an output<list> rather than an output<list<output>.

[Frassle]

Oh just realised I missed the ip attribute, so I think that's another layer so another apply and loop

[nicoonoclaste]

Oh, thanks a bunch. I missed that Output.all(*x) converts a List[Output[_]] into an Output[List[_]], and that I could apply that monad transformation here 😅

[nicoonoclaste]

There's something I'm not understanding in this example, because used as-is I get the following error:

AttributeError: 'RoleBinding' object has no attribute 'status'

... and indeed, if I modify the 2nd apply to output svc (instead of svc.status.load_balancer.ingress) I get RoleBinding objects which don't match the predicate svc.metadata.name == "cilium-ingress"; see output extracted with

pulumi stack output cilium-ingress --json | jq . | gh gist create -d 'Buggy(?) pulumi output' -f 'cilium-ingress.json' -

PS: I published the repo where I was encountering the issue, with instruction how to reproduce it with kind.

[nicoonoclaste]

BTW, this is what I ended up using for the moment; it still doesn't filter on metadata.name, as I couldn't get that to work, and instead assumes nothing else (in the chart) defines a LB:

chart.resources.apply(lambda resources: [
    svc.status.load_balancer.apply(lambda lb: [ ingress.ip for ingress in lb.ingress or [] ])
    for svc in resources
    if isinstance(svc, Service)
]).apply(lambda out:
    # convert a List[Output[_]] into an Output[List[_]]
    pulumi.Output.all(*out)
).apply(lambda out: reduce(operator.add, out, [])),

[nicoonoclaste]

Sorry for the late answer, I thought I had already posted my solution:

chart.resources.apply(lambda resources: pulumi.Output.all(*(
    pulumi.Output.all(
        ips = svc.status.load_balancer.apply(lambda lb: [ ingress.ip for ingress in lb.ingress or [] ]),
        pred = svc.metadata.apply(lambda m: m.name == "cilium-ingress"),
    )
    for svc in resources
    if isinstance(svc, k8s.core.v1.Service)
))).apply(lambda out: reduce(lambda acc, x: acc + (x["ips"] if x["pred"] else []), out, [])),

Thanks again, @Frassle, for the help <3

I plan to write a more-ergonomic API for that sort of thing (and I'm now wishing Python had monad comprehensions) which I'll happily share if/when that happens :3