fix(trino,pyspark): improve null handling in array filter

[stephen-bowser]

Description of changes

This fixes an issue with the pyspark array filter function. The original implementation does not account for handling nulls correctly in the input array.

I'm not too faimiliar with SqlGlot, but by copying the implementation from duckdb, I was able to get all the test cases passing. Happy to take feedback if there's something I've missed though.

See this issue for further details

Issues closed

Resolves #10201

[stephen-bowser]

Looks like there is also an issue in the trino implementation for the same reason as there was in pyspark.
I had a go at fixing that one as well, but I'm not familiar enough with SqlGlot to work through it.
That being said, I think a better implementation for this backend could be something like:

• use zip to combine array elements with their indices into a struct
• apply the filter function
• return back the values that passed the filter function, but without their associated indexes

[gforsyth]

Hey @stephen-bowser -- thanks for putting this together!
It's fine to only fix the pyspark backend, you can add a marker to xfail Trino with a TODO for us to handle redoing the implementation.

I can see that you copied the values check from the previous test -- that would work if we weren't dealing with Pandas NULL/NaN nonsense, so you're getting a test failure because Pandas makes things nan and also coerces columns to float because of that.

You might be better served by using to_pyarrow() to trigger execution and then comparing values that way, with something that has a proper notion of NULL

[cpcloud]

Went ahead and fixed the trino backend here.

[cpcloud]

Fix on the way for using pyarrow to test

[gforsyth]

Thanks for putting this in @stephen-bowser !