feat(pyspark): support partition_by key for PySpark file writes

[deepyaman]

Is your feature request related to a problem?

I can't specify partition key.

Describe the solution you'd like

I'd like to be able to specify partition key. Basically, PySpark takes partitionBy key. Need to figure out if just want to alias partition_by passed to the write method as partitionBy.

Need to also verify read path works; current test works using / * / * wildcard pattern; PySpark may require resolving that to a set of paths to read.

What version of ibis are you running?

8

What backend(s) are you using, if any?

PySpark

Code of Conduct

• I agree to follow this project's Code of Conduct

[mark-druffel]

My team came across this and need to use partition_by. Is this already being worked on, if not we'd love to open a PR and give it a shot

@jakepenzak

[cpcloud]

It is not being worked on, please give it a go!

[jakepenzak]

Threw up this PR (#10850) that adds partition_by argument for create_table method to enable partitioning in writes to hive database, but missed to_parquet method not working for partitioning.

Looks like for PySpark backend, partitioning is available for to_delta natively via pyspark.sql.DataFrameWriter kwargs. However, it isn't supported for to_parquet due to, what I believe to be, no native support for partitioning in pyarrow.parquet.ParquetWriter? We can potentially override to_parquet method in PySpark backend to leverage pyspark.sql.DataFrameWriter following same pattern as overridden to_delta method. I'll go ahead and follow this pattern, but open to any other thoughts or recommendations?