feat(pyspark): support partitioning in PySpark backend file writes

[jakepenzak]

Description of changes

• Enabled partitioning in create_table and to_parquet methods for pyspark backend (already existed for to_delta)
  • Added partition_by argument to create_table method for PySpark backend
  • Overrode to_parquet method for PySpark backend to leverage pyspark.sql.DataFrameWriter using a similar pattern as to_delta override, enabling corresponding kwargs for partitioning
  • Added corresponding tests to ensure partitioning behaves as expected

Issues closed

• Resolves feat(pyspark): support partition_by key for PySpark file writes #8900

[cpcloud]

LGTM, thanks!

🚢 it!

[cpcloud]

I'll fix up the remote test failures, which should probably be skipped given that the writing location for a remote spark instance isn't well-defined (unless it's a bucket, but the tests deal only in local file paths).