[bitnami/spark] Including py4j lib in PYTHONPATH

[ShivKJ]

Name and Version

docker.io/bitnami/spark:3.5.1-debian-12-r12

What is the problem this feature will solve?

It will enable user to run pyspark application in docker container.

What is the feature you are proposing to solve the problem?

py4j is not in the PAYTHONPATH, so one can not run/submit spark application for python application in the docker container. We should add py4j in PYTHONPATH.

What alternatives have you considered?

I am using PYTHONPATH=$(ZIPS=(/opt/bitnami/spark/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH, to update the PYTHONPATH.

[ShivKJ]

Note that, the above is NOT a problem when we run the spark-submit command and is a problem when python command is used to execute the python script.

[javsalgar]

Hi,

Thank you so much for reporting. Would you like to submit a PR updating this environment variable? It could be done in the dockerfile or in the entrypoint, depending on the complexity

[ShivKJ]

@javsalgar Sure, I will create a PR. I think, it will be better to add py4j directly in the Dockerfile where the PYTHONPATH is updated, as it is a crucial library for enabling Python applications to communicate with Spark.

[ShivKJ]

@javsalgar I have created the PR. Initially, I intended to add the changes to the Dockerfile, but it became complex. Therefore, I moved the changes to entrypoint.sh as you suggested.