spark 3.0.0 module (ES 6.5)

[avnerl]

added spark 3.0.0 module to branch 6.5

[mmigdiso]

Btw, although I desperately need this feature (and I have a fork on which Ive added scala 212+spark3.0 support), I think
@jbaiera may not merge this.
see #1308
and
#1423

On the other hand, I believe that we should somehow unblock this situation....

[avnerl]

Btw, although I desperately need this feature (and I have a fork on which Ive added scala 212+spark3.0 support), I think the @jbaiera may not merge this.
see #1308
and
#1423

On the other hand, I believe that we should somehow unblock this situation....

well, this is pending for too long, IMHO.

merge or not - this fork was tested on a live environment where we write dataframes to elasticsearch.
tech stack (for those curious):

• spark 3.0.0 on k8s 1.15.3
• elastic 6.5.4
• hadoop 2.7.3

tests works for all hadoop distros:

./gradlew -Pdistro=hadoopStable
./gradlew -Pdistro=hadoopYarn
./gradlew -Pdistro=hadoopYarn3

[jbaiera]

I understand the frustration with the speed that we've been moving toward supporting newer versions of Spark. There is a lot that needs to happen in order for the project to get there, but we've been making steady progress to that end. That said, I don't think this PR can be accepted in its current form for a number of reasons. First and foremost, this is targeting 6.5, which is an end of life version line. There will be no more releases for this version, so there's not much reason to make changes to the branch that will not see the light of day in a release. Secondly, I'm not sure how compatible this PR is with the number of changes we've already made to the build process in the main line. There may need to be a number of changes implemented before it can be accepted.

[avnerl]

I understand the frustration with the speed that we've been moving toward supporting newer versions of Spark. There is a lot that needs to happen in order for the project to get there, but we've been making steady progress to that end. That said, I don't think this PR can be accepted in its current form for a number of reasons. First and foremost, this is targeting 6.5, which is an end of life version line. There will be no more releases for this version, so there's not much reason to make changes to the branch that will not see the light of day in a release. Secondly, I'm not sure how compatible this PR is with the number of changes we've already made to the build process in the main line. There may need to be a number of changes implemented before it can be accepted.

that is unfortunate, I was just about opening a new spark 3.0.0 pull request into master branch but stopped everything since @mmigdiso alerted me to the fact that my efforts will go to waste since you started some code overhaul.

[jbaiera]

For what it's worth, the code overhaul is being developed in master directly, much of the work is done in there already, so there are probably only a few changes that would need to be made. I'm happy to help with figuring those things out as well. All help is appreciated!

In the mean time, I'm going to close this PR out. We can address any PR feedback on the new PR if you feel inclined to work on it still. If not, no worries.