Skip to content
This repository was archived by the owner on May 12, 2021. It is now read-only.

[PIO-210] Add support for Scala 2.12; Update dependencies #518

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from
Open

[PIO-210] Add support for Scala 2.12; Update dependencies #518

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5d5f38f
[PIO-210] Add support for Scala 2.12; Update dependencies
EmergentOrder Jun 25, 2019
0496da7
cleanup LICENSE.txt
EmergentOrder Jun 25, 2019
618a36a
Drop specs2 version to fix build
EmergentOrder Jun 25, 2019
9368fe5
Add Scala 2.12 tests to travis
EmergentOrder Jun 25, 2019
54cae9c
Attempt to fix travis build
EmergentOrder Jun 25, 2019
b6262a1
Attempt to fix travis build
EmergentOrder Jun 25, 2019
ec817b0
Don't use 2.11 version of elasticsearch-spark in 2.12
EmergentOrder Jun 26, 2019
4289256
Revert assembly plugin
EmergentOrder Jun 26, 2019
d8c0796
Be explicit about Scala version
EmergentOrder Jun 27, 2019
2ea73d4
Fix pio-core dep in test engine
EmergentOrder Jun 28, 2019
0748158
Merge branch 'develop' into develop
EmergentOrder Nov 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ env:
matrix:
- BUILD_TYPE=Unit
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=PGSQL
- BUILD_TYPE=Unit
PIO_SCALA_VERSION=2.12.8
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=PGSQL

- BUILD_TYPE=Integration
METADATA_REP=ELASTICSEARCH EVENTDATA_REP=ELASTICSEARCH MODELDATA_REP=S3
PIO_ELASTICSEARCH_VERSION=5.6.9
Expand All @@ -57,6 +61,12 @@ env:
METADATA_REP=ELASTICSEARCH EVENTDATA_REP=HBASE MODELDATA_REP=LOCALFS
PIO_HBASE_VERSION=1.2.6

- BUILD_TYPE=Integration
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=PGSQL
PIO_SCALA_VERSION=2.12.8
PIO_SPARK_VERSION=2.4.3
PIO_HADOOP_VERSION=2.6.5

- BUILD_TYPE=Integration
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=PGSQL
PIO_SCALA_VERSION=2.11.12
Expand All @@ -78,6 +88,12 @@ env:
PIO_SPARK_VERSION=2.3.3
PIO_HADOOP_VERSION=2.6.5

- BUILD_TYPE=Integration
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=PGSQL
PIO_SCALA_VERSION=2.12.8
PIO_SPARK_VERSION=2.4.3
PIO_HADOOP_VERSION=2.7.7

- BUILD_TYPE=Integration
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=PGSQL
PIO_SCALA_VERSION=2.11.12
Expand Down
13 changes: 4 additions & 9 deletions LICENSE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1433,11 +1433,9 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
Binary distribution bundles

com.esotericsoftware # kryo # 3.0.3 (https://github.com/EsotericSoftware/kryo)
com.esotericsoftware # kryo-shaded # 4.0.2 (https://github.com/EsotericSoftware/kryo)
com.esotericsoftware # minlog # 1.3.0 (https://github.com/EsotericSoftware/minlog)
com.esotericsoftware # reflectasm # 1.10.1 (https://github.com/EsotericSoftware/reflectasm)
com.esotericsoftware.kryo # kryo # 2.21 (https://github.com/EsotericSoftware/kryo)
com.esotericsoftware.minlog # minlog # 1.2 (https://github.com/EsotericSoftware/minlog)
com.esotericsoftware.reflectasm # reflectasm # 1.07 (https://github.com/EsotericSoftware/reflectasm)

which is available under the BSD license (http://www.opensource.org/licenses/bsd-license.php)

Expand Down Expand Up @@ -1529,8 +1527,8 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
Binary distribution bundles

com.thoughtworks.paranamer # paranamer # 2.3 (https://github.com/paul-hammant/paranamer)
com.thoughtworks.paranamer # paranamer # 2.6 (https://github.com/paul-hammant/paranamer)
com.thoughtworks.paranamer # paranamer # 2.3 (https://github.com/paul-hammant/paranamer)
com.thoughtworks.paranamer # paranamer # 2.8 (https://github.com/paul-hammant/paranamer)

which is available under the BSD license (http://www.opensource.org/licenses/bsd-license.php)

Expand Down Expand Up @@ -1632,7 +1630,7 @@ THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
Binary distribution bundles

org.clapper # grizzled-slf4j_2.11 # 1.0.2 (http://software.clapper.org/grizzled-slf4j/)
org.clapper # grizzled-slf4j_2.11 # 1.3.4 (http://software.clapper.org/grizzled-slf4j/)

which is available under the BSD license (http://www.opensource.org/licenses/bsd-license.php)

Expand Down Expand Up @@ -1701,13 +1699,10 @@ EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Binary distribution bundles

org.scala-lang # scala-library # 2.11.12 (http://scala-lang.org/)
org.scala-lang # scala-compiler # 2.11.12 (http://scala-lang.org/)
org.scala-lang # scala-reflect # 2.11.12 (http://scala-lang.org/)
org.scala-lang # scalap # 2.11.12 (http://scala-lang.org/)
org.scala-lang.modules # scala-java8-compat_2.11 # 0.7.0 (http://scala-lang.org/)
org.scala-lang.modules # scala-parser-combinators_2.11 # 1.0.6 (http://scala-lang.org/)
org.scala-lang.modules # scala-parser-combinators_2.11 # 1.1.0 (http://scala-lang.org/)
org.scala-lang.modules # scala-xml_2.11 # 1.0.5 (http://scala-lang.org/)
org.scala-lang.modules # scala-xml_2.11 # 1.0.6 (http://scala-lang.org/)

which is available under the BSD license (http://www.scala-lang.org/downloads/license.html)
Expand Down
4 changes: 2 additions & 2 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ scalaVersion in ThisBuild := sys.props.getOrElse("scala.version", "2.11.12")

scalaBinaryVersion in ThisBuild := binaryVersion(scalaVersion.value)

crossScalaVersions in ThisBuild := Seq("2.11.12")
crossScalaVersions in ThisBuild := Seq("2.11.12", "2.12.8")

scalacOptions in ThisBuild ++= Seq("-deprecation", "-unchecked", "-feature")

Expand All @@ -37,7 +37,7 @@ javacOptions in (ThisBuild, compile) ++= Seq("-source", "1.8", "-target", "1.8",
"-Xlint:deprecation", "-Xlint:unchecked")

// Ignore differentiation of Spark patch levels
sparkVersion in ThisBuild := sys.props.getOrElse("spark.version", "2.1.3")
sparkVersion in ThisBuild := sys.props.getOrElse("spark.version", "2.4.3")

sparkBinaryVersion in ThisBuild := binaryVersion(sparkVersion.value)

Expand Down
8 changes: 4 additions & 4 deletions core/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ name := "apache-predictionio-core"
libraryDependencies ++= Seq(
"com.github.scopt" %% "scopt" % "3.5.0",
"com.google.code.gson" % "gson" % "2.5",
"com.twitter" %% "chill-bijection" % "0.7.2",
"com.twitter" %% "chill-bijection" % "0.9.3",
"de.javakaffee" % "kryo-serializers" % "0.37",
"net.jodah" % "typetools" % "0.3.1",
"org.apache.spark" %% "spark-core" % sparkVersion.value % "provided",
"org.json4s" %% "json4s-ext" % json4sVersion.value,
"org.scalaj" %% "scalaj-http" % "1.1.6",
"org.scalaj" %% "scalaj-http" % "2.4.2",
"org.slf4j" % "slf4j-log4j12" % "1.7.18",
"org.scalatest" %% "scalatest" % "2.1.7" % "test",
"org.specs2" %% "specs2" % "2.3.13" % "test",
"org.scalatest" %% "scalatest" % "3.0.5" % "test",
"org.specs2" %% "specs2-core" % "4.5.1" % "test",
"org.scalamock" %% "scalamock-scalatest-support" % "3.5.0" % "test",
"com.h2database" % "h2" % "1.4.196" % "test"
)
Expand Down
10 changes: 5 additions & 5 deletions data/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,15 @@ name := "apache-predictionio-data"

libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"com.github.nscala-time" %% "nscala-time" % "2.6.0",
"com.github.nscala-time" %% "nscala-time" % "2.22.0",
"com.google.guava" % "guava" % "14.0.1",
"com.typesafe.akka" %% "akka-http-testkit" % "10.1.5" % "test",
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
"org.clapper" %% "grizzled-slf4j" % "1.0.2",
"org.scalatest" %% "scalatest" % "2.1.7" % "test",
"org.specs2" %% "specs2" % "3.3.1" % "test"
"org.clapper" %% "grizzled-slf4j" % "1.3.4",
"org.scalatest" %% "scalatest" % "3.0.5" % "test",
"org.specs2" %% "specs2-core" % "4.5.1" % "test"
exclude("org.scalaz.stream", s"scalaz-stream_${scalaBinaryVersion.value}"),
"org.scalamock" %% "scalamock-specs2-support" % "3.5.0" % "test",
"org.scalamock" %% "scalamock-specs2-support" % "3.6.0" % "test",
"com.h2database" % "h2" % "1.4.196" % "test")

parallelExecution in Test := false
Expand Down
2 changes: 1 addition & 1 deletion e2/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ parallelExecution in Test := false

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-mllib" % sparkVersion.value % "provided",
"org.scalatest" %% "scalatest" % "2.2.5" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

pomExtra := childrenPomExtra.value
2 changes: 1 addition & 1 deletion project/assembly.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.7")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")
2 changes: 1 addition & 1 deletion project/build.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sbt.version=1.2.3
sbt.version=1.2.8
2 changes: 1 addition & 1 deletion storage/elasticsearch/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ libraryDependencies ++= Seq(
"org.elasticsearch" %% "elasticsearch-spark-20" % elasticsearchVersion.value
exclude("org.apache.spark", "*"),
"org.elasticsearch" % "elasticsearch-hadoop-mr" % elasticsearchVersion.value,
"org.specs2" %% "specs2" % "2.3.13" % "test")
"org.specs2" %% "specs2-core" % "2.4.17" % "test")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why we choose not the latest version (4.5.1) but 2.4.17?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests failed to compile under the latest version.


parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/hbase/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ libraryDependencies ++= Seq(
exclude("org.mortbay.jetty", "servlet-api-2.5")
exclude("org.mortbay.jetty", "jsp-api-2.1")
exclude("org.mortbay.jetty", "jsp-2.1"),
"org.specs2" %% "specs2" % "2.3.13" % "test")
"org.specs2" %% "specs2-core" % "2.4.17" % "test")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above


parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/hdfs/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ libraryDependencies ++= Seq(
exclude("commons-beanutils", "*"),
"org.apache.hadoop" % "hadoop-hdfs" % hadoopVersion.value,
"org.apache.predictionio" %% "apache-predictionio-data" % version.value % "provided",
"org.scalatest" %% "scalatest" % "2.1.7" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/jdbc/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
"org.scalikejdbc" %% "scalikejdbc" % "3.1.0",
"org.postgresql" % "postgresql" % "9.4-1204-jdbc41" % "test",
"org.specs2" %% "specs2" % "2.3.13" % "test")
"org.specs2" %% "specs2-core" % "2.4.17" % "test")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above


parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/localfs/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ name := "apache-predictionio-data-localfs"

libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % version.value % "provided",
"org.scalatest" %% "scalatest" % "2.1.7" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/s3/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % version.value % "provided",
"com.google.guava" % "guava" % "14.0.1" % "provided",
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.132",
"org.scalatest" %% "scalatest" % "2.1.7" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion tests/pio_tests/engines/recommendation-engine/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,6 @@ name := "template-scala-parallel-recommendation"
organization := "org.apache.predictionio"

libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % "0.14.0" % "provided",
"org.apache.predictionio" % "apache-predictionio-core" % "0.15.0-SNAPSHOT" % "provided",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use the current (local) SNAPSHOT here, otherwise things like 2.12 support will never be able to pass in Travis before being merged.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shimamoto Do you have any idea why this fails to resolve? It should have been published locally inside docker on make-distribution, no? We need to be testing the current snapshot here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@EmergentOrder You need to write the double %% after the groupID.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah of course. I removed that and then missed putting it back. Thanks.

"org.apache.spark" %% "spark-core" % sys.env.getOrElse("PIO_SPARK_VERSION", "2.1.1") % "provided",
"org.apache.spark" %% "spark-mllib" % sys.env.getOrElse("PIO_SPARK_VERSION", "2.1.1") % "provided")
2 changes: 1 addition & 1 deletion tools/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion.value,
"com.typesafe.akka" %% "akka-http-testkit" % "10.1.5" % "test",
"org.specs2" %% "specs2-core" % "4.2.0" % "test")
"org.specs2" %% "specs2-core" % "4.5.1" % "test")

assemblyMergeStrategy in assembly := {
case PathList("META-INF", "LICENSE.txt") => MergeStrategy.concat
Expand Down