Iceberg unit tests, support Iceberg + nonhive catalogs, Iceberg Kryo Serializer #993
Conversation
abbywh
changed the title
| @@ -10,6 +10,11 @@ | |||
| *.logs | |||
| *.iml | |||
| .idea/ | |||
| *.jvmopts | |||
| .bloop* | |||
| .metals* | |||
There was a problem hiding this comment.
how is working with metals relative to intellij? does the debugger work as well?
There was a problem hiding this comment.
It's really good actually. The debugged worked out of the box, I found it comparable to IntelliJ overall.
I'd recommend it to anyone who has remote dev boxes since VSCode's integration is far better in my experience. All the tests run a lot faster and I got in way more dev cycles. I probably would only recommend over IntelliJ with a dev box though.
| @@ -239,9 +239,15 @@ case object Iceberg extends Format { | |||
| override def partitions(tableName: String, partitionColumns: Seq[String])(implicit | |||
| sparkSession: SparkSession): Seq[Map[String, String]] = { | |||
| sparkSession.sqlContext | |||
| .sql(s"SHOW PARTITIONS $tableName") | |||
| .sql(s"SELECT partition FROM $tableName" ++ ".partitions") | |||
There was a problem hiding this comment.
ooc does this work for regular hive tables?
There was a problem hiding this comment.
This is for iceberg, Hive support is here
There was a problem hiding this comment.
it should work for hive tables, and the internal target I'm hitting are more or less "regular hive tables". Iceberg abstracts itself from the catalog implementation, so as long as your iceberg has an interface to your catalog implementation, it will work.
Summary
We want to unit test with Iceberg test via CI as well as improve the support in the Chronon OSS package.
Why / Goal
Follow Ups
Test Plan
Added circleCI check
Checklist
Reviewers