Turn off sbt delta lake tests in CI #903
Conversation
|
@pengyu-hou / @caiocamatta-stripe / @mickjermsurawong-openai - could either of you folks review? |
There was a problem hiding this comment.
Thanks. Once we have the bazel PR merged, we should have spark 3.5. Let's revisit this later.
| @@ -177,9 +147,6 @@ workflows: | |||
| - "Scala 13 -- Tests": | |||
| requires: | |||
| - "Pull Docker Image" | |||
| - "Scala 13 -- Delta Lake Format Tests": | |||
There was a problem hiding this comment.
On a second thought, is it possible to make it optional? So it won't be a blocker to merge PRs.
There was a problem hiding this comment.
let me check if circle ci supports this - if possible I agree, it would be nice
There was a problem hiding this comment.
@piyush-zlai any follow up here? Working on a similar PR for iceberg #993
There was a problem hiding this comment.
@abbywh - I haven't had the chance to revisit these. Based on my (limited) understanding of Spark the way we're creating sessions per test coupled with kicking off all the tests in parallel does often result in flakes as the sessions seem to conflict. The tests pass standalone in this repo (and we have them enabled for months now in our fork) but they seem to be flaking out when turned on with other tests triggered at the same time I think.
Summary
Turn off delta lake tests in CI - We are seeing these tests flake out fairly often and they result in holding up the rest of our dev loops. We are exercising testing of the table utils path already (the Hive flow). We can explore enabling these on Spark 3.5 and when we're running tests on Bazel there. Isolation and Spark 3.5 are the only differences with our internal fork's setup that come to my mind and these tests have not been flaky on our end.
Why / Goal
Test Plan
Checklist
Reviewers