Skip to content

Turn off sbt delta lake tests in CI #903

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 27, 2025
Merged

Turn off sbt delta lake tests in CI #903

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
37accf6
Turn off sbt delta lake tests in CI
piyush-zlai Jan 16, 2025
912c20c
Remove from workflows
piyush-zlai Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 0 additions & 33 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,36 +78,6 @@ jobs:
destination: spark_warehouse.tar.gz
when: on_fail

# run these separately as we need a isolated JVM to not have Spark session settings interfere with other runs
# long term goal is to refactor the current testing spark session builder and avoid adding new single test to CI
"Scala 13 -- Delta Lake Format Tests":
executor: docker_baseimg_executor
steps:
- checkout
- run:
name: Run Scala 13 tests for Delta Lake format
environment:
format_test: deltalake
shell: /bin/bash -leuxo pipefail
command: |
conda activate chronon_py
# Increase if we see OOM.
export SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=4G -Xmx4G -Xms2G"
sbt '++ 2.13.6' "testOnly ai.chronon.spark.test.TableUtilsFormatTest"
- store_test_results:
path: /chronon/spark/target/test-reports
- store_test_results:
path: /chronon/aggregator/target/test-reports
- run:
name: Compress spark-warehouse
command: |
cd /tmp/ && tar -czvf spark-warehouse.tar.gz chronon/spark-warehouse
when: on_fail
- store_artifacts:
path: /tmp/spark-warehouse.tar.gz
destination: spark_warehouse.tar.gz
when: on_fail

"Scala 11 -- Compile":
executor: docker_baseimg_executor
steps:
Expand Down Expand Up @@ -177,9 +147,6 @@ workflows:
- "Scala 13 -- Tests":
requires:
- "Pull Docker Image"
- "Scala 13 -- Delta Lake Format Tests":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a second thought, is it possible to make it optional? So it won't be a blocker to merge PRs.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me check if circle ci supports this - if possible I agree, it would be nice

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@piyush-zlai any follow up here? Working on a similar PR for iceberg #993

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abbywh - I haven't had the chance to revisit these. Based on my (limited) understanding of Spark the way we're creating sessions per test coupled with kicking off all the tests in parallel does often result in flakes as the sessions seem to conflict. The tests pass standalone in this repo (and we have them enabled for months now in our fork) but they seem to be flaking out when turned on with other tests triggered at the same time I think.

requires:
- "Pull Docker Image"
- "Scalafmt Check":
requires:
- "Pull Docker Image"
Expand Down