Skip to content

Commit 3cfd5eb

Browse files
varant-zlaiezvz
andauthored
Fixing failing AWS integration tests (#654)
## Summary I think this is the fix for the failing CI test -- we now require this to be set to create airflow deps. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved error handling when accessing the partition column configuration, preventing unexpected errors if the configuration is missing. - **Tests** - Added a default partition column configuration for teams to enhance test coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: ezvz <[email protected]>
1 parent 3ae88a5 commit 3cfd5eb

File tree

3 files changed

+13
-8
lines changed

3 files changed

+13
-8
lines changed

api/python/ai/chronon/airflow_helpers.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,11 @@ def _get_airflow_deps_from_source(source, partition_column=None):
7373

7474

7575
def extract_default_partition_column(obj):
76-
return obj.metaData.executionInfo.conf.common.get("spark.chronon.partition.column")
77-
76+
try:
77+
return obj.metaData.executionInfo.conf.common.get("spark.chronon.partition.column")
78+
except Exception:
79+
# Error handling occurs in `create_airflow_dependency`
80+
return None
7881

7982
def _set_join_deps(join):
8083
default_partition_col = extract_default_partition_column(join)

api/python/test/canary/teams.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,11 @@
66
description="Default team",
77
email="ml-infra@<customer>.com", # TODO: Infra team email
88
outputNamespace="default",
9+
conf=ConfigProperties(
10+
common={
11+
"spark.chronon.partition.column": "ds",
12+
}
13+
),
914
env=EnvironmentVariables(
1015
common={
1116
"VERSION": "latest",

api/thrift/agent.thrift

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
namespace java ai.chronon.api
2+
include "common.thrift"
23

34
// TODO: Need to brainstorm and make necessary changes. just a starting point to unblock other work.
45
struct YarnAutoScalingSpec {
@@ -33,7 +34,7 @@ struct YarnJob {
3334
1: optional string appName
3435
2: optional YarnJobType jobType
3536

36-
10: optional list<string> args
37+
10: optional list<string> argsList
3738
11: optional map<string, string> env
3839
12: optional map<string, string> conf
3940
// creates local file with this name and contents - relative to cwd
@@ -121,13 +122,9 @@ struct JobInfo {
121122
10: optional YarnIncrementalJobStatus yarnIncrementalStatus
122123
}
123124

124-
struct DatePartitionRange {
125-
1: optional string start
126-
2: optional string end
127-
}
128125

129126
struct PartitionListingPutRequest {
130-
1: optional map<PartitionListingJob, list<DatePartitionRange>> partitions
127+
1: optional map<PartitionListingJob, list<common.DateRange>> partitions
131128
2: optional map<PartitionListingJob, string> errors
132129
}
133130

0 commit comments

Comments
 (0)