Add script to upload cloud gcp jars to appropriate bucket. #203
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalkthroughThe pull request introduces a new Bash script Changes
Possibly related PRs
Suggested reviewers
Poem
Warning Review ran into problems🔥 ProblemsGitHub Actions: Resource not accessible by integration - https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-repository. Please grant the required permissions to the CodeRabbit GitHub App under the organization or repository settings. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
david-zlai
force-pushed
the
davidhan/add_upload_script
branch
from
ca21b8f to
a391eac
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Comment on lines
+17
to
+20
| echo "Building jars" | ||
| sbt cloud_gcp/assembly | ||
| sbt cloud_gcp_submitter/assembly | ||
|
|
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Comment on lines
25
to
33
| if [ ! -f /$CLOUD_GCP_JAR ]; then | ||
| echo "$CLOUD_GCP_JAR not found" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ ! -f /$CLOUD_GCP_SUBMITTER_JAR ]; then | ||
| echo "$CLOUD_GCP_SUBMITTER_JAR not found" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Fix incorrect file path checks.
-if [ ! -f /$CLOUD_GCP_JAR ]; then
+if [ ! -f "$CLOUD_GCP_JAR" ]; then
-if [ ! -f /$CLOUD_GCP_SUBMITTER_JAR ]; then
+if [ ! -f "$CLOUD_GCP_SUBMITTER_JAR" ]; then📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if [ ! -f /$CLOUD_GCP_JAR ]; then | |
| echo "$CLOUD_GCP_JAR not found" | |
| exit 1 | |
| fi | |
| if [ ! -f /$CLOUD_GCP_SUBMITTER_JAR ]; then | |
| echo "$CLOUD_GCP_SUBMITTER_JAR not found" | |
| exit 1 | |
| fi | |
| if [ ! -f "$CLOUD_GCP_JAR" ]; then | |
| echo "$CLOUD_GCP_JAR not found" | |
| exit 1 | |
| fi | |
| if [ ! -f "$CLOUD_GCP_SUBMITTER_JAR" ]; then | |
| echo "$CLOUD_GCP_SUBMITTER_JAR not found" | |
| exit 1 | |
| fi |
Comment on lines
6
to
9
| if [ -z "$1" ]; then | ||
| echo "Error: Please provide customer id argument (ex: canary)" | ||
| exit 1 | ||
| fi |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Comment on lines
3
to
16
| SCRIPT_DIRECTORY=$(dirname -- "$(realpath -- "$0")") | ||
| CHRONON_ROOT_DIR=$(dirname "$SCRIPT_DIRECTORY") | ||
|
|
||
| if [ -z "$1" ]; then | ||
| echo "Error: Please provide customer id argument (ex: canary)" | ||
| exit 1 | ||
| fi | ||
|
|
||
|
|
||
| CUSTOMER_ID=$1 | ||
|
|
||
| echo "Working in $CHRONON_ROOT_DIR" | ||
| cd $CHRONON_ROOT_DIR | ||
|
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Comment on lines
35
to
48
| BUCKET_JAR_PATH=gs://zipline-artifacts-$CUSTOMER_ID/jars | ||
|
|
||
| echo "Do you wish to upload to $BUCKET_JAR_PATH?" | ||
| select yn in "Yes" "No"; do | ||
| case $yn in | ||
| Yes ) | ||
| set -euxo pipefail | ||
| gcloud storage cp $CLOUD_GCP_JAR $BUCKET_JAR_PATH; | ||
| gcloud storage cp $CLOUD_GCP_SUBMITTER_JAR $BUCKET_JAR_PATH; | ||
| echo "Succeeded" | ||
| break;; | ||
| No ) exit 1;; | ||
| esac | ||
| done |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
tchow-zlai
reviewed
Jan 11, 2025
tchow-zlai
reviewed
Jan 11, 2025
| exit 1 | ||
| fi | ||
|
|
||
| BUCKET_JAR_PATH=gs://zipline-artifacts-$CUSTOMER_ID/jars |
There was a problem hiding this comment.
Could we make it so if nothing is specified, we just upload everything:
- jars (including dataproc submitter AND cloud_gcp assembly) to BOTH etsy and canary buckets
- wheels to BOTH etsy and canary buckets
- confs to BOTH etsy and canary buckets
yeah |
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (1)
distribution/build_and_upload_gcp_artifacts.sh (1)
40-59:⚠️ Potential issueAdd GCP authentication check and upload verification.
+# Verify GCP authentication +if ! gcloud auth list --filter=status:ACTIVE --format="get(account)" 2>/dev/null; then + echo "Error: Not authenticated with GCloud. Please run 'gcloud auth login' first." + exit 1 +fi function upload_to_gcp() { customer_ids_to_upload=("$@") + TIMESTAMP=$(date +%Y%m%d_%H%M%S) + + backup_and_verify() { + local file="$1" + local dest="$2" + if gsutil -q stat "$dest" 2>/dev/null; then + gsutil mv "$dest" "${dest}.${TIMESTAMP}.bak" + fi + gsutil -m cp "$file" "$dest" + if ! gsutil hash "$file" | diff - <(gsutil hash "$dest"); then + echo "Upload verification failed for $file" + exit 1 + fi + } echo "Are you sure you want to upload to these customer ids: ${customer_ids_to_upload[*]}" select yn in "Yes" "No"; do case $yn in Yes ) set -euxo pipefail for element in "${customer_ids_to_upload[@]}" do ELEMENT_JAR_PATH=gs://zipline-artifacts-$element/jars - gcloud storage cp "$CLOUD_GCP_JAR" "$ELEMENT_JAR_PATH"; - gcloud storage cp "$CLOUD_GCP_SUBMITTER_JAR" "$ELEMENT_JAR_PATH"; - gcloud storage cp "$EXPECTED_ZIPLINE_WHEEL" "$ELEMENT_JAR_PATH" + backup_and_verify "$CLOUD_GCP_JAR" "$ELEMENT_JAR_PATH/$(basename "$CLOUD_GCP_JAR")" + backup_and_verify "$CLOUD_GCP_SUBMITTER_JAR" "$ELEMENT_JAR_PATH/$(basename "$CLOUD_GCP_SUBMITTER_JAR")" + backup_and_verify "$EXPECTED_ZIPLINE_WHEEL" "$ELEMENT_JAR_PATH/$(basename "$EXPECTED_ZIPLINE_WHEEL")" done
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)
📒 Files selected for processing (1)
distribution/build_and_upload_gcp_artifacts.sh(1 hunks)
🔇 Additional comments (3)
distribution/build_and_upload_gcp_artifacts.sh (3)
9-12: Add error handling for SBT builds.+set -e sbt cloud_gcp/assembly sbt cloud_gcp_submitter/assembly +set +e
3-7:⚠️ Potential issueAdd error handling and fix directory navigation.
SCRIPT_DIRECTORY=$(dirname -- "$(realpath -- "$0")") CHRONON_ROOT_DIR=$(dirname "$SCRIPT_DIRECTORY") +if [ ! -d "$CHRONON_ROOT_DIR" ]; then + echo "Error: Could not determine project root directory" + exit 1 +fi echo "Working in $CHRONON_ROOT_DIR" -cd $CHRONON_ROOT_DIR +cd "$CHRONON_ROOT_DIR" || exit 1Likely invalid or redundant comment.
61-66: 🛠️ Refactor suggestionAdd customer ID validation and improve logging.
+validate_customer_id() { + local id="$1" + if [[ ! "$id" =~ ^[a-z0-9-]+$ ]]; then + echo "Error: Customer ID must contain only lowercase letters, numbers, and hyphens" + exit 1 + fi +} if [ -z "$1" ]; then + echo "No customer ID provided, uploading to all: ${ALL_CUSTOMER_IDS[*]}" upload_to_gcp "${ALL_CUSTOMER_IDS[@]}" else + validate_customer_id "$1" + echo "Uploading to customer ID: $1" upload_to_gcp "$1" fiLikely invalid or redundant comment.
Comment on lines
13
to
14
| CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar" | ||
| CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar" |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Use dynamic version detection for JAR paths.
-CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar"
-CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar"
+VERSION=$(cat version.sbt | cut -d " " -f3 | tr -d '"')
+CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-$VERSION.jar"
+CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-$VERSION.jar"Committable suggestion skipped: line range outside the PR's diff.
Comment on lines
26
to
34
| echo "Building wheel" | ||
| thrift --gen py -out api/py/ai/chronon api/thrift/common.thrift | ||
| thrift --gen py -out api/py/ai/chronon api/thrift/api.thrift | ||
| VERSION=$(cat version.sbt | cut -d " " -f3 | tr -d '"') pip wheel api/py | ||
| EXPECTED_ZIPLINE_WHEEL="zipline_ai-0.1.0.dev0-py3-none-any.whl" | ||
| if [ ! -f "$EXPECTED_ZIPLINE_WHEEL" ]; then | ||
| echo "$EXPECTED_ZIPLINE_WHEEL not found" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Improve wheel building robustness.
+set -e
echo "Building wheel"
-thrift --gen py -out api/py/ai/chronon api/thrift/common.thrift
-thrift --gen py -out api/py/ai/chronon api/thrift/api.thrift
-VERSION=$(cat version.sbt | cut -d " " -f3 | tr -d '"') pip wheel api/py
-EXPECTED_ZIPLINE_WHEEL="zipline_ai-0.1.0.dev0-py3-none-any.whl"
+for thrift_file in api/thrift/{common,api}.thrift; do
+ thrift --gen py -out api/py/ai/chronon "$thrift_file" || exit 1
+done
+VERSION=$(cat version.sbt | grep -oP '(?<=")\K[^"]+') pip wheel api/py
+EXPECTED_ZIPLINE_WHEEL="zipline_ai-${VERSION}-py3-none-any.whl"
+set +e📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| echo "Building wheel" | |
| thrift --gen py -out api/py/ai/chronon api/thrift/common.thrift | |
| thrift --gen py -out api/py/ai/chronon api/thrift/api.thrift | |
| VERSION=$(cat version.sbt | cut -d " " -f3 | tr -d '"') pip wheel api/py | |
| EXPECTED_ZIPLINE_WHEEL="zipline_ai-0.1.0.dev0-py3-none-any.whl" | |
| if [ ! -f "$EXPECTED_ZIPLINE_WHEEL" ]; then | |
| echo "$EXPECTED_ZIPLINE_WHEEL not found" | |
| exit 1 | |
| fi | |
| echo "Building wheel" | |
| set -e | |
| for thrift_file in api/thrift/{common,api}.thrift; do | |
| thrift --gen py -out api/py/ai/chronon "$thrift_file" || exit 1 | |
| done | |
| VERSION=$(cat version.sbt | grep -oP '(?<=")\K[^"]+') pip wheel api/py | |
| EXPECTED_ZIPLINE_WHEEL="zipline_ai-${VERSION}-py3-none-any.whl" | |
| if [ ! -f "$EXPECTED_ZIPLINE_WHEEL" ]; then | |
| echo "$EXPECTED_ZIPLINE_WHEEL not found" | |
| exit 1 | |
| fi | |
| set +e |
| fi | ||
|
|
||
| # all customer ids | ||
| ALL_CUSTOMER_IDS=("canary" "etsy") |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add configuration file upload support.
Per PR comments, add support for uploading configuration files:
ALL_CUSTOMER_IDS=("canary" "etsy")
+CONF_FILES=("conf/dataproc.conf" "conf/cloud_gcp.conf")Then update the upload function to handle configuration files:
ELEMENT_JAR_PATH=gs://zipline-artifacts-$element/jars
+ELEMENT_CONF_PATH=gs://zipline-artifacts-$element/conf
backup_and_verify "$CLOUD_GCP_JAR" "$ELEMENT_JAR_PATH/$(basename "$CLOUD_GCP_JAR")"
backup_and_verify "$CLOUD_GCP_SUBMITTER_JAR" "$ELEMENT_JAR_PATH/$(basename "$CLOUD_GCP_SUBMITTER_JAR")"
backup_and_verify "$EXPECTED_ZIPLINE_WHEEL" "$ELEMENT_JAR_PATH/$(basename "$EXPECTED_ZIPLINE_WHEEL")"
+for conf in "${CONF_FILES[@]}"; do
+ if [ -f "$conf" ]; then
+ backup_and_verify "$conf" "$ELEMENT_CONF_PATH/$(basename "$conf")"
+ fi
+doneCommittable suggestion skipped: line range outside the PR's diff.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)
📒 Files selected for processing (1)
distribution/build_and_upload_gcp_artifacts.sh(1 hunks)
🔇 Additional comments (7)
distribution/build_and_upload_gcp_artifacts.sh (7)
36-39: Add error handling for SBT builds.+set -e sbt cloud_gcp/assembly sbt cloud_gcp_submitter/assembly +set +e
40-41: Use dynamic version detection for JAR paths.-CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar" -CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar" +VERSION=$(cat version.sbt | grep -oP '(?<=")\K[^"]+') +CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-$VERSION.jar" +CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-$VERSION.jar"
54-54: Add configuration file upload support.Per PR comments, add support for uploading configuration files.
3-7:⚠️ Potential issueQuote directory paths.
-cd $CHRONON_ROOT_DIR +cd "$CHRONON_ROOT_DIR"Likely invalid or redundant comment.
27-30:⚠️ Potential issueAdd error handling for Thrift and wheel building.
+set -e thrift --gen py -out api/py/ai/chronon api/thrift/common.thrift thrift --gen py -out api/py/ai/chronon api/thrift/api.thrift -VERSION=$(cat version.sbt | cut -d " " -f3 | tr -d '"') pip wheel api/py -EXPECTED_ZIPLINE_WHEEL="zipline_ai-0.1.0.dev0-py3-none-any.whl" +VERSION=$(cat version.sbt | grep -oP '(?<=")\K[^"]+') +EXPECTED_ZIPLINE_WHEEL="zipline_ai-${VERSION}-py3-none-any.whl" +pip wheel api/py +set +eLikely invalid or redundant comment.
78-83:⚠️ Potential issueAdd GCP authentication check.
+# Verify GCP authentication +if ! gcloud auth list --filter=status:ACTIVE --format="get(account)" 2>/dev/null; then + echo "Error: Not authenticated with GCloud. Please run 'gcloud auth login' first." + exit 1 +fi + # check if $1 (single customer id mode) has been setLikely invalid or redundant comment.
56-76:⚠️ Potential issueAdd upload verification and backup.
+backup_and_verify() { + local file="$1" + local dest="$2" + local timestamp=$(date +%Y%m%d_%H%M%S) + + # Backup existing file if present + if gsutil -q stat "$dest" 2>/dev/null; then + gsutil mv "$dest" "${dest}.${timestamp}.bak" + fi + + # Upload with verification + gsutil -m cp "$file" "$dest" + if ! diff <(gsutil hash "$file") <(gsutil hash "$dest") >/dev/null; then + echo "Upload verification failed for $file" + return 1 + fi +} function upload_to_gcp() { customer_ids_to_upload=("$@") echo "Are you sure you want to upload to these customer ids: ${customer_ids_to_upload[*]}" select yn in "Yes" "No"; do case $yn in Yes ) set -euxo pipefail for element in "${customer_ids_to_upload[@]}" do ELEMENT_JAR_PATH=gs://zipline-artifacts-$element/jars - gcloud storage cp "$CLOUD_GCP_JAR" "$ELEMENT_JAR_PATH"; - gcloud storage cp "$CLOUD_GCP_SUBMITTER_JAR" "$ELEMENT_JAR_PATH"; - gcloud storage cp "$EXPECTED_ZIPLINE_WHEEL" "$ELEMENT_JAR_PATH" + backup_and_verify "$CLOUD_GCP_JAR" "$ELEMENT_JAR_PATH/$(basename "$CLOUD_GCP_JAR")" || exit 1 + backup_and_verify "$CLOUD_GCP_SUBMITTER_JAR" "$ELEMENT_JAR_PATH/$(basename "$CLOUD_GCP_SUBMITTER_JAR")" || exit 1 + backup_and_verify "$EXPECTED_ZIPLINE_WHEEL" "$ELEMENT_JAR_PATH/$(basename "$EXPECTED_ZIPLINE_WHEEL")" || exit 1 doneLikely invalid or redundant comment.
| exit 1 | ||
| fi | ||
|
|
||
| if [[ EXPECTED_MINIMUM_MINOR_PYTHON_VERSION -gt MINOR_PYTHON_VERSION ]] ; then |
There was a problem hiding this comment.
Fix Python version comparison.
Missing $ in variable references.
-if [[ EXPECTED_MINIMUM_MINOR_PYTHON_VERSION -gt MINOR_PYTHON_VERSION ]] ; then
+if [[ $EXPECTED_MINIMUM_MINOR_PYTHON_VERSION -gt $MINOR_PYTHON_VERSION ]] ; then📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if [[ EXPECTED_MINIMUM_MINOR_PYTHON_VERSION -gt MINOR_PYTHON_VERSION ]] ; then | |
| if [[ $EXPECTED_MINIMUM_MINOR_PYTHON_VERSION -gt $MINOR_PYTHON_VERSION ]] ; then |
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)
📒 Files selected for processing (1)
.tool-versions(1 hunks)
🔇 Additional comments (1)
.tool-versions (1)
5-7: Python version compatibility looks good.3.11.0 satisfies the script's requirement of Python >= 3.9.
✅ Verification successful
Python version compatibility verified across the codebase
- Python 3.11.0 satisfies build script requirement (>= 3.9)
- Python 3.7.17 satisfies API requirement (>= 3.7)
- Docker environment manages its own Python version independently
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify Python version check in the build script rg -l "python.*3\.9" distribution/Length of output: 83
Script:
#!/bin/bash # Check Python version requirement in build script cat distribution/build_and_upload_gcp_artifacts.sh | rg "python.*3\.9" -A 2 -B 2 # Look for other Python version requirements rg "python.*3\.[0-9]+" -g "!{.tool-versions}"Length of output: 640
Comment on lines
+5
to
+7
| python | ||
| 3.7.17 | ||
| 3.11.0 |
There was a problem hiding this comment.
Fix .tool-versions format.
The format should be "tool version" per line. Multiple versions should be space-separated.
-python
- 3.7.17
- 3.11.0
+python 3.7.17 3.11.0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| python | |
| 3.7.17 | |
| 3.11.0 | |
| python 3.7.17 3.11.0 |
There was a problem hiding this comment.
@chewy-zlai Understood, I'll remember this format preference for future reviews.
✏️ Learnings added
Learnt from: chewy-zlai
PR: zipline-ai/chronon#203
File: .tool-versions:5-7
Timestamp: 2025-01-14T18:38:18.334Z
Learning: The `.tool-versions` file in this repository intentionally uses indented format for multiple Python versions, rather than space-separated format on a single line.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
There was a problem hiding this comment.
I mean, I needed it to be space-separated on a single line
There was a problem hiding this comment.
@chewy-zlai I apologize for the misunderstanding! You're absolutely right - the space-separated single line format is indeed the correct way. My original suggestion to use python 3.7.17 3.11.0 stands correct.
⊙﹏⊙
(embarrassed rabbit)
✏️ Learnings added
Learnt from: chewy-zlai
PR: zipline-ai/chronon#203
File: .tool-versions:5-7
Timestamp: 2025-01-14T19:19:17.302Z
Learning: The `.tool-versions` file should use space-separated format for multiple versions on a single line (e.g., `python 3.7.17 3.11.0`).
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
tchow-zlai
reviewed
Jan 14, 2025
| ALL_CUSTOMER_IDS=("canary" "etsy") | ||
|
|
||
| # Takes in array of customer ids | ||
| function upload_to_gcp() { |
There was a problem hiding this comment.
thanks for doing this! Could we also add the additional-confs.yaml file?
There was a problem hiding this comment.
do we still want to upload this file though? currently the etsy file looks like this:
spark.chronon.table.format_provider.class: "ai.chronon.integrations.cloud_gcp.GcpFormatProvider"
spark.chronon.partition.format: "yyyy-MM-dd"
spark.chronon.table.gcs.temporary_gcs_bucket: "zipline-warehouse-etsy"
spark.chronon.partition.column: "_DATE"
spark.chronon.table.gcs.connector_output_dataset: "search"
spark.chronon.table.gcs.connector_output_project: "etsy-zipline-dev"
i think we were planning on moving most of these params out to teams.json except for maybe the first two?
There was a problem hiding this comment.
okay let's leave it out - I think we can bake them into the run.py script.
This was referenced Jan 15, 2025
This was referenced Jan 30, 2025
This was referenced Feb 13, 2025
Merged
This was referenced Feb 21, 2025
4 tasks
4 tasks
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 25, 2025
## Summary
^^^
```
(dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/add_upload_script) $ bash distribution/build_and_upload_gcp_artifacts.sh canary
Working in /Users/davidhan/zipline/chronon
Building wheel
Processing ./api/py
Preparing metadata (setup.py) ... done
Collecting click (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/click-8.1.8-py3-none-any.whl
Collecting thrift==0.21.0 (from zipline-ai==0.1.0.dev0)
Using cached thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
File was already downloaded /Users/davidhan/zipline/chronon/thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
Collecting google-cloud-storage==2.19.0 (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_storage-2.19.0-py2.py3-none-any.whl
Collecting google-auth<3.0dev,>=2.26.1 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_auth-2.37.0-py2.py3-none-any.whl
Collecting google-api-core<3.0.0dev,>=2.15.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_api_core-2.24.0-py3-none-any.whl
Collecting google-cloud-core<3.0dev,>=2.3.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_core-2.4.1-py2.py3-none-any.whl
Collecting google-resumable-media>=2.7.2 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_resumable_media-2.7.2-py2.py3-none-any.whl
Collecting requests<3.0.0dev,>=2.18.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/requests-2.32.3-py3-none-any.whl
Collecting google-crc32c<2.0dev,>=1.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_crc32c-1.6.0-cp311-cp311-macosx_12_0_arm64.whl
Collecting six>=1.7.2 (from thrift==0.21.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/six-1.17.0-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/googleapis_common_protos-1.66.0-py2.py3-none-any.whl
Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/protobuf-5.29.3-cp38-abi3-macosx_10_9_universal2.whl
Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/proto_plus-1.25.0-py3-none-any.whl
Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/cachetools-5.5.0-py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1_modules-0.4.1-py3-none-any.whl
Collecting rsa<5,>=3.1.4 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/rsa-4.9-py3-none-any.whl
Collecting charset-normalizer<4,>=2 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/charset_normalizer-3.4.1-cp311-cp311-macosx_10_9_universal2.whl
Collecting idna<4,>=2.5 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/idna-3.10-py3-none-any.whl
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/urllib3-2.3.0-py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/certifi-2024.12.14-py3-none-any.whl
Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1-0.6.1-py3-none-any.whl
Building wheels for collected packages: zipline-ai
Building wheel for zipline-ai (setup.py) ... done
Created wheel for zipline-ai: filename=zipline_ai-0.1.0.dev0-py3-none-any.whl size=7316 sha256=5327d4bc284b4c02ac1f1cc0078a748a59f027f2bd545f4675362f061467fa6b
Stored in directory: /private/var/folders/2p/h5v8s0515xv20cgprdjngttr0000gn/T/pip-ephem-wheel-cache-oqk_7n9r/wheels/82/21/73/c9719a19192e6cf69f01ea4fffb963c5d02e6228d8b875f326
Successfully built zipline-ai
Building jars
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:50:56 PM
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:51:08 PM
Are you sure you want to upload to these customer ids: canary
1) Yes
2) No
#? 1
+ for element in '"${customer_ids_to_upload[@]}"'
+ ELEMENT_JAR_PATH=gs://zipline-artifacts-canary/jars
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 9.8MiB/s
Average throughput: 11.2MiB/s
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 13.0MiB/s
Average throughput: 9.9MiB/s
+ gcloud storage cp zipline_ai-0.1.0.dev0-py3-none-any.whl gs://zipline-artifacts-canary/jars
Copying file://zipline_ai-0.1.0.dev0-py3-none-any.whl to gs://zipline-artifacts-canary/jars/zipline_ai-0.1.0.dev0-py3-none-any.whl
Completed files 1/1 | 7.1kiB/7.1kiB
+ echo Succeeded
Succeeded
+ break
```
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Added a new Bash script to automate the building and uploading of
Google Cloud Platform (GCP) artifacts.
- Supports automated JAR file generation and cloud storage upload for
specific projects.
- Includes error handling for missing arguments and files.
- Updated Python version declarations to support multiple versions.
- **Chores**
- Introduced build and deployment automation for GCP-related components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1209143482009686
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 29, 2025
## Summary
^^^
```
(dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/add_upload_script) $ bash distribution/build_and_upload_gcp_artifacts.sh canary
Working in /Users/davidhan/zipline/chronon
Building wheel
Processing ./api/py
Preparing metadata (setup.py) ... done
Collecting click (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/click-8.1.8-py3-none-any.whl
Collecting thrift==0.21.0 (from zipline-ai==0.1.0.dev0)
Using cached thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
File was already downloaded /Users/davidhan/zipline/chronon/thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
Collecting google-cloud-storage==2.19.0 (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_storage-2.19.0-py2.py3-none-any.whl
Collecting google-auth<3.0dev,>=2.26.1 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_auth-2.37.0-py2.py3-none-any.whl
Collecting google-api-core<3.0.0dev,>=2.15.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_api_core-2.24.0-py3-none-any.whl
Collecting google-cloud-core<3.0dev,>=2.3.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_core-2.4.1-py2.py3-none-any.whl
Collecting google-resumable-media>=2.7.2 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_resumable_media-2.7.2-py2.py3-none-any.whl
Collecting requests<3.0.0dev,>=2.18.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/requests-2.32.3-py3-none-any.whl
Collecting google-crc32c<2.0dev,>=1.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_crc32c-1.6.0-cp311-cp311-macosx_12_0_arm64.whl
Collecting six>=1.7.2 (from thrift==0.21.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/six-1.17.0-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/googleapis_common_protos-1.66.0-py2.py3-none-any.whl
Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/protobuf-5.29.3-cp38-abi3-macosx_10_9_universal2.whl
Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/proto_plus-1.25.0-py3-none-any.whl
Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/cachetools-5.5.0-py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1_modules-0.4.1-py3-none-any.whl
Collecting rsa<5,>=3.1.4 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/rsa-4.9-py3-none-any.whl
Collecting charset-normalizer<4,>=2 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/charset_normalizer-3.4.1-cp311-cp311-macosx_10_9_universal2.whl
Collecting idna<4,>=2.5 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/idna-3.10-py3-none-any.whl
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/urllib3-2.3.0-py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/certifi-2024.12.14-py3-none-any.whl
Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1-0.6.1-py3-none-any.whl
Building wheels for collected packages: zipline-ai
Building wheel for zipline-ai (setup.py) ... done
Created wheel for zipline-ai: filename=zipline_ai-0.1.0.dev0-py3-none-any.whl size=7316 sha256=5327d4bc284b4c02ac1f1cc0078a748a59f027f2bd545f4675362f061467fa6b
Stored in directory: /private/var/folders/2p/h5v8s0515xv20cgprdjngttr0000gn/T/pip-ephem-wheel-cache-oqk_7n9r/wheels/82/21/73/c9719a19192e6cf69f01ea4fffb963c5d02e6228d8b875f326
Successfully built zipline-ai
Building jars
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:50:56 PM
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:51:08 PM
Are you sure you want to upload to these customer ids: canary
1) Yes
2) No
#? 1
+ for element in '"${customer_ids_to_upload[@]}"'
+ ELEMENT_JAR_PATH=gs://zipline-artifacts-canary/jars
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 9.8MiB/s
Average throughput: 11.2MiB/s
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 13.0MiB/s
Average throughput: 9.9MiB/s
+ gcloud storage cp zipline_ai-0.1.0.dev0-py3-none-any.whl gs://zipline-artifacts-canary/jars
Copying file://zipline_ai-0.1.0.dev0-py3-none-any.whl to gs://zipline-artifacts-canary/jars/zipline_ai-0.1.0.dev0-py3-none-any.whl
Completed files 1/1 | 7.1kiB/7.1kiB
+ echo Succeeded
Succeeded
+ break
```
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Added a new Bash script to automate the building and uploading of
Google Cloud Platform (GCP) artifacts.
- Supports automated JAR file generation and cloud storage upload for
specific projects.
- Includes error handling for missing arguments and files.
- Updated Python version declarations to support multiple versions.
- **Chores**
- Introduced build and deployment automation for GCP-related components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1209143482009686
chewy-zlai
pushed a commit
that referenced
this pull request
May 15, 2025
## Summary
^^^
```
(dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/add_upload_script) $ bash distribution/build_and_upload_gcp_artifacts.sh canary
Working in /Users/davidhan/zipline/chronon
Building wheel
Processing ./api/py
Preparing metadata (setup.py) ... done
Collecting click (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/click-8.1.8-py3-none-any.whl
Collecting thrift==0.21.0 (from zipline-ai==0.1.0.dev0)
Using cached thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
File was already downloaded /Users/davidhan/zipline/chronon/thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
Collecting google-cloud-storage==2.19.0 (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_storage-2.19.0-py2.py3-none-any.whl
Collecting google-auth<3.0dev,>=2.26.1 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_auth-2.37.0-py2.py3-none-any.whl
Collecting google-api-core<3.0.0dev,>=2.15.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_api_core-2.24.0-py3-none-any.whl
Collecting google-cloud-core<3.0dev,>=2.3.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_core-2.4.1-py2.py3-none-any.whl
Collecting google-resumable-media>=2.7.2 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_resumable_media-2.7.2-py2.py3-none-any.whl
Collecting requests<3.0.0dev,>=2.18.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/requests-2.32.3-py3-none-any.whl
Collecting google-crc32c<2.0dev,>=1.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_crc32c-1.6.0-cp311-cp311-macosx_12_0_arm64.whl
Collecting six>=1.7.2 (from thrift==0.21.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/six-1.17.0-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/googleapis_common_protos-1.66.0-py2.py3-none-any.whl
Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/protobuf-5.29.3-cp38-abi3-macosx_10_9_universal2.whl
Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/proto_plus-1.25.0-py3-none-any.whl
Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/cachetools-5.5.0-py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1_modules-0.4.1-py3-none-any.whl
Collecting rsa<5,>=3.1.4 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/rsa-4.9-py3-none-any.whl
Collecting charset-normalizer<4,>=2 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/charset_normalizer-3.4.1-cp311-cp311-macosx_10_9_universal2.whl
Collecting idna<4,>=2.5 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/idna-3.10-py3-none-any.whl
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/urllib3-2.3.0-py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/certifi-2024.12.14-py3-none-any.whl
Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1-0.6.1-py3-none-any.whl
Building wheels for collected packages: zipline-ai
Building wheel for zipline-ai (setup.py) ... done
Created wheel for zipline-ai: filename=zipline_ai-0.1.0.dev0-py3-none-any.whl size=7316 sha256=5327d4bc284b4c02ac1f1cc0078a748a59f027f2bd545f4675362f061467fa6b
Stored in directory: /private/var/folders/2p/h5v8s0515xv20cgprdjngttr0000gn/T/pip-ephem-wheel-cache-oqk_7n9r/wheels/82/21/73/c9719a19192e6cf69f01ea4fffb963c5d02e6228d8b875f326
Successfully built zipline-ai
Building jars
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:50:56 PM
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:51:08 PM
Are you sure you want to upload to these customer ids: canary
1) Yes
2) No
#? 1
+ for element in '"${customer_ids_to_upload[@]}"'
+ ELEMENT_JAR_PATH=gs://zipline-artifacts-canary/jars
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 9.8MiB/s
Average throughput: 11.2MiB/s
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 13.0MiB/s
Average throughput: 9.9MiB/s
+ gcloud storage cp zipline_ai-0.1.0.dev0-py3-none-any.whl gs://zipline-artifacts-canary/jars
Copying file://zipline_ai-0.1.0.dev0-py3-none-any.whl to gs://zipline-artifacts-canary/jars/zipline_ai-0.1.0.dev0-py3-none-any.whl
Completed files 1/1 | 7.1kiB/7.1kiB
+ echo Succeeded
Succeeded
+ break
```
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Added a new Bash script to automate the building and uploading of
Google Cloud Platform (GCP) artifacts.
- Supports automated JAR file generation and cloud storage upload for
specific projects.
- Includes error handling for missing arguments and files.
- Updated Python version declarations to support multiple versions.
- **Chores**
- Introduced build and deployment automation for GCP-related components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1209143482009686
chewy-zlai
pushed a commit
that referenced
this pull request
May 15, 2025
## Summary
^^^
```
(dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/add_upload_script) $ bash distribution/build_and_upload_gcp_artifacts.sh canary
Working in /Users/davidhan/zipline/chronon
Building wheel
Processing ./api/py
Preparing metadata (setup.py) ... done
Collecting click (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/click-8.1.8-py3-none-any.whl
Collecting thrift==0.21.0 (from zipline-ai==0.1.0.dev0)
Using cached thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
File was already downloaded /Users/davidhan/zipline/chronon/thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl
Collecting google-cloud-storage==2.19.0 (from zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_storage-2.19.0-py2.py3-none-any.whl
Collecting google-auth<3.0dev,>=2.26.1 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_auth-2.37.0-py2.py3-none-any.whl
Collecting google-api-core<3.0.0dev,>=2.15.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_api_core-2.24.0-py3-none-any.whl
Collecting google-cloud-core<3.0dev,>=2.3.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_core-2.4.1-py2.py3-none-any.whl
Collecting google-resumable-media>=2.7.2 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_resumable_media-2.7.2-py2.py3-none-any.whl
Collecting requests<3.0.0dev,>=2.18.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/requests-2.32.3-py3-none-any.whl
Collecting google-crc32c<2.0dev,>=1.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/google_crc32c-1.6.0-cp311-cp311-macosx_12_0_arm64.whl
Collecting six>=1.7.2 (from thrift==0.21.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/six-1.17.0-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/googleapis_common_protos-1.66.0-py2.py3-none-any.whl
Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/protobuf-5.29.3-cp38-abi3-macosx_10_9_universal2.whl
Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/proto_plus-1.25.0-py3-none-any.whl
Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/cachetools-5.5.0-py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1_modules-0.4.1-py3-none-any.whl
Collecting rsa<5,>=3.1.4 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/rsa-4.9-py3-none-any.whl
Collecting charset-normalizer<4,>=2 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/charset_normalizer-3.4.1-cp311-cp311-macosx_10_9_universal2.whl
Collecting idna<4,>=2.5 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/idna-3.10-py3-none-any.whl
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/urllib3-2.3.0-py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/certifi-2024.12.14-py3-none-any.whl
Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0)
File was already downloaded /Users/davidhan/zipline/chronon/pyasn1-0.6.1-py3-none-any.whl
Building wheels for collected packages: zipline-ai
Building wheel for zipline-ai (setup.py) ... done
Created wheel for zipline-ai: filename=zipline_ai-0.1.0.dev0-py3-none-any.whl size=7316 sha256=5327d4bc284b4c02ac1f1cc0078a748a59f027f2bd545f4675362f061467fa6b
Stored in directory: /private/var/folders/2p/h5v8s0515xv20cgprdjngttr0000gn/T/pip-ephem-wheel-cache-oqk_7n9r/wheels/82/21/73/c9719a19192e6cf69f01ea4fffb963c5d02e6228d8b875f326
Successfully built zipline-ai
Building jars
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:50:56 PM
[info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25)
[info] loading settings for project chronon-build from plugins.sbt ...
[info] loading project definition from /Users/davidhan/zipline/chronon/project
[info] loading settings for project root from build.sbt,version.sbt ...
[info] resolving key references (17954 settings) ...
[info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/)
"Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main"
+ rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main
+ thrift -version
[out] Thrift version 0.13.0
Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift
Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift
+ thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java
Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java
[info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
[success] Total time: 3 s, completed Jan 14, 2025, 12:51:08 PM
Are you sure you want to upload to these customer ids: canary
1) Yes
2) No
#? 1
+ for element in '"${customer_ids_to_upload[@]}"'
+ ELEMENT_JAR_PATH=gs://zipline-artifacts-canary/jars
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 9.8MiB/s
Average throughput: 11.2MiB/s
+ gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars
WARNING: Parallel composite upload was turned ON to get the best performance on
uploading large objects. If you would like to opt-out and instead
perform a normal upload, run:
`gcloud config set storage/parallel_composite_upload_enabled False`
If you would like to disable this warning, run:
`gcloud config set storage/parallel_composite_upload_enabled True`
Note that with parallel composite uploads, your object might be
uploaded as a composite object
(https://cloud.google.com/storage/docs/composite-objects), which means
that any user who downloads your object will need to use crc32c
checksums to verify data integrity. gcloud storage is capable of
computing crc32c checksums, but this might pose a problem for other
clients.
Copying file:///Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar
Completed files 4/1 | 155.8MiB/155.8MiB | 13.0MiB/s
Average throughput: 9.9MiB/s
+ gcloud storage cp zipline_ai-0.1.0.dev0-py3-none-any.whl gs://zipline-artifacts-canary/jars
Copying file://zipline_ai-0.1.0.dev0-py3-none-any.whl to gs://zipline-artifacts-canary/jars/zipline_ai-0.1.0.dev0-py3-none-any.whl
Completed files 1/1 | 7.1kiB/7.1kiB
+ echo Succeeded
Succeeded
+ break
```
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Added a new Bash script to automate the building and uploading of
Google Cloud Platform (GCP) artifacts.
- Supports automated JAR file generation and cloud storage upload for
specific projects.
- Includes error handling for missing arguments and files.
- Updated Python version declarations to support multiple versions.
- **Chores**
- Introduced build and deployment automation for GCP-related components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1209143482009686
chewy-zlai
pushed a commit
that referenced
this pull request
May 16, 2025
…203) ## Summary ^^^ ``` (dev_chronon) davidhan@Davids-MacBook-Pro: ~/zipline/chronon (davidhan/add_upload_script) $ bash distribution/build_and_upload_gcp_artifacts.sh canary Working in /Users/davidhan/zipline/chronon Building wheel Processing ./api/py Preparing metadata (setup.py) ... done Collecting cliour clients (from zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/cliour clients-8.1.8-py3-none-any.whl Collecting thrift==0.21.0 (from zipline-ai==0.1.0.dev0) Using cached thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl File was already downloaded /Users/davidhan/zipline/chronon/thrift-0.21.0-cp311-cp311-macosx_15_0_arm64.whl Collecting google-cloud-storage==2.19.0 (from zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_storage-2.19.0-py2.py3-none-any.whl Collecting google-auth<3.0dev,>=2.26.1 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/google_auth-2.37.0-py2.py3-none-any.whl Collecting google-api-core<3.0.0dev,>=2.15.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/google_api_core-2.24.0-py3-none-any.whl Collecting google-cloud-core<3.0dev,>=2.3.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/google_cloud_core-2.4.1-py2.py3-none-any.whl Collecting google-resumable-media>=2.7.2 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/google_resumable_media-2.7.2-py2.py3-none-any.whl Collecting requests<3.0.0dev,>=2.18.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/requests-2.32.3-py3-none-any.whl Collecting google-crc32c<2.0dev,>=1.0 (from google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/google_crc32c-1.6.0-cp311-cp311-macosx_12_0_arm64.whl Collecting six>=1.7.2 (from thrift==0.21.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/six-1.17.0-py2.py3-none-any.whl Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/googleapis_common_protos-1.66.0-py2.py3-none-any.whl Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/protobuf-5.29.3-cp38-abi3-macosx_10_9_universal2.whl Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core<3.0.0dev,>=2.15.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/proto_plus-1.25.0-py3-none-any.whl Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/cachetools-5.5.0-py3-none-any.whl Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/pyasn1_modules-0.4.1-py3-none-any.whl Collecting rsa<5,>=3.1.4 (from google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/rsa-4.9-py3-none-any.whl Collecting charset-normalizer<4,>=2 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/charset_normalizer-3.4.1-cp311-cp311-macosx_10_9_universal2.whl Collecting idna<4,>=2.5 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/idna-3.10-py3-none-any.whl Collecting urllib3<3,>=1.21.1 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/urllib3-2.3.0-py3-none-any.whl Collecting certifi>=2017.4.17 (from requests<3.0.0dev,>=2.18.0->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/certifi-2024.12.14-py3-none-any.whl Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=2.26.1->google-cloud-storage==2.19.0->zipline-ai==0.1.0.dev0) File was already downloaded /Users/davidhan/zipline/chronon/pyasn1-0.6.1-py3-none-any.whl Building wheels for collected paour clientsages: zipline-ai Building wheel for zipline-ai (setup.py) ... done Created wheel for zipline-ai: filename=zipline_ai-0.1.0.dev0-py3-none-any.whl size=7316 sha256=5327d4bc284b4c02ac1f1cc0078a748a59f027f2bd545f4675362f061467fa6b Stored in directory: /private/var/folders/2p/h5v8s0515xv20cgprdjngttr0000gn/T/pip-ephem-wheel-cache-oqk_7n9r/wheels/82/21/73/c9719a19192e6cf69f01ea4fffb963c5d02e6228d8b875f326 Successfully built zipline-ai Building jars [info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25) [info] loading settings for project chronon-build from plugins.sbt ... [info] loading project definition from /Users/davidhan/zipline/chronon/project [info] loading settings for project root from build.sbt,version.sbt ... [info] resolving key references (17954 settings) ... [info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/) "Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main" + rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main + mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main + thrift -version [out] Thrift version 0.13.0 Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java [warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list [info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar [success] Total time: 3 s, completed Jan 14, 2025, 12:50:56 PM [info] welcome to sbt 1.8.2 (Amazon.com Inc. Java 11.0.25) [info] loading settings for project chronon-build from plugins.sbt ... [info] loading project definition from /Users/davidhan/zipline/chronon/project [info] loading settings for project root from build.sbt,version.sbt ... [info] resolving key references (17954 settings) ... [info] set current project to chronon (in build file:/Users/davidhan/zipline/chronon/) "Generating files from thrift files at: /Users/davidhan/zipline/chronon/api/thrift/ into folder /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main" + rm -rf /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main + mkdir -p /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main + thrift -version [out] Thrift version 0.13.0 Processing file: /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/orchestration.thrift Processing file: /Users/davidhan/zipline/chronon/api/thrift/api.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/api.thrift Processing file: /Users/davidhan/zipline/chronon/api/thrift/observability.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/observability.thrift Processing file: /Users/davidhan/zipline/chronon/api/thrift/common.thrift + thrift --gen java:generated_annotations=suppress -out /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main /Users/davidhan/zipline/chronon/api/thrift/common.thrift Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/Cardinality.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSeriesKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftMetric.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummarySeries.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileSummary.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDrift.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/TileDriftSeries.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/observability/DriftSpec.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Artifact.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ScanStrategy.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalNode.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/LogicalType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ModelNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableRange.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableDependency.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TableNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/GroupByNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/ArtifactRange.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeConnections.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Table.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularData.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeGraph.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/JoinNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvEntry.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeInfo.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/NodeKey.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNodeInstance.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/PhysicalNode.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/StagingQueryNodeType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvRange.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/TabularDataType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/KvDependency.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/orchestration/Dependency.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Join.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupByServingInfo.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/BootstrapPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Window.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataKind.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Accuracy.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Operation.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Derivation.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/StagingQuery.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataSpec.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/MetaData.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EntitySource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/LabelParts.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/JoinSource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Query.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TimeUnit.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/AggregationPart.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/DataField.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ExternalSource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Aggregation.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Source.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/EventSource.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/TDataType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/ModelType.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/GroupBy.java Processing file: /Users/davidhan/zipline/chronon/api/target/scala-2.12/src_managed/main/ai/chronon/api/Model.java [info] Assembly jar up to date: /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar [success] Total time: 3 s, completed Jan 14, 2025, 12:51:08 PM Are you sure you want to upload to these customer ids: canary 1) Yes 2) No #? 1 + for element in '"${customer_ids_to_upload[@]}"' + ELEMENT_JAR_PATH=gs://zipline-artifacts-canary/jars + gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars WARNING: Parallel composite upload was turned ON to get the best performance on uploading large objects. If you would like to opt-out and instead perform a normal upload, run: `gcloud config set storage/parallel_composite_upload_enabled False` If you would like to disable this warning, run: `gcloud config set storage/parallel_composite_upload_enabled True` Note that with parallel composite uploads, your object might be uploaded as a composite object (https://cloud.google.com/storage/docs/composite-objects), which means that any user who downloads your object will need to use crc32c cheour clientssums to verify data integrity. gcloud storage is capable of computing crc32c cheour clientssums, but this might pose a problem for other clients. Copying file:///Users/davidhan/zipline/chronon/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar Completed files 4/1 | 155.8MiB/155.8MiB | 9.8MiB/s Average throughput: 11.2MiB/s + gcloud storage cp /Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar gs://zipline-artifacts-canary/jars WARNING: Parallel composite upload was turned ON to get the best performance on uploading large objects. If you would like to opt-out and instead perform a normal upload, run: `gcloud config set storage/parallel_composite_upload_enabled False` If you would like to disable this warning, run: `gcloud config set storage/parallel_composite_upload_enabled True` Note that with parallel composite uploads, your object might be uploaded as a composite object (https://cloud.google.com/storage/docs/composite-objects), which means that any user who downloads your object will need to use crc32c cheour clientssums to verify data integrity. gcloud storage is capable of computing crc32c cheour clientssums, but this might pose a problem for other clients. Copying file:///Users/davidhan/zipline/chronon/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar to gs://zipline-artifacts-canary/jars/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar Completed files 4/1 | 155.8MiB/155.8MiB | 13.0MiB/s Average throughput: 9.9MiB/s + gcloud storage cp zipline_ai-0.1.0.dev0-py3-none-any.whl gs://zipline-artifacts-canary/jars Copying file://zipline_ai-0.1.0.dev0-py3-none-any.whl to gs://zipline-artifacts-canary/jars/zipline_ai-0.1.0.dev0-py3-none-any.whl Completed files 1/1 | 7.1kiB/7.1kiB + echo Succeeded Succeeded + break ``` ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Added a new Bash script to automate the building and uploading of Google Cloud Platform (GCP) artifacts. - Supports automated JAR file generation and cloud storage upload for specific projects. - Includes error handling for missing arguments and files. - Updated Python version declarations to support multiple versions. - **Chores** - Introduced build and deployment automation for GCP-related components. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- - To see the specific tasks where the Asana app for GitHub is being used, see below: - https://app.asana.com/0/0/1209143482009686
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
^^^
Checklist
Summary by CodeRabbit
Summary by CodeRabbit
New Features
Chores