Merge pull request #38 from maxshine/upgrade-gh-codebase-1.4

Upgrade gh codebase 1.4

Author: maxshine

.github/actions/test-automation-docker-action/CHANGELOG.md

@@ -40,4 +40,10 @@ N/A
 - Bugfix
 N/A
 - Dependencies
-  1. DBT Core version 1.8.5 with Bigquery adapter 1.8.2
+  1. DBT Core version 1.8.5 with Bigquery adapter 1.8.2
+
+### v1.4.1 (general release succeeding from v1.3.4 to latest codebase)
+- Features
+N/A
+- Bugfix
+N/A

.github/actions/test-automation-docker-action/Dockerfile

@@ -2,8 +2,8 @@ FROM python:3.10.11-slim-bullseye
 
 RUN apt update && apt install -y git curl && apt clean \
     && pip install --upgrade pip virtualenv \
-    && curl -sSL https://github.com/maxshine/personal-continuous-integrations/releases/download/v1.3.4/customizable_continuous_integration-1.3.4-py3-none-any.whl -o customizable_continuous_integration-1.3.4-py3-none-any.whl \
-    && pip install customizable_continuous_integration-1.3.4-py3-none-any.whl \
-    && rm customizable_continuous_integration-1.3.4-py3-none-any.whl
+    && curl -sSL https://github.com/maxshine/personal-continuous-integrations/releases/download/v1.4.1/customizable_continuous_integration-1.4.1-py3-none-any.whl -o customizable_continuous_integration-1.4.1-py3-none-any.whl \
+    && pip install customizable_continuous_integration-1.4.1-py3-none-any.whl \
+    && rm customizable_continuous_integration-1.4.1-py3-none-any.whl
 
 ENTRYPOINT ["ci_cli"]

.github/actions/write-protection-docker-action/CHANGELOG.md

@@ -76,4 +76,10 @@ N/A
 - Features
 N/A
 - Bugfix
+N/A
+
+### v1.4.1 (general release succeeding from v1.3.4 to latest codebase)
+- Features
+N/A
+- Bugfix
 N/A

.github/actions/write-protection-docker-action/Dockerfile

@@ -2,9 +2,9 @@ FROM python:3.10.11-slim-bullseye
 
 RUN apt update && apt install -y git curl && apt clean \
     && pip install --upgrade pip virtualenv \
-    && curl -sSL https://github.com/maxshine/personal-continuous-integrations/releases/download/v1.3.4/customizable_continuous_integration-1.3.4-py3-none-any.whl -o customizable_continuous_integration-1.3.4-py3-none-any.whl \
-    && pip install customizable_continuous_integration-1.3.4-py3-none-any.whl \
-    && rm customizable_continuous_integration-1.3.4-py3-none-any.whl
+    && curl -sSL https://github.com/maxshine/personal-continuous-integrations/releases/download/v1.4.1/customizable_continuous_integration-1.4.1-py3-none-any.whl -o customizable_continuous_integration-1.4.1-py3-none-any.whl \
+    && pip install customizable_continuous_integration-1.4.1-py3-none-any.whl \
+    && rm customizable_continuous_integration-1.4.1-py3-none-any.whl
 
 COPY write-protection-pr.sh /usr/local/bin/
 

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "customizable_continuous_integration"
-version = "1.4.0"
+version = "1.4.1"
 description = "The CI automation project"
 authors = ["Ryan,Gao <[email protected]>"]
 license = "MIT"

src/customizable_continuous_integration/automations/bigquery_archiver/README.md

@@ -97,4 +97,14 @@ Following table describes the common fields for both archive and restore task co
 ## Limitations
 1. The archive / restore leverage the user's GCP credentials to access the Bigquery and GCS resources. The user should have the necessary permissions to access the resources.
 2. While restoring the entities having interdependencies, the restoring process only check the completion of the previous task. In a case of failed requisites, the dependents will be restored anyway even if they are doomed to fail all the time.
-3. While restoring the entities having interdependencies, the built DAG assumes the interdependencies are one-way that Bigquery has checked this.
+3. While restoring the entities having interdependencies, the built DAG assumes the interdependencies are one-way that Bigquery has checked this.
+
+## Persistent data versioning
+### metadata_version (used to track GCP Bigquery metadata changes)
+1. `v1`
+   1. introduced on 2025-02-23 and tested with GCP Bigquery by 2025-04-02
+   2. supported from archiver version 1.4.0
+### archiver_version (used to track the archiver implementation changes)
+1. `v1` --  introduced on 2025-04-02 and tested with GCP Bigquery by 2025-04-02
+   1. introduced on 2025-02-23 and tested with GCP Bigquery by 2025-04-02
+   2. supported from archiver version 1.4.0

src/customizable_continuous_integration/automations/bigquery_archiver/entity/base.py

@@ -8,6 +8,7 @@
   23/02/2025   Ryan, Gao       Initial creation
   03/03/2025   Ryan, Gao       Add dependencies property
   23/03/2025   Ryan, Gao       Add DAGNodeInterface
+  02/04/2025   Ryan, Gao       Add archiver version for backwards compatibility
 """
 
 import datetime
@@ -50,6 +51,7 @@ def to_bigquery_schema_field(self) -> google.cloud.bigquery.SchemaField:
 class BigqueryBaseArchiveEntity(pydantic.BaseModel, DAGNodeInterface):
     bigquery_metadata: BigqueryBaseMetadata
     metadata_version: str = "v1"
+    archiver_version: str = "v1"
     gcs_prefix: str
     archived_datetime: datetime.datetime
     is_archived: bool = False

src/customizable_continuous_integration/automations/commands/README.md

@@ -52,6 +52,7 @@ Deprecated from *N/A*.
 | No. | Version | Features                                                                         | Bugfixes                                                |
 |:----|:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------|
 | 1   | v1.4.0  | - Initial function archiving Bigquery dataset into GCS and restoring it from GCS | N/A                                                     |
+| 2   | v1.4.1  | - Add archiver version field for future compatibility                            | N/A                                                     |
 
 ### help
 Available from **v1.4.0**.