Skip to content

Commit 206815c

Browse files
committed
add archiver version information
1 parent fb12d5b commit 206815c

File tree

3 files changed

+14
-1
lines changed

3 files changed

+14
-1
lines changed

src/customizable_continuous_integration/automations/bigquery_archiver/README.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,14 @@ Following table describes the common fields for both archive and restore task co
9797
## Limitations
9898
1. The archive / restore leverage the user's GCP credentials to access the Bigquery and GCS resources. The user should have the necessary permissions to access the resources.
9999
2. While restoring the entities having interdependencies, the restoring process only check the completion of the previous task. In a case of failed requisites, the dependents will be restored anyway even if they are doomed to fail all the time.
100-
3. While restoring the entities having interdependencies, the built DAG assumes the interdependencies are one-way that Bigquery has checked this.
100+
3. While restoring the entities having interdependencies, the built DAG assumes the interdependencies are one-way that Bigquery has checked this.
101+
102+
## Persistent data versioning
103+
### metadata_version (used to track GCP Bigquery metadata changes)
104+
1. `v1`
105+
1. introduced on 2025-02-23 and tested with GCP Bigquery by 2025-04-02
106+
2. supported from archiver version 1.4.0
107+
### archiver_version (used to track the archiver implementation changes)
108+
1. `v1` -- introduced on 2025-04-02 and tested with GCP Bigquery by 2025-04-02
109+
1. introduced on 2025-02-23 and tested with GCP Bigquery by 2025-04-02
110+
2. supported from archiver version 1.4.0

src/customizable_continuous_integration/automations/bigquery_archiver/entity/base.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
23/02/2025 Ryan, Gao Initial creation
99
03/03/2025 Ryan, Gao Add dependencies property
1010
23/03/2025 Ryan, Gao Add DAGNodeInterface
11+
02/04/2025 Ryan, Gao Add archiver version for backwards compatibility
1112
"""
1213

1314
import datetime
@@ -50,6 +51,7 @@ def to_bigquery_schema_field(self) -> google.cloud.bigquery.SchemaField:
5051
class BigqueryBaseArchiveEntity(pydantic.BaseModel, DAGNodeInterface):
5152
bigquery_metadata: BigqueryBaseMetadata
5253
metadata_version: str = "v1"
54+
archiver_version: str = "v1"
5355
gcs_prefix: str
5456
archived_datetime: datetime.datetime
5557
is_archived: bool = False

src/customizable_continuous_integration/automations/commands/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ Deprecated from *N/A*.
5252
| No. | Version | Features | Bugfixes |
5353
|:----|:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------|
5454
| 1 | v1.4.0 | - Initial function archiving Bigquery dataset into GCS and restoring it from GCS | N/A |
55+
| 2 | v1.4.1 | - Add archiver version field for future compatibility | N/A |
5556

5657
### help
5758
Available from **v1.4.0**.

0 commit comments

Comments
 (0)