Skip to content

Fix range wheres #781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 452 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
452 commits
Select commit Hold shift + click to select a range
fb9a931
fix: properly set partition column (#407)
tchow-zlai Feb 20, 2025
6274388
Add CI checks for scala 2.13 builds/tests (#409)
kumar-zlai Feb 20, 2025
a212666
Add maven artifact pinning support (#412)
kumar-zlai Feb 21, 2025
d4c7bb6
Eval plugin for vscode (#388)
nikhil-zlai Feb 21, 2025
324c160
Move scala dependencies to separate repos for 2.12 and 2.13 (#413)
kumar-zlai Feb 21, 2025
7355b43
AWS Jar Distribution (#373)
chewy-zlai Feb 21, 2025
ac82c4a
Fixes to make fetch Join work in CLI including use `name` over nameTo…
david-zlai Feb 21, 2025
281a289
Revert "AWS Jar Distribution" (#420)
chewy-zlai Feb 21, 2025
ce8ab5f
chore: moving join codec from fetcher into metadata store (#401)
nikhil-zlai Feb 21, 2025
c57b03b
refactor: fetcher sub package + kill old stats in fetcher (#423)
nikhil-zlai Feb 22, 2025
96a2dee
Add support for running fetcher in docker & publishing image (#422)
piyush-zlai Feb 24, 2025
2ffa98b
Add AWS to Artifact Distribution Script (#421)
chewy-zlai Feb 24, 2025
7311b00
feat: StagingQuery param (#406)
tchow-zlai Feb 25, 2025
0ee2991
refactor: split fetcher logic into multiple files (#425)
nikhil-zlai Feb 25, 2025
4eac78b
feat: try out temporary views in staging query (#429)
tchow-zlai Feb 25, 2025
6022dac
fix: Fix approx_histogram_k aggregations (#411)
nikhil-zlai Feb 25, 2025
8366e5a
Add list & join schema fetcher APIs (#431)
piyush-zlai Feb 25, 2025
6360b22
update code to use . instead of / (#432)
ken-zlai Feb 25, 2025
0de5bff
fix: support non-string types for iceberg partition listing (#436)
tchow-zlai Feb 25, 2025
c707f3a
Enable batch IR caching by default + fix vertx bt startup issue (#435)
piyush-zlai Feb 26, 2025
34530e6
modify functions to allow frontend to pass percentiles (#434)
ken-zlai Feb 26, 2025
c041017
Switch fetcher service to log4j2 (#441)
piyush-zlai Feb 26, 2025
af16c88
feat: enriched staging query macros + unified import + source wrappin…
nikhil-zlai Feb 26, 2025
87f71c6
cleanup: remove unused files/dirs from the top level (#446)
nikhil-zlai Feb 27, 2025
0820860
cleanup: remove airflow dir (#447)
nikhil-zlai Feb 27, 2025
fc8b64d
cleanup: relocate docsite release scripts (#448)
nikhil-zlai Feb 27, 2025
ee8fbad
feat: basic hudi support (#437)
tchow-zlai Feb 27, 2025
336458d
Fix fetcher entrypoint in run.py (#450)
david-zlai Feb 27, 2025
38fe7b3
cleanup: remove tagged filter suite (#457)
nikhil-zlai Mar 2, 2025
b7abbbd
chore: slim down tableutils (#458)
tchow-zlai Mar 2, 2025
71b0842
Flink updates - dial down to one slot per TM to get around CU issues …
piyush-zlai Mar 2, 2025
e1a7fe6
Refactor spark to create modular jobs for orchestrator (#402)
varant-zlai Mar 3, 2025
10d8148
Cli improvements (#462)
nikhil-zlai Mar 4, 2025
fc36d9f
chore: Move gcp specific code out of run.py (#465)
david-zlai Mar 4, 2025
c6427cc
Stop using cloud_gcp_submitter jar and just use cloud_gcp jar, and ad…
david-zlai Mar 5, 2025
32d9031
feat: bigquery catalog with iceberg support (#393)
tchow-zlai Mar 5, 2025
2bacbe1
update dataproc url with region and project and update tail logs with…
david-zlai Mar 5, 2025
6a410a7
Migrate existing run quickstart scripts to `dev` customer id (#473)
david-zlai Mar 6, 2025
f5f5d2f
chore: reenable fetcher tests in bazel (#478)
tchow-zlai Mar 6, 2025
373466e
fix: adjust classpath so flink side can work (#475)
tchow-zlai Mar 6, 2025
d57a49a
chore: Split up run.py to gcp runner and default runner (#472)
david-zlai Mar 6, 2025
f0c950b
cleanup: remove unused api flags (#463)
nikhil-zlai Mar 7, 2025
b732cff
fix: downgrade jetty version so that we don't run into jdk compat iss…
tchow-zlai Mar 7, 2025
d8f5d11
fix: minor adjustment (#486)
tchow-zlai Mar 7, 2025
c463237
fix: bootstrap unit tests should be deterministic (#487)
tchow-zlai Mar 7, 2025
9b87516
Implement EMR submitter (#439)
david-zlai Mar 7, 2025
918aa0f
Skeleton node execution dao implementation using postgresSql driver (…
kumar-zlai Mar 8, 2025
742c8bb
feat: Support iceberg reads and writes (#424)
tchow-zlai Mar 8, 2025
98bb4d0
unified all artifacts to a single maven repository with pinned versio…
kumar-zlai Mar 9, 2025
c191038
Add Flink validation job + expose verb in streaming (#495)
piyush-zlai Mar 12, 2025
6a28184
chore: remove writeFormat as it is no longer used (#491)
tchow-zlai Mar 12, 2025
2bdc89b
Fix additional confs and update log uri for EMR (#498)
david-zlai Mar 13, 2025
48a6926
Add Hudi format (#496)
david-zlai Mar 13, 2025
fa29dd4
Point to release/latest/jars for application jar of emr (#504)
david-zlai Mar 13, 2025
6d8afd0
fix: remove references to custom json, float essential apis to top (#…
nikhil-zlai Mar 13, 2025
ddae560
feat: Use project id that's specified in the table name for the bigqu…
tchow-zlai Mar 14, 2025
b4e266e
chore: Simplify format abstraction (#508)
tchow-zlai Mar 14, 2025
331f7e2
Changes to shade & pull logging jars to help Etsy Patina lib integrat…
piyush-zlai Mar 14, 2025
5858d8b
Use Version Parameter to Get Jars (#511)
chewy-zlai Mar 14, 2025
247adf0
println in fetchermain (#513)
david-zlai Mar 14, 2025
f1519fe
Allow for partitionColumn to be formatted in getIcebergPartitions (#519)
david-zlai Mar 15, 2025
099ea8c
feat: `zipline init` to create project scaffolding (#512)
tchow-zlai Mar 16, 2025
137e9c6
feat: Make delegating BigQueryMetastore just a SparkCatalog (#520)
tchow-zlai Mar 18, 2025
f9563aa
stub/part_1: physical graph for workflow submission + column lineage …
nikhil-zlai Mar 19, 2025
e2a08e0
Execution layer interfaces with basic implementation (#443)
kumar-zlai Mar 19, 2025
a69684f
chore: replace instances of `spark.table` with `tableUtils.loadTable`…
tchow-zlai Mar 19, 2025
0bd12df
Catalyst Util updates to handle Split Whole Stage code gen + generate…
piyush-zlai Mar 19, 2025
017e76c
Add missing lineage.thrift in wheel gen scripts (#526)
piyush-zlai Mar 19, 2025
7296bc1
try target partition bytes (#523)
tchow-zlai Mar 20, 2025
d64ca4f
Enable flamegraphs in Flink (#527)
piyush-zlai Mar 20, 2025
fbe1424
Clean up DAO implementation test to make them more readable (#528)
kumar-zlai Mar 20, 2025
bf9c46e
chore: cutover canary-confs (#529)
tchow-zlai Mar 20, 2025
47b4eee
feat: Set iceberg table options through table properties (#531)
tchow-zlai Mar 21, 2025
54a156c
chore:remove unused method (#533)
tchow-zlai Mar 21, 2025
76652e7
Catalyst util refactoring to pull code out of hot path (#534)
piyush-zlai Mar 21, 2025
31a1f8a
chore: add ruff linter (#532)
nikhil-zlai Mar 21, 2025
1e52561
chore: bump bqms and iceberg version (#530)
tchow-zlai Mar 24, 2025
ddba2c3
Vz add orchestor service and cli interaction (#535)
varant-zlai Mar 25, 2025
2b77fb4
chore: remove caching and repartitioning feature (#539)
tchow-zlai Mar 26, 2025
ca28123
Etsy Patina / MMX shading reverts and other fixes (#544)
piyush-zlai Mar 27, 2025
92afca0
Rename py to python directory because pytest collision with its py di…
david-zlai Mar 27, 2025
57caa02
chore: bump spark (#555)
tchow-zlai Mar 28, 2025
615eb7c
chore: bump hudi to 1.X (#556)
tchow-zlai Mar 28, 2025
fef9b2c
Bump BigTable version + plumb through BT app profile id (#554)
piyush-zlai Mar 28, 2025
7487f99
Add error logging for when unable to wait for dataproc job (#550)
david-zlai Mar 28, 2025
0d8d557
feat: Partition Sensor (#547)
tchow-zlai Mar 28, 2025
3042f04
chore: Add more classes to serializer (#558)
tchow-zlai Mar 29, 2025
304c9fd
Update Flink job code to perform source projection (#545)
piyush-zlai Mar 31, 2025
3ce37f1
Temporary fix for analyzer since full conf path isn't preserved on da…
david-zlai Mar 31, 2025
1556afa
Rework BigTableKVStore multiget to issue a bulkGet request rather tha…
piyush-zlai Mar 31, 2025
0818927
fix: Integration Tests Use Candidate Version (#567)
chewy-zlai Apr 1, 2025
6d95ee9
Release Promotion (#514)
chewy-zlai Apr 1, 2025
9f3ac8d
Move common table info from TableDependency to a separate struct (#568)
kumar-zlai Apr 1, 2025
dbc6f40
Revert Flink 1 task slot per TM and bump parallelism (#565)
piyush-zlai Apr 1, 2025
243f3c0
Remove try catch around dataproc wait in run.py for gcp (#566)
david-zlai Apr 1, 2025
c5ba51e
Cherry-pick OSS PR: 870 - Add fetcher metrics (#569)
piyush-zlai Apr 1, 2025
63f6291
Add dataset tag to BigTableKVStore metrics to get dataset level break…
piyush-zlai Apr 1, 2025
50c480d
Adding verbs for modular join flow to driver (#575)
varant-zlai Apr 2, 2025
0daddfc
fix: thread the table props through properly (#576)
tchow-zlai Apr 2, 2025
6e1f9c0
spark opt: turn off time range check by default and select only relev…
varant-zlai Apr 2, 2025
63e3c03
Adding step days of 1 to source job (#578)
varant-zlai Apr 2, 2025
30353e7
Using tableUtils.partitionColumn rather than hardcoded ds (#579)
varant-zlai Apr 2, 2025
3942314
Simple LabelJoin flow (#546)
varant-zlai Apr 2, 2025
3f5851d
chore:remove flake8 (#583)
tchow-zlai Apr 3, 2025
3523db8
PubSub interface with GCP implementation and updated our workflow job…
kumar-zlai Apr 4, 2025
862a538
Use teams.py instead of teams.json for compile (#541)
david-zlai Apr 4, 2025
51c9bc0
feat: align wheel and jar versions (#559)
tchow-zlai Apr 4, 2025
a76fec8
Spark Opt: Remove BootstrapInfo from MergeJob (#587)
varant-zlai Apr 4, 2025
14a2f66
chore: move modular architecture pieces into dedicated module (#590)
tchow-zlai Apr 5, 2025
d0048b9
chore: delete unused prop (#585)
tchow-zlai Apr 5, 2025
3ba5a46
feat: nested thrift support for requests (#595)
nikhil-zlai Apr 7, 2025
32885aa
chore: remove unused schema invocation (#594)
tchow-zlai Apr 7, 2025
3202567
Remove more lineage.thrift references (#596)
david-zlai Apr 7, 2025
81f6008
chore: pull metrics into its own target (#592)
tchow-zlai Apr 7, 2025
a51f361
Use execution spark configs from compiled confs in JobSubmitter (#549)
david-zlai Apr 7, 2025
4d6924d
Check for final job status for gcp runs (#588)
david-zlai Apr 7, 2025
1fc8154
chore: more refactoring (#591)
tchow-zlai Apr 7, 2025
5079bf3
Dependency resolver logic refactor to api module with unit tests (#598)
kumar-zlai Apr 7, 2025
f16d25f
refactor: prior to introducing planner (#586)
nikhil-zlai Apr 7, 2025
5400413
Follow up fixes for compiler cutover (#603)
david-zlai Apr 7, 2025
d5b0242
Remove conf type from common because it already exists in hub (#607)
david-zlai Apr 8, 2025
6385e9d
Switch BigTableKV store implementation to leverage bulk read rows + s…
piyush-zlai Apr 8, 2025
9f86ab3
Fix integration tests for dev loop (#608)
david-zlai Apr 8, 2025
2402d6a
Update BigTable admin client and BQ client creation to be disabled in…
piyush-zlai Apr 8, 2025
5c66ad1
Set the outputNamespace for join parts and label parts (#612)
david-zlai Apr 9, 2025
4fc1f49
Support for multiple partitions in CheckPartitions (#597)
david-zlai Apr 9, 2025
16dac3c
Fix mode configs logic for when no modeConfigs are set (#613)
david-zlai Apr 9, 2025
dc36b56
Logging fix -- make root cause more clear if label job has misaligned…
varant-zlai Apr 9, 2025
d974ddd
fix: if outputNamespace is set, do not use the one from teams (#616)
tchow-zlai Apr 10, 2025
db9a2d7
chore: rename dev wheel to be pip friendly (#620)
tchow-zlai Apr 10, 2025
4e5a580
relative paths for compile (#614)
tchow-zlai Apr 10, 2025
6d4331d
Setting run first hole default to true (#621)
varant-zlai Apr 10, 2025
25b2129
chore: Pull schema stuff into its own online lib (#593)
tchow-zlai Apr 10, 2025
0987251
feat: GCPFormatProvider should do the default checks first (#617)
tchow-zlai Apr 10, 2025
92084d1
fix: compiler fixes (#619)
nikhil-zlai Apr 10, 2025
944f33f
fix: handle escaping characters properly (#624)
tchow-zlai Apr 10, 2025
1d69c8e
fix: fix a bug in the backticks (#628)
tchow-zlai Apr 11, 2025
0131c2e
Swap ordering of figuring out format and fix iceberg check (#626)
david-zlai Apr 11, 2025
f18e1be
new purchases (#623)
david-zlai Apr 11, 2025
4b40e88
fix: labels should be JoinParts not GroupBys (#631)
tchow-zlai Apr 11, 2025
8f73a3e
Add --project to gcloud cli calls (#632)
david-zlai Apr 11, 2025
1412eeb
fix: properly detect bigquery catalog (#629)
tchow-zlai Apr 12, 2025
c0be5a5
fix: escape column names before selecting with them (#635)
tchow-zlai Apr 13, 2025
e15d96e
fix: address bug in loadTable handling namespaces (#637)
tchow-zlai Apr 13, 2025
90589ea
chore: move partitionRunner to the stats module (#633)
tchow-zlai Apr 13, 2025
86fd9bc
Update LabelJoin to support sub-day windows and temporal accuracy (#636)
varant-zlai Apr 14, 2025
bc2e93f
Temporal persistence layer integration (#606)
kumar-zlai Apr 14, 2025
dae9dff
fix: Use package resources instead of data files (#642)
tchow-zlai Apr 15, 2025
7802446
Added new endpoint to orchestration service for dispatching pending j…
kumar-zlai Apr 15, 2025
de9c7bb
fix: get the full table name properly from Driver (#641)
tchow-zlai Apr 15, 2025
b73e8d4
Add bazel target for SBOM json file generation (#548)
kumar-zlai Apr 15, 2025
6353afb
Changes needed to support check-partitions verb (#615)
david-zlai Apr 15, 2025
a827ae4
fix: need to pass partition_column through from the helper method (#649)
tchow-zlai Apr 16, 2025
812b5cc
[airflow] -- add dependencies for airflow to customJson (#648)
varant-zlai Apr 16, 2025
a45cf93
BigTable / Fetcher updates - use closeAsync, setTimeouts, allow bulkR…
piyush-zlai Apr 16, 2025
0da2a12
Fixing failing AWS integration tests (#654)
varant-zlai Apr 16, 2025
b1511bd
chore: Remove use of DelegatingTable and bubble up exceptions properl…
tchow-zlai Apr 17, 2025
eda1e99
perf: Online + Avro path optimizations (#655)
nikhil-zlai Apr 17, 2025
b876cb1
fix: check for table reachability first (#660)
tchow-zlai Apr 17, 2025
b4b2bca
fix: Propagate all input partition columns through to the partition s…
tchow-zlai Apr 17, 2025
2a0bc38
Fix missing slfj4 implementation causing error logs at dataproc job s…
david-zlai Apr 17, 2025
1eb81b0
feat: improvements to zipline cli (#644)
tchow-zlai Apr 17, 2025
db8bf11
Make CLI Script Executable (#666)
chewy-zlai Apr 18, 2025
e963082
ck followups (#667)
david-zlai Apr 18, 2025
41e5edb
more ck followups (#669)
david-zlai Apr 18, 2025
be9ea4c
Fix: remove error logs showing up (#671)
david-zlai Apr 18, 2025
8e34052
feat: reorder logs so we can see what we query before doing it (#670)
tchow-zlai Apr 18, 2025
5afc499
Initial skeleton code for Agent service (#651)
kumar-zlai Apr 18, 2025
a849504
feat: use spark bq connector v1 (#664)
tchow-zlai Apr 18, 2025
a1f7bfd
artifact_prefix wasn't being set by teams.py env vars because those g…
david-zlai Apr 20, 2025
99cef59
fix: Couple more spots of addressing the proper name (#673)
tchow-zlai Apr 20, 2025
d4be200
fix: check-partitions missing an args (#677)
david-zlai Apr 21, 2025
fb08c34
Deduping airflow dependencies before writing json to file (#674)
varant-zlai Apr 21, 2025
3a7aa61
Removing label part assertions that no longer apply (#676)
varant-zlai Apr 21, 2025
82cf248
refactor tableutils into its own module catalog (#679)
tchow-zlai Apr 21, 2025
7a3d733
Set up Flink to report metrics via Prometheus (#678)
piyush-zlai Apr 22, 2025
96bc79f
Bump KV store + BT gax threadpools by 4x (#662)
piyush-zlai Apr 22, 2025
5023dee
chore: move kryo and spark session builder to submission module (#680)
tchow-zlai Apr 23, 2025
7d4c116
chore: remove dep on Extensions from TableUtils (#682)
tchow-zlai Apr 25, 2025
cafaddd
fix: broken integ tests (#688)
tchow-zlai Apr 26, 2025
daa234e
[spark] bugfix label join steps (#687)
varant-zlai Apr 26, 2025
10dc0ad
feat: support pseudocolumns in bigquery native tables (#689)
tchow-zlai Apr 26, 2025
a5b82b0
feat: do partition filtering on bq native tables by union individual …
tchow-zlai Apr 28, 2025
6849799
fix: preserve col ordering in selects (#691)
nikhil-zlai Apr 28, 2025
bdeda94
[Spark] disable checks in join monolith (#694)
varant-zlai Apr 28, 2025
abd7556
perf: resolve schema only once and cache (#696)
nikhil-zlai Apr 29, 2025
f5e9d4a
feat: BigQuery views (#698)
tchow-zlai Apr 29, 2025
da867e7
fix: respect customer artifact prefix everywhere (#702)
tchow-zlai Apr 29, 2025
1da5c3f
feat: support primary partition listing for bq native tables and view…
tchow-zlai Apr 30, 2025
6759ce4
Remove BigTable batcher implementation (#692)
piyush-zlai Apr 30, 2025
d6cabc5
fix: customer warehouse bucket (#704)
tchow-zlai Apr 30, 2025
db26796
Cherrypick OSS fetcher failure handling PRs - #932 and #964 (#706)
piyush-zlai Apr 30, 2025
226ad3b
Update JobSubmitter interface to take in clusterName and return submi…
kumar-zlai Apr 30, 2025
cafbc13
feat: still use information_schema for bigquery native tables (#710)
tchow-zlai May 1, 2025
f16cb7b
feat: Plumb partition ranges through partition listing (#711)
tchow-zlai May 1, 2025
82e82d5
Undo bazel targets for SBOM json file generation and add thrift codem…
kumar-zlai Apr 27, 2025
5c8785c
Added necessary github workflow yaml files
kumar-zlai Apr 29, 2025
7212304
Handle the case where partition filters are empty. (#717)
kumar-zlai May 1, 2025
123f1bc
chore: bring scripts distribution back (#718)
tchow-zlai May 1, 2025
9499ce5
chore: slim down TableUtils (#719)
tchow-zlai May 2, 2025
281c219
feat: support unpartitioned tables (#724)
tchow-zlai May 2, 2025
d9e403e
fix: Do not check partitions and use requested range only (#723)
tchow-zlai May 2, 2025
1af1abd
Add more tests (#720)
tchow-zlai May 2, 2025
2ff8387
Swap our metrics provider from statsd to otel-metrics + instrument ou…
piyush-zlai May 2, 2025
c9a08a5
Add request/response thrift types for start workflow http endpoint (#…
kumar-zlai May 2, 2025
10bb5d3
chore: Remove wrapwithCache (#721)
tchow-zlai May 5, 2025
a684b5a
feat: add a join to the gcp integration tests (#727)
tchow-zlai May 5, 2025
c16b583
Add timestamp type support (#733)
david-zlai May 5, 2025
742958d
chore: drop tables in the join integration test (#735)
tchow-zlai May 5, 2025
f08593b
feat: Use BigQuery exports as part of table loading in spark (#738)
tchow-zlai May 6, 2025
0fe9f53
Remove DDBKVStore rate limits (#741)
piyush-zlai May 6, 2025
a151d84
chore: Workflow to Sync Platform Subtree on Pushes to Main (#734)
chewy-zlai May 6, 2025
8fdbd10
WIP - planner concepts + join backfill impl (#739)
nikhil-zlai May 7, 2025
23cafb1
feat: add col to partition-spec (#731)
nikhil-zlai May 7, 2025
54ad5fb
Fix `risk` confs compiling (#745)
sean-zlai May 7, 2025
5305b6f
fix: remove some nondeterministic timestamp in ChainingFetcherTest (#…
tchow-zlai May 7, 2025
d80f165
oss/release: readme update (#744)
nikhil-zlai May 7, 2025
89b6525
Adding ability to set additional partitions in table deps (#747)
varant-zlai May 7, 2025
070bf42
Fix Airbnb Chronon repo link (#746)
sean-zlai May 7, 2025
3779e99
fix: Sync Workflow Lives in Chronon (#743)
chewy-zlai May 7, 2025
51032a3
Add partition_format to Query (#750)
david-zlai May 8, 2025
b7b804d
Add additional sub partitions to wait for in query API (#752)
david-zlai May 8, 2025
f1c27d7
Revive push_to_canary (#754)
chewy-zlai May 8, 2025
487d4d3
Vz/add test case for different partition formats (#753)
nikhil-zlai May 8, 2025
6edb34b
Zlib fix for updated MacOS (#755)
chewy-zlai May 8, 2025
6fcedb5
chore: Remove some prints (#756)
tchow-zlai May 9, 2025
74f0b86
chore: Sync Platform on Pull Requests Only (#757)
chewy-zlai May 9, 2025
d2d3acb
remove datagen script (#758)
tchow-zlai May 9, 2025
c2bdb61
Cleanup files (#759)
david-zlai May 10, 2025
0545a42
Fix push_to_canary to upload to passing-candidate instead of platform…
david-zlai May 10, 2025
98c616e
Add value info struct to join schema response to pull feature -> key …
piyush-zlai May 12, 2025
6c5862d
Fix modes allowed in CLI (#764)
david-zlai May 12, 2025
f027879
Delete Publish Release (#765)
chewy-zlai May 12, 2025
997e7f4
Rework complex CU tests to update schema a bit more generically (#766)
piyush-zlai May 12, 2025
80893b7
Make metrics optional (#767)
piyush-zlai May 12, 2025
19b4015
chore: Add Cloud Gcp Embedded Jar (#769)
chewy-zlai May 13, 2025
5f42b76
Rework Flink Schema providers to use existing SerDe and Mutation inte…
piyush-zlai May 13, 2025
998add7
Add a Flink canary app that can be run on demand (#762)
piyush-zlai May 13, 2025
53a367a
Adding logical type to conf thrift (#763)
varant-zlai May 14, 2025
d482c17
Add streaming verbs to check or deploy Flink jobs for GCP (#748)
david-zlai May 14, 2025
d312f52
Remove Duplicate Sync Workflow (#768)
chewy-zlai May 14, 2025
30f5356
Update to a different metadata path in GCS everytime to avoid collisi…
david-zlai May 15, 2025
f207aac
fix: Partition sensor check needs to search for the primary partition…
tchow-zlai May 15, 2025
c44643e
Add label join flag to custom json for airflow (#775)
varant-zlai May 15, 2025
e223f31
fix: run setups regardless of partitioning (#777)
tchow-zlai May 15, 2025
af07b90
Skip thrift json roundtrip check while reading GroupByServingInfo (#774)
piyush-zlai May 15, 2025
7413cde
replace partition spec with column (#776)
varant-zlai May 15, 2025
f354448
Fix range wheres
david-zlai May 16, 2025
00fc59d
comments
david-zlai May 16, 2025
06b8085
Fix
david-zlai May 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .bazelignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.git
1 change: 1 addition & 0 deletions .bazeliskrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
USE_BAZEL_VERSION=6.4.0
22 changes: 22 additions & 0 deletions .bazelproject
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
directories:
# Add the directories you want added as source here
# By default, we've added your entire workspace ('.')
.

# Automatically includes all relevant targets under the 'directories' above
derive_targets_from_directories: true

targets:
# If source code isn't resolving, add additional targets that compile it here

additional_languages:
# Uncomment any additional languages you want supported
# android
# dart
# go
# javascript
# kotlin
python
scala
typescript
java
31 changes: 31 additions & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
try-import %workspace%/.bazelrc.local

# To build with Scala 2.12, pass "--config scala_2.12" to "bazel build"
common:scala_2.12 --repo_env=SCALA_VERSION=2.12.18
common:scala_2.12 --define=SCALA_VERSION=2.12.18
common:scala_2.13 --repo_env=SCALA_VERSION=2.13.12
common:scala_2.13 --define=SCALA_VERSION=2.13.12

# Default scala version to 2.12
# To set a different default Scala version, add the following to .bazelrc.local:
# common --config scala_2.12
common --repo_env=SCALA_VERSION=2.12.18
common --define=SCALA_VERSION=2.12.18

build --java_language_version=11
build --java_runtime_version=11
build --remote_cache=https://storage.googleapis.com/zipline-bazel-cache
test --test_output=errors
test --test_timeout=1200

# Enable platform-specific configuration
build --enable_platform_specific_config

# macOS-specific flags
build:macos --copt=-Wno-macro-redefined
build:macos --copt=-Wno-deprecated-non-prototype
build:macos --host_copt=-Wno-macro-redefined
build:macos --host_copt=-Wno-deprecated-non-prototype

# CI-specific flags
build:linux --copt=-Wno-deprecated-function
38 changes: 38 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''

---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]

**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
20 changes: 20 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''

---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
76 changes: 76 additions & 0 deletions .github/image/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
FROM ubuntu:latest

# build using command: docker build --progress=plain -t chronon-base .

# Install necessary tools and Python
RUN apt update && apt install -y wget curl bash python3 python3-pip openjdk-17-jdk python3.12-venv

# java
ENV JAVA_HOME=/usr/lib/jvm/default-jvm
ENV PATH=$PATH:$JAVA_HOME/bin

# sbt for scala
RUN curl -L "https://github.com/sbt/sbt/releases/download/v1.8.2/sbt-1.8.2.tgz" | tar -xz -C /usr/local
ENV PATH="/usr/local/sbt/bin:${PATH}"

# bazel
RUN curl -fsSL "https://github.com/bazelbuild/bazelisk/releases/download/v1.18.0/bazelisk-linux-amd64" -o /usr/local/bin/bazel
RUN chmod +x /usr/local/bin/bazel
ENV PATH="/usr/local/bin:${PATH}"

# thrift
ARG THRIFT_VERSION=0.21.0
RUN apt install -y \
build-essential \
cmake \
libboost-dev \
libssl-dev \
libevent-dev \
bison \
flex \
autoconf \
automake \
libtool \
curl && \
curl -LSs https://archive.apache.org/dist/thrift/${THRIFT_VERSION}/thrift-${THRIFT_VERSION}.tar.gz -o thrift-${THRIFT_VERSION}.tar.gz && \
tar -xzf thrift-${THRIFT_VERSION}.tar.gz && \
cd thrift-${THRIFT_VERSION} && \
./configure --without-python --without-cpp --without-nodejs --without-java && \
make && \
make install && \
cd .. && \
rm -rf thrift-${THRIFT_VERSION} thrift-${THRIFT_VERSION}.tar.gz && \
apt purge -y \
build-essential \
cmake \
libboost-dev \
libssl-dev \
libevent-dev \
bison \
flex \
autoconf \
automake \
libtool \
curl && \
apt autoremove -y && \
rm -rf /var/lib/apt/lists/*

# Upgrade pip and install some common Python packages
RUN pip3 install --break-system-packages pytest tox flake8 ruff

RUN apt update && apt install -y build-essential git
RUN mkdir -p /usr/lib/jvm && ln -s /usr/lib/jvm/java-17-openjdk-amd64/ /usr/lib/jvm/default-jvm

# Verify installations
RUN java -version && \
thrift -version && \
python3 --version && \
pip3 --version && \
bazel --version && \
git --version

# Set working directory
WORKDIR /app

# Cmd to run when starting the container
CMD ["/bin/bash"]
8 changes: 8 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
## Summary

## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update

17 changes: 17 additions & 0 deletions .github/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
changelog:
exclude:
labels:
- ignore-for-release
categories:
- title: Major features / breaking changes
labels:
- Semver-Major
- title: Minor features
labels:
- Semver-Minor
- title: Bug fixes
labels:
- Semver-Patch
- title: Other changes
labels:
- "*"
35 changes: 35 additions & 0 deletions .github/workflows/build_and_push_docker.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Build and Push Docker Image

on:
push:
paths:
- '.github/image/Dockerfile'
workflow_dispatch:

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}-ci

jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- uses: actions/checkout@v4

- name: Log in to the Container registry
uses: docker/login-action@v1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build and push Docker image
uses: docker/build-push-action@v6
with:
context: .github/image
push: true
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
Loading
Loading