Skip to content

feat: HA for schema registry #11982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Sep 5, 2023
Merged

feat: HA for schema registry #11982

merged 10 commits into from
Sep 5, 2023

Conversation

tabVersion
Copy link
Contributor

@tabVersion tabVersion commented Aug 31, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

high availability for schema registry.
RisingWave will request all provided URLs and return the first successful one.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

for schema registry related, we gonna allow multiple URLs chained up by , like kafka brokers

CREATE TABLE upsert_student_avro_json ()
WITH (
	connector = 'kafka',
  properties.bootstrap.server = 'message_queue:29092',
	topic = 'upsert_student_avro_json')
FORMAT UPSERT ENCODE AVRO (schema.registry = 'http://message_queue1:8081,http://message_queue2:8081,http://message_queue3:8081');

- Refactored and fixed various method calls and error handling in multiple files
- Updated `Client::new` method calls to accept a vector of URLs instead of a single URL in multiple files
- Added `await` keyword before certain method calls in multiple files
- Added error handling blocks for various error scenarios in multiple files
- Modified code execution paths based on certain conditions
- Removed unused arguments and functions in multiple files
- Renamed functions and variables for clarity in multiple files
- Implemented a `concurrent_req` method in `schema_registry/client.rs` to send concurrent requests to multiple URLs
- Removed unnecessary structs and methods in `schema_registry/client.rs`
- Defined utility functions and structs in `schema_registry/util.rs` for schema ID extraction and request handling

Signed-off-by: tabVersion <[email protected]>
@tabVersion tabVersion requested review from fuyufjh and wugouzi August 31, 2023 04:44
@github-actions github-actions bot added the type/feature Type: New feature. label Aug 31, 2023
@wugouzi
Copy link
Contributor

wugouzi commented Aug 31, 2023

Are we going to support this feature in frontend in this pr?

@tabVersion
Copy link
Contributor Author

Are we going to support this feature in frontend in this pr?

No, we will let user to specify multiple addrs in schema registry

eg. schema.registry = 'http://127.0.0.1:6789,http://127.0.0.1:7890'

- Add `handle_sr_list` function to handle schema registry URLs in multiple files
- Parse multiple URLs separated by commas in schema registry utility file
- Modify initialization of URL in JSON parser file to use the `handle_sr_list` function
- Remove unused import and update import of `handle_sr_list` in protobuf parser file
- Fix client initialization, URL retrieval, and error handling in protobuf parser file
- Modify `except` function to work with variable number of arguments in debezium avro parser file
- Fix bug in `display_filter` function and add utility function for handling errors in debezium avro parser file
- Update `parse_request` function to parse additional request fields in debezium avro parser file

Signed-off-by: tabVersion <[email protected]>
@tabVersion tabVersion marked this pull request as ready for review August 31, 2023 07:45
@tabVersion tabVersion requested a review from shanicky August 31, 2023 07:53
…rove request concurrency

- Modified the `new` function to accept a `Vec<Url>` parameter and a `client_config` parameter of type `SchemaRegistryAuth`
- Implemented logic to filter out invalid URLs and collect the valid URLs into a vector
- Implemented the `concurrent_req` function to perform concurrent HTTP requests and wait for the first successful response
- Implemented several functions to retrieve schemas and subjects from the schema registry
- Implemented a breadth-first search algorithm to retrieve all references for a subject
- Fixed bugs and improved error handling throughout the codebase

Signed-off-by: tabVersion <[email protected]>
@tabVersion tabVersion requested a review from hzxa21 August 31, 2023 13:02
Copy link
Contributor

@wugouzi wugouzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@hzxa21 hzxa21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

… function

- Refactor `concurrent_req` function in `client.rs` to handle remaining requests after one request succeeds.

Signed-off-by: tabVersion <[email protected]>
Copy link
Contributor

@neverchanje neverchanje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Bohan, I've read the code and it looks great.

@tabVersion tabVersion enabled auto-merge September 5, 2023 09:56
@tabVersion tabVersion added this pull request to the merge queue Sep 5, 2023
@codecov
Copy link

codecov bot commented Sep 5, 2023

Codecov Report

Merging #11982 (a0d8da9) into main (ff623ee) will decrease coverage by 0.03%.
Report is 7 commits behind head on main.
The diff coverage is 26.03%.

@@            Coverage Diff             @@
##             main   #11982      +/-   ##
==========================================
- Coverage   69.81%   69.79%   -0.03%     
==========================================
  Files        1402     1402              
  Lines      234730   234820      +90     
==========================================
+ Hits       163888   163890       +2     
- Misses      70842    70930      +88     
Flag Coverage Δ
rust 69.79% <26.03%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
src/connector/src/parser/debezium/avro_parser.rs 45.53% <0.00%> (+0.40%) ⬆️
src/connector/src/parser/json_parser.rs 83.98% <0.00%> (ø)
src/connector/src/parser/schema_registry/client.rs 0.00% <0.00%> (ø)
src/connector/src/parser/schema_registry/util.rs 33.61% <40.40%> (+33.61%) ⬆️
src/connector/src/parser/avro/parser.rs 69.83% <50.00%> (+0.16%) ⬆️
src/connector/src/parser/protobuf/parser.rs 68.81% <50.00%> (ø)

... and 6 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Merged via the queue into main with commit 69bde53 Sep 5, 2023
@tabVersion tabVersion deleted the tab/sr-ha branch September 5, 2023 10:27
neverchanje pushed a commit that referenced this pull request Sep 8, 2023
lmatz pushed a commit that referenced this pull request Sep 8, 2023
Signed-off-by: tabVersion <[email protected]>
Co-authored-by: Bohan Zhang <[email protected]>
Little-Wallace added a commit to Little-Wallace/risingwave that referenced this pull request Sep 8, 2023
commit e097adc
Author: StrikeW <[email protected]>
Date:   Fri Sep 8 13:16:15 2023 +0800

    fix: mysql-sink integration test (risingwavelabs#12161)

commit d972225
Author: congyi wang <[email protected]>
Date:   Fri Sep 8 12:16:06 2023 +0800

    chore(object store): change streaming write buffer size to 16MB for gcs (risingwavelabs#12150)

commit 8bd524b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Sep 8 11:36:49 2023 +0800

    chore(deps): Bump xxhash-rust from 0.8.6 to 0.8.7 (risingwavelabs#12159)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c924c37
Author: Yufan Song <[email protected]>
Date:   Thu Sep 7 10:55:57 2023 -0700

    feat(stream): add multiple server url feature into nats, delete useless field (risingwavelabs#12138)

commit 471aa2b
Author: William Wen <[email protected]>
Date:   Thu Sep 7 22:31:44 2023 +0800

    refactor(source): specify cdc generic type parameter for different cdc source (risingwavelabs#12109)

commit 7d940cf
Author: Xinhao Xu <[email protected]>
Date:   Thu Sep 7 21:22:06 2023 +0800

    feat(sink): Support cassandra sink (risingwavelabs#11878)

    Co-authored-by: Patrick Huang <[email protected]>

commit cc3faf6
Author: Yuhao Su <[email protected]>
Date:   Thu Sep 7 18:58:44 2023 +0800

    feat(watermark): handle watermark in project_set (risingwavelabs#12128)

commit 4bb7c66
Author: Li0k <[email protected]>
Date:   Thu Sep 7 18:44:17 2023 +0800

    feat(storage): reduce group deltas for l0 task (risingwavelabs#12085)

commit b2dc4eb
Author: Eric Fu <[email protected]>
Date:   Thu Sep 7 18:33:01 2023 +0800

    refactor(metrics): unify storage_metric_level & streaming_metric_level to 'metrics_level' (risingwavelabs#12099)

commit 5f75f9b
Author: Li0k <[email protected]>
Date:   Thu Sep 7 17:52:33 2023 +0800

    feat(storage): sepearte intra picker (risingwavelabs#12147)

commit f1672f7
Author: Richard Chien <[email protected]>
Date:   Thu Sep 7 16:26:22 2023 +0800

    fix(eowc): fix SortBuffer when there are multiple records having same timestamp (risingwavelabs#12146)

    Signed-off-by: Richard Chien <[email protected]>

commit 183a98b
Author: Xu <[email protected]>
Date:   Thu Sep 7 04:23:15 2023 -0400

    feat(expr): add `array_max` (risingwavelabs#12100)

    Co-authored-by: Runji Wang <[email protected]>

commit 893f636
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 08:06:35 2023 +0000

    chore(deps): Bump chrono from 0.4.26 to 0.4.29 (risingwavelabs#12136)

    Signed-off-by: dependabot[bot] <[email protected]>
    Signed-off-by: Runji Wang <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Runji Wang <[email protected]>

commit 3383291
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 15:49:07 2023 +0800

    chore(deps): Bump regex-automata from 0.3.7 to 0.3.8 (risingwavelabs#12134)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 5a8866d
Author: Li0k <[email protected]>
Date:   Thu Sep 7 15:28:56 2023 +0800

    refactor(storage): seperate validator logic from picker (risingwavelabs#11984)

commit b578db6
Author: Bugen Zhao <[email protected]>
Date:   Thu Sep 7 14:35:14 2023 +0800

    fix(parser): more error reporting with locations (risingwavelabs#12129)

    Signed-off-by: Bugen Zhao <[email protected]>

commit 3ee8a87
Author: Bugen Zhao <[email protected]>
Date:   Thu Sep 7 14:09:08 2023 +0800

    refactor(meta): refine `pause`/`resume` response (risingwavelabs#12079)

    Signed-off-by: Bugen Zhao <[email protected]>

commit f16809d
Author: xiangjinwu <[email protected]>
Date:   Thu Sep 7 13:58:38 2023 +0800

    fix(expr): parse `timestamptz` without seconds but with offset (risingwavelabs#12084)

    Co-authored-by: StrikeW <[email protected]>

commit 01ce1bb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 11:56:53 2023 +0800

    chore(deps): Bump num-bigint from 0.4.3 to 0.4.4 (risingwavelabs#12137)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit d79113e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Sep 7 10:42:13 2023 +0800

    chore(deps): Bump sysinfo from 0.29.9 to 0.29.10 (risingwavelabs#12135)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit ff4ef3a
Author: Noel Kwan <[email protected]>
Date:   Thu Sep 7 10:03:57 2023 +0800

    fix(ci): fix backwards compat with `ci-release` profile (risingwavelabs#12139)

commit 9c98d26
Author: Xu <[email protected]>
Date:   Wed Sep 6 14:31:10 2023 -0400

    feat(expr): add `array_min` (risingwavelabs#12071)

    Signed-off-by: Runji Wang <[email protected]>
    Co-authored-by: Runji Wang <[email protected]>

commit 084daa8
Author: Noel Kwan <[email protected]>
Date:   Wed Sep 6 23:07:24 2023 +0800

    test: Add nexmark test, kafka test for backwards compatibility testing (risingwavelabs#10413)

commit 3c19064
Author: Yufan Song <[email protected]>
Date:   Wed Sep 6 07:07:40 2023 -0700

    feat(stream): Add recovery feature for nats source (risingwavelabs#12073)

    Signed-off-by: tabVersion <[email protected]>
    Co-authored-by: tabVersion <[email protected]>

commit 52192e6
Author: Croxx <[email protected]>
Date:   Wed Sep 6 19:18:32 2023 +0800

    refactor: refactor file cache metrics build, bump foyer (risingwavelabs#12121)

    Signed-off-by: MrCroxx <[email protected]>

commit da89875
Author: William Wen <[email protected]>
Date:   Wed Sep 6 18:27:52 2023 +0800

    feat(log-store): refine log store trait and add update vnode bitmap in reader (risingwavelabs#11959)

commit e64a075
Author: William Wen <[email protected]>
Date:   Wed Sep 6 18:21:39 2023 +0800

    refactor(meta): introduce meta store ref and avoid pass generic parameter (risingwavelabs#12114)

commit b3a4195
Author: Yufan Song <[email protected]>
Date:   Tue Sep 5 23:57:14 2023 -0700

    fix(stream): remove useless stream in nats sink (risingwavelabs#12107)

commit 68087f3
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 06:24:42 2023 +0000

    chore(deps): bump flate2 from 1.0.26 to 1.0.27 (risingwavelabs#12103)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Bugen Zhao <[email protected]>

commit e3f3d47
Author: Shanicky Chen <[email protected]>
Date:   Wed Sep 6 14:07:14 2023 +0800

    fix: temporarily remove the assert checking for split migration (risingwavelabs#12117)

commit 61ef35b
Author: Li0k <[email protected]>
Date:   Wed Sep 6 14:04:14 2023 +0800

    chore(storage): remove compactor workload (risingwavelabs#12005)

commit a76e519
Author: Dylan <[email protected]>
Date:   Wed Sep 6 13:17:22 2023 +0800

    refactor(java_binding): extract java_binding to jni_core (risingwavelabs#12108)

    Co-authored-by: Eric Fu <[email protected]>
    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 99ecd5d
Author: Noel Kwan <[email protected]>
Date:   Wed Sep 6 12:48:47 2023 +0800

    fix(streaming): handle missing `cache_policy` field in `OverWindow` (risingwavelabs#12110)

commit 50792cc
Author: congyi wang <[email protected]>
Date:   Wed Sep 6 12:15:17 2023 +0800

    refactor(compaction): remove meta client from compactor context (risingwavelabs#12020)

commit b5dd428
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 12:02:52 2023 +0800

    chore(deps): bump openssl-sys from 0.9.92 to 0.9.93 (risingwavelabs#12106)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 1c87421
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 12:02:49 2023 +0800

    chore(deps): bump walkdir from 2.3.3 to 2.4.0 (risingwavelabs#12101)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 8c0da22
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Sep 6 12:02:46 2023 +0800

    chore(deps): bump deranged from 0.3.7 to 0.3.8 (risingwavelabs#12102)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 9ccdbe3
Author: zwang28 <[email protected]>
Date:   Tue Sep 5 21:26:31 2023 +0800

    chore(rpc): set message size to max for stream client (risingwavelabs#12082)

commit 4f50903
Author: Eric Fu <[email protected]>
Date:   Tue Sep 5 20:42:25 2023 +0800

    refactor(metrics): support streaming metrics level (risingwavelabs#11994)

commit 74c96fc
Author: Richard Chien <[email protected]>
Date:   Tue Sep 5 19:47:23 2023 +0800

    perf(over window): fix over window performance degradation after risingwavelabs#11576 (risingwavelabs#12093)

    Signed-off-by: Richard Chien <[email protected]>

commit fdaf30e
Author: Richard Chien <[email protected]>
Date:   Tue Sep 5 18:59:16 2023 +0800

    chore: add over window executor cache metrics (risingwavelabs#12092)

    Signed-off-by: Richard Chien <[email protected]>

commit ec0027e
Author: William Wen <[email protected]>
Date:   Tue Sep 5 18:22:38 2023 +0800

    refactor(source): implement the common split reader as a generic function (risingwavelabs#12059)

commit 5cf0351
Author: Shanicky Chen <[email protected]>
Date:   Tue Sep 5 18:09:27 2023 +0800

    feat: modify the scale command to support both horizontal and vertical scaling. (risingwavelabs#12087)

commit 69bde53
Author: Bohan Zhang <[email protected]>
Date:   Tue Sep 5 18:03:18 2023 +0800

    feat: HA for schema registry (risingwavelabs#11982)

    Signed-off-by: tabVersion <[email protected]>

commit 477bfd4
Author: StrikeW <[email protected]>
Date:   Tue Sep 5 16:57:55 2023 +0800

    fix(jdbc-sink):  quote identifier to handle case-sensitive case (risingwavelabs#12078)

commit aea50f5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 07:29:58 2023 +0000

    chore(deps): bump async-recursion from 1.0.4 to 1.0.5 (risingwavelabs#12069)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 579405e
Author: Bugen Zhao <[email protected]>
Date:   Tue Sep 5 14:43:46 2023 +0800

    feat(grafana): report local barrier manager progress (risingwavelabs#12001)

    Signed-off-by: Bugen Zhao <[email protected]>

commit e4b9a08
Author: Bugen Zhao <[email protected]>
Date:   Tue Sep 5 14:39:58 2023 +0800

    refactor(streaming): better traces for development (risingwavelabs#12024)

    Signed-off-by: Bugen Zhao <[email protected]>

commit 65fc959
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 06:37:26 2023 +0000

    chore(deps): bump tracing-opentelemetry from 0.20.0 to 0.21.0 (risingwavelabs#12010)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Runji Wang <[email protected]>

commit aadc605
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 14:17:42 2023 +0800

    chore(deps): bump google-cloud-pubsub from 0.19.0 to 0.20.0 (risingwavelabs#12066)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit a5ef008
Author: Noel Kwan <[email protected]>
Date:   Tue Sep 5 14:05:59 2023 +0800

    feat(storage): refactor `LocalStateStore::init` and `StateTable::init` (risingwavelabs#12050)

commit ff623ee
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 03:49:12 2023 +0000

    chore(deps): bump syn from 2.0.29 to 2.0.31 (risingwavelabs#12067)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit f95d972
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Sep 5 03:49:01 2023 +0000

    chore(deps): bump memchr from 2.6.2 to 2.6.3 (risingwavelabs#12068)

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 3f5a6c8
Author: Tao Wu <[email protected]>
Date:   Tue Sep 5 11:23:49 2023 +0800

    feat: support information_schema.views (risingwavelabs#12045)

commit 999b9d2
Author: wu <[email protected]>
Date:   Mon Sep 4 18:10:08 2023 +0800

    feat: auto rebuild source reader in executor (risingwavelabs#11993)

    Co-authored-by: Bohan Zhang <[email protected]>

commit fa8b5e3
Author: Shanicky Chen <[email protected]>
Date:   Mon Sep 4 17:10:38 2023 +0800

    feat: add `target_parallelism_per_worker` field for scaling (risingwavelabs#11945)

commit deac61e
Author: Croxx <[email protected]>
Date:   Mon Sep 4 17:07:26 2023 +0800

    feat(hummock): refill cache before apply version delta (risingwavelabs#11950)

    Signed-off-by: MrCroxx <[email protected]>

commit 2ea66b8
Author: Bohan Zhang <[email protected]>
Date:   Mon Sep 4 16:15:30 2023 +0800

    chore: refine error message on unhandled error in kinesis (risingwavelabs#12055)

    Signed-off-by: tabVersion <[email protected]>

commit dcfbfb5
Author: August <[email protected]>
Date:   Mon Sep 4 14:50:29 2023 +0800

    feat: support for migration by new parallel units (risingwavelabs#11892)

commit 0d4da3b
Author: Bohan Zhang <[email protected]>
Date:   Mon Sep 4 14:10:49 2023 +0800

    fix: reject kinesis source if the start up mode is not sequence number but sequence number is provided (risingwavelabs#12048)

Signed-off-by: Little-Wallace <[email protected]>
Li0k pushed a commit that referenced this pull request Sep 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature Type: New feature.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants