-
Notifications
You must be signed in to change notification settings - Fork 637
fix(source): resolve avro Ref
during avro_schema_to_column_descs
without hack
#19601
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rubber stamp
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
For context, this is the error reported for the added test case if this PR was not applied:
Both parts will be included in v2.2. In this intermediate state, |
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Part of #17020.
Ref
in avro used to be supported by a hack we added in our apache_avro fork. This PR avoids the hack (replacingRef
with their referee in-place, resulting in an invalid tree containing duplicate definitions). The hack fails to work when there isRef
insideRef
- resulting in either unresolvedRef
or infinite recursion.This PR only corrects part of the usage - in
avro_schema_to_column_descs
to derive RisingWave column data types from avro schema. There will be a follow-up to correct the usage inconvert_to_datum
/AvroAccess
. Without the latter part, simple data types likeint
(inside aRef
in anotherRef
) can already be supported.Instead of building the expanded-yet-invalid tree as in the hack, this solution passes a
NamesRef
obtained from the root. Its complexity of having an associated lifetime is easy to deal with in this context. (To contrast,prost_reflect
for protobuf builds the tree withArc
- no lifetime and no duplication. Butapache_avro::Schema
does not useArc
.)The circular reference rejection logic is same as #10499 for protobuf. It can be DRY'ed with error message improved later.
This is intended to be part of v2.2 and NOT cherry-picked into earlier versions. There has been a further hack available for earlier versions that can be cherry-picked on demand - where the user is responsible for not using circular reference or an infinite recursion would happen.
Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.