[improve][io][kca] support fully-qualified topic names in source records #24248
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@lhotari, @AlvaroStream - I hope you don't mind me tagging you in this pull request. I took the liberty to do this because of the discussions we had around #24221 😊 |
change LGTM, @efcasado. Thanks for continuing the contributions! |
Sure thing. |
efcasado
force-pushed
the
kca-fq-destination-topic-names
branch
2 times, most recently
from
5b35aaa to
6afa94c
Compare
efcasado
force-pushed
the
kca-fq-destination-topic-names
branch
from
6afa94c to
fa9a291
Compare
|
Done. I added a couple of simple tests (ie. short and fully-qualified topic names). Passing build can be found here. |
manas-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 14, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit fd7bfac)
manas-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 14, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit 2cb88c6)
manas-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 14, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit fd7bfac)
manas-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 14, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit 2cb88c6)
manas-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 15, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit 2cb88c6)
manas-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 15, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit 2cb88c6)
srinath-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 16, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit 2cb88c6)
srinath-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 18, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9) (cherry picked from commit fd7bfac)
nodece
pushed a commit
to ascentstream/pulsar
that referenced
this pull request
May 28, 2025
…rds (apache#24248) (cherry picked from commit de1d4c9)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This change builds on the work previously done in #24221.
The current implementation of the Kafka Connect adaptor in Pulsar IO does not support fully-qualified Pulsar topic names in source records. Instead, it forcefully prepends the default
persistent://<tenant>/<namespace>/prefix to all destination topics. This behavior can be problematic in multi-tenant environments where dynamic topic routing is required.For example, consider a setup with:
tenant1,tenant2)tenant1,tenant2)global(shared) tenantUsing Kafka Connect transformations, users may want to route records to tenant-specific topics based on the PostgreSQL schema:
With this configuration, changes to
tenant1.ordersshould go topersistent://tenant1/procurement/orders, andtenant2.orderstopersistent://tenant2/procurement/orders.However, the current implementation prepends
persistent://global/procurement/to the already fully-qualified topic, resulting in invalid topic names likepersistent://global/procurement/persistent://tenant1/procurement/orders. This causes runtime exceptions and connector failure loops.By supporting fully-qualified topic names, this change enables more flexible and tenant-aware architectures without requiring additional processing layers.
Modifications
This change adjusts
AbstractKafkaConnectSourceto correctly handle fully-qualified topic names by using theorg.apache.pulsar.common.naming.TopicNameutility. If the topic name is valid and fully-qualified, it is respected as-is. Otherwise, the adaptor falls back to the existing behavior, ensuring backward compatibility.Verifying this change
This change added tests and can be verified as follows:
KafkaConnectSourceTesttest suite with a couple of tests to confirm Kafka Connect Adaptor can handle short and fully-qualified topic namesDocumentation
docdoc-requireddoc-not-neededdoc-complete