Multi-Topic subscription across tenant and namespace using regex in Java client throws IllegalArgumentException

[rnowacoski]

Describe the bug
When using a regex to create a consumer and a regex pattern is present in the tenant or namespace section of the pattern this error is thrown.

java.lang.IllegalArgumentException: Invalid named entity: \w+
	at org.apache.pulsar.common.naming.NamedEntity.checkName(NamedEntity.java:39)
	at org.apache.pulsar.common.naming.NamespaceName.validateNamespaceName(NamespaceName.java:179)
	at org.apache.pulsar.common.naming.NamespaceName.get(NamespaceName.java:53)
	at org.apache.pulsar.common.naming.TopicName.<init>(TopicName.java:151)
	at org.apache.pulsar.common.naming.TopicName.<init>(TopicName.java:38)
	at org.apache.pulsar.common.naming.TopicName$1.load(TopicName.java:63)
	at org.apache.pulsar.common.naming.TopicName$1.load(TopicName.java:60)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973)
	at org.apache.pulsar.shade.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957)
	at org.apache.pulsar.common.naming.TopicName.get(TopicName.java:88)
	at org.apache.pulsar.client.impl.PulsarClientImpl.patternTopicSubscribeAsync(PulsarClientImpl.java:403)
	at org.apache.pulsar.client.impl.PulsarClientImpl.subscribeAsync(PulsarClientImpl.java:333)
	at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribeAsync(ConsumerBuilderImpl.java:142)
	at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:99)

Example pattern: persistent://\w+/Test/Test1 or persistent://Test/\w+/Test1

This is because PulsarClientImpl.patternTopicSubscribeAsync is called when creating a consumer using a Pattern. This calls topicName.get(regex); where regex is the Pattern passed in. This method attempts to validate the tenant and namespace names against this regex ^[-=:.\\w]*$ in org.apache.pulsar.common.naming.NamedEntity. This fails for any regex pattern. The namespace name is needed in the current code because it is used by LookupService.getTopicsUnderNamespace to find the topics to subscribe to.

To Reproduce
Steps to reproduce the behavior:

1. Create a Consumer like below

client
      .newConsumer(Schema.BYTES)
      .topicsPattern(Pattern.compile("persistent://\w+/Test/Test1"))
      .subscribe()

2. Observe java.lang.IllegalArgumentException: Invalid named entity: \w+ is thrown

Expected behavior
Given a wildcard in a tenant or namespace I would expect the code to look up all tenants/namespaces that match and then lookup all topics to subscribe to.

Proposed Solution

1. LookupService implementations should implement a method like getNamespaces() that will return all tenant/namespaces.
2. PulsarClientImpl.patternTopicSubscribeAsync should filter these namespaces based on the regex given
3. For each renaming namespace that matches the given regex PulsarClientImpl.patternTopicSubscribeAsync should call LookupService.getTopicsUnderNamespace
4. List of topics for each matching namespace should be combined into one list and the consumer should subscribe to all of those topics

[jiazhai]

@rnowacoski currently regex subscription supports the topics in the same namespace. change this issue into a feature request. from my view, it is not hard to do, maybe we should pay more attention to the authorization.

[tisonkun]

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.

[tisonkun]

Moved to discussion forum. This seems an interesting feature request, but it requires volunteers work on it and more design details.

Whether or not support regexp in looking up tenant/ns/topic has no consensus yet.