offset.lag.max: 100 vs offset.lag.max: 9999999 in mirror maker 2

[Guruhbk]

Hi,
We are planning to use mirror-maker-2 for the migration of our kafka cluster. I'm working on the POC. On doing so, I find a strange pattern. I have 4000 consumer groups. If I set offset.lag.max: 100, only 2054 consumer groups are getting translated. However, if I set offset.lag.max: 9999999 all 4000 consumer groups are getting translated. I find it bit strange though.

What is the pros and cons of setting offset.lag.max to 100 or 9999999.

[katheris]

Hi @Guruhbk, the offset.lag.max configuration informs the MirrorSourceConnector how often to create a new record in the offset-syncs topic. The higher the number the less often the records will get created. The actual work of translating the committed offsets to the target cluster is done by the MirrorCheckpointConnector. So these two things are not related.

Is it possible it just took some time for the MirrorCheckpointConnector to translate all the committed offsets?

If you have more queries about how this works I'd recommend asking on the Apache Kafka users mailing list, since this is more of a Kafka question than a Strimzi one.

[Guruhbk]

Hi @katheris Thanks for your response. I don't think delay is the reason for it. I checked it multiple times and I also waited for a longer period. Somehow this offset.lag.max is playing a role in missing the consumer group translation. I just can't figure out why.

[katheris]

Hi @Guruhbk I think the best thing is to raise this through the Apache Kafka users mailing list that I suggested above. When you do I'd recommend sharing the exact steps you take in both scenarios as that will make it easier for people to help you investigate the root cause of the missing consumer groups.