Bump the Cruise Control version to 2.5.142 #11288

ShubhamRwt · 2025-03-25T13:42:17Z

Type of change

Select the type of your PR

Task

Description

This PR fixes #11199. This PR bumps Cruise Control to 2.5.142 which is compatible with Kafka 4.0.0

Checklist

Please go through this checklist and make sure all applicable tasks have been done

Write tests
Make sure all tests pass
Update documentation
Check RBAC rights for Kubernetes / OpenShift roles
Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
Reference relevant issue(s) and close them after merging
Update CHANGELOG.md
Supply screenshots for visual changes, such as Grafana dashboards

Signed-off-by: ShubhamRwt <[email protected]>

docker-images/artifacts/kafka-thirdparty-libs/3.9.x/pom.xml

kyguy

Looks great @ShubhamRwt! Could you add the link of the related issue in the PR description too?

ppatierno · 2025-03-25T23:47:01Z

/azp run regression

azure-pipelines · 2025-03-25T23:47:14Z

Azure Pipelines successfully started running 1 pipeline(s).

ShubhamRwt · 2025-03-26T16:09:57Z

@ppatierno @im-konge Hi, Can we re-run the failed regression. I am not sure if its related to this PR

im-konge · 2025-03-27T08:15:55Z

@ShubhamRwt the test that is failing is the one that you enabled again for the CC. Could you please check it? Thanks.

ShubhamRwt · 2025-03-27T15:39:10Z

Hi, the failure on this tests seems to be unrelated to Cruise Control. The CC is pod is not deployed during this whole test. It seems like some logging changes in kafka 4.0.0 is triggering this failure.

I spent a lot of time today to understand the failure. It looks like the pods are suppose to roll when we change the specification of logging to from no logging related configuration to new external logging but with Kafka 4.0.0 there is no rolling of the pods happening and I don't see any error in the logs.

I also noticed that in some places we have comments like Kafka 3.9 -> rolling update is expected only with Kafka 3.9 for eg in test testDynamicallySetUnknownKafkaLogger().

I have multiple things to ask with respect to the observations:

It would be great to know if thats the case here also(i.e no roll of pods expected)
Do we need the logging configuration changes we have for kafka 4.0.0 i.e Creating an if-else to use the logging configuration based upon the kafka version 3.9.0 or 4.0.0.

One other question is should we have a separate issue to track this problem and we can merge this PR since its not associated with failure.

@ppatierno @im-konge @kyguy

im-konge · 2025-03-28T07:42:19Z

As I mentioned offline (and as you said in the comment as well), if it's not related to the CC and you don't want to fix this now, you can create an issue so someone else will have a look at it.
Maybe it was disabled because of some other issue and it shouldn't be run with Kafka 4.0.0 - I didn't have time to have a look at the test and what is the issue.

So yeah, in order to move this PR forward, I would just create a new issue, write there what you observed, add it to the assumption comment and leave it disabled for now. For example me or someone else will have a look at it later.

Signed-off-by: ShubhamRwt <[email protected]>

ShubhamRwt · 2025-04-01T07:04:06Z

@im-konge I have created the issue #11312 for tracking the failing test and disabled the test here with correct issue and comment

ShubhamRwt · 2025-04-01T12:42:21Z

@im-konge Hi, Can you run the regressions again on this, whenever you get some time? Thanks

ppatierno · 2025-04-01T13:06:46Z

/azp run regression

azure-pipelines · 2025-04-01T13:06:59Z

Azure Pipelines successfully started running 1 pipeline(s).

ShubhamRwt · 2025-04-02T08:31:44Z

The failure seems to be related to CC sending request to old pod rather then sending the request to new Pod. This is something which is coming up a lot lately even in the community. Maybe we should have some discussion around it on how we can tackle this. But again I guess this is something not intended to fix along with this PR

kyguy · 2025-04-02T14:48:50Z

Is it a test that we have had issues with before? Is it something we can reproduce locally?

The failure seems to be related to CC sending request to old pod rather then sending the request to new Pod. This is something which is coming up a lot lately even in the community.

Looking at the error, it appears to be related to the issue here [1], I assume this is the issue from the community you are mentioning, correct?

2025-04-01T16:29:08.2744375Z 2025-04-01 16:29:08 [main] INFO  [DeploymentUtils:183] Deployment: multiple-co-cluster-test/cluster-548f3edf-cruise-control is ready
2025-04-01T16:29:08.2745885Z 2025-04-01 16:29:08 [main] INFO  [DeploymentUtils:135] Deployment: multiple-co-cluster-test/cluster-548f3edf-cruise-control rolling update finished
2025-04-01T16:29:08.2766751Z 2025-04-01 16:29:08 [main] INFO  [KafkaUtils:159] Waiting for cluster stability
2025-04-01T16:30:11.1372669Z 2025-04-01 16:30:10 [main] INFO  [KafkaUtils:190] Kafka cluster is stable after: 61 polls
2025-04-01T16:30:11.1377649Z 2025-04-01 16:30:10 [main] INFO  [KafkaRebalanceUtils:83] ============================================================================
2025-04-01T16:30:11.1378296Z 2025-04-01 16:30:10 [main] INFO  [KafkaRebalanceUtils:84] NotReady
2025-04-01T16:30:11.1379151Z 2025-04-01 16:30:10 [main] INFO  [KafkaRebalanceUtils:85] ============================================================================
2025-04-01T16:30:11.1379664Z 2025-04-01 16:30:10 [main] INFO  [KafkaRebalanceUtils:90] Verifying that KafkaRebalance resource is in ProposalReady state
2025-04-01T16:30:11.1380268Z 2025-04-01 16:30:10 [main] INFO  [ResourceManager:493] Waiting for KafkaRebalance: multiple-co-cluster-test/cluster-548f3edf will have desired state: ProposalReady
2025-04-01T16:44:10.7412373Z 2025-04-01 16:44:10 [main] INFO  [ResourceManager:467] KafkaRebalance status:
2025-04-01T16:44:10.7413146Z 
2025-04-01T16:44:10.7413365Z Conditions:
2025-04-01T16:44:10.7413560Z 	Type: NotReady
2025-04-01T16:44:10.7414425Z 	Message: Error for request: cluster-548f3edf-cruise-control.multiple-co-cluster-test.svc:9090/kafkacruisecontrol/rebalance?json=true&dryrun=true&verbose=true&skip_hard_goal_check=true&goals=DiskCapacityGoal%2CCpuCapacityGoal%2CNetworkInboundCapacityGoal%2CMinTopicLeadersPerBrokerGoal%2CNetworkOutboundCapacityGoal%2CReplicaCapacityGoal&rebalance_disk=false. Server returned: Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: java.lang.NullPointerException: Cannot invoke "com.linkedin.kafka.cruisecontrol.config.BrokerCapacityInfo.capacity()" because the return value of "java.util.Map.get(Object)" is null'.

If we've determined that this issue is unrelated to this PR, could we consider one of the following options?

(A) Keep this test disabled, as we are already re-enabling it as part of this PR. We can address this specific issue in another PR.

(B) Fix the system test - perhaps by adding a line to refresh the KafkaRebalance resource after the new Cruise Control pod starts up.

[1] #11195

ShubhamRwt · 2025-04-02T15:23:11Z

@kyguy Hi, I have tried the 2nd solution you mentioned last week and it works fine. But I guess some users are having this issue even with autorebalance etc [1]. I think going with option 2 can be good for now but we should definitely revisit this problem on some discussion/call.

[1] #11296

kyguy · 2025-04-03T08:23:13Z

I think going with option 2 can be good for now but we should definitely revisit this problem on some discussion/call.

Did you already make the update to this PR? Or did you want to discuss it further before making the changes?

ShubhamRwt · 2025-04-03T09:20:08Z

@kyguy not yet, I was hoping to get views from @im-konge @ppatierno on this

im-konge · 2025-04-03T11:48:55Z

@ShubhamRwt it failed for the first time I guess or? Would adding the annotation fix the issue? I was referring in the issue you and @kyguy mentioned to little bit different problem, which seems to be not related.

TBH I don't really like the idea of just disabling each of the test and creating a lot of issues when you can have a look and fix it by one line (if it's the case). Same for the comment of the logging test -> you can just change the comment there that when Kafka 3.9.x will be dropped, we can remove the test -> and you don't need to leave it on me or someone else.

The issue there about failing test is there for quite a long time, so you can try the refresh annotation if it really fix something, but anyway if it fails on the CC rolling update (as I mentioned in the issue comment), someone will actually have a look at it.

systemtest/src/test/java/io/strimzi/systemtest/log/LoggingChangeST.java

ShubhamRwt · 2025-04-03T12:29:02Z

I agree with you @im-konge, Lets for now add the refresh annotation and see how effective it is. Thanks

Signed-off-by: ShubhamRwt <[email protected]>

scholzj · 2025-04-03T20:40:00Z

docker-images/artifacts/kafka-thirdparty-libs/4.0.x/pom.xml

+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-server</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-server-common</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-metadata</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-raft</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-group-coordinator</artifactId>
+                </exclusion>
+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-storage</artifactId>
+                </exclusion>


How did you come up with these exclusions? They do not seem to be a dependcies in the first place, so why exclude them?

When we build the operator and push the images, I was getting a lot collisions w.r.t these dependencies/jars

I can see some of the dependencies added in the build.gradle file for CC ->

api "org.apache.kafka:kafka-server:$kafkaVersion" api "org.apache.kafka:kafka-server-common:$kafkaVersion" api "org.apache.kafka:kafka-storage:$kafkaVersion"

You should likely check the pom file(s) and not the Gradle build files which mean many things. Also, if you see only "some" of them, it confirms that this is wrong and should be fixed.

Thanks for the review/suggestion Jakub, I was able to remove extra exclusions since I saw kafka-server is the only exclusion which was required. Itt was the one bringing in all the other stuff. Thanks again

Signed-off-by: ShubhamRwt <[email protected]>

im-konge · 2025-04-04T06:44:48Z

/azp run regression

azure-pipelines · 2025-04-04T06:44:59Z

Azure Pipelines successfully started running 1 pipeline(s).

ppatierno · 2025-04-07T08:26:42Z

So the issue we are facing in the commnunity needs more investigation but I think that applying "refresh" here should be enough. Anyway, I see the testKafkaCCAndRebalanceWithMultipleCOs test failing but for a different reasons AFAICS, which is not related to the PR itself. The test seems to "approve" by applying the annotation but, when using kubectl underneath, it's not overwriting the already existing annotation. Tbh I didn't expect this to fail because it works all times elsewhere. Maybe another reasons here @im-konge ?

im-konge · 2025-04-07T08:56:39Z

As I'm discussing with Shubham. Should we annotate it with refresh annotation only if the KafkaRebalance is NotReady state? Or should we apply the approve annotation with force? That would mean changing the behavior a bit for other tests as well.
Or operator if it's in different state then NotReady and Ready will not refresh the rebalance?

ppatierno · 2025-04-07T10:01:34Z

so as agreed offline, we can add the check of adding then "refresh" annotation only when the KafkaRebalance is in NotReady state (which is what we are facing here due to the old CC pod where the request goes). We'll try to address the main issue later on.

Signed-off-by: ShubhamRwt <[email protected]>

ppatierno · 2025-04-07T11:44:46Z

/azp run regression

azure-pipelines · 2025-04-07T11:44:59Z

Azure Pipelines successfully started running 1 pipeline(s).

ppatierno

LGTM

ppatierno · 2025-03-26T01:34:13Z

docker-images/artifacts/kafka-thirdparty-libs/4.0.x/pom.xml

+                <exclusion>
+                    <groupId>org.apache.kafka</groupId>
+                    <artifactId>kafka-raft</artifactId>
+                </exclusion>


are the above two kafka-metadata and kafka-raft being brought as dependencies of the others? Just asking.

Yes, its just kafka-server which was a new addition to the CC pom

Bump the Cruise Control version to 2.5.142

b0589e2

Signed-off-by: ShubhamRwt <[email protected]>

ShubhamRwt requested review from ppatierno, kyguy and im-konge March 25, 2025 13:43

im-konge added this to the 0.46.0 milestone Mar 25, 2025

im-konge reviewed Mar 25, 2025

View reviewed changes

docker-images/artifacts/kafka-thirdparty-libs/3.9.x/pom.xml Outdated Show resolved Hide resolved

kyguy approved these changes Mar 25, 2025

View reviewed changes

Disabled the loging related test

d9939e9

Signed-off-by: ShubhamRwt <[email protected]>

im-konge reviewed Apr 3, 2025

View reviewed changes

systemtest/src/test/java/io/strimzi/systemtest/log/LoggingChangeST.java Outdated Show resolved Hide resolved

Fixed the comment and added refresh annotation to the test

93355d6

Signed-off-by: ShubhamRwt <[email protected]>

scholzj reviewed Apr 3, 2025

View reviewed changes

Added changes suggested by Jakub

939a3f0

Signed-off-by: ShubhamRwt <[email protected]>

Fix the test failure

576eddd

Signed-off-by: ShubhamRwt <[email protected]>

im-konge approved these changes Apr 7, 2025

View reviewed changes

ppatierno approved these changes Apr 8, 2025

View reviewed changes

ppatierno merged commit 6019073 into strimzi:main Apr 8, 2025
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump the Cruise Control version to 2.5.142 #11288

Bump the Cruise Control version to 2.5.142 #11288

ShubhamRwt commented Mar 25, 2025 •

edited

Loading

kyguy left a comment

ppatierno commented Mar 25, 2025

azure-pipelines bot commented Mar 25, 2025

ShubhamRwt commented Mar 26, 2025 •

edited

Loading

im-konge commented Mar 27, 2025

ShubhamRwt commented Mar 27, 2025 •

edited

Loading

im-konge commented Mar 28, 2025

ShubhamRwt commented Apr 1, 2025

ShubhamRwt commented Apr 1, 2025 •

edited

Loading

ppatierno commented Apr 1, 2025

azure-pipelines bot commented Apr 1, 2025

ShubhamRwt commented Apr 2, 2025 •

edited

Loading

kyguy commented Apr 2, 2025

ShubhamRwt commented Apr 2, 2025

kyguy commented Apr 3, 2025

ShubhamRwt commented Apr 3, 2025

im-konge commented Apr 3, 2025

ShubhamRwt commented Apr 3, 2025

scholzj Apr 3, 2025

ShubhamRwt Apr 3, 2025 •

edited

Loading

ShubhamRwt Apr 3, 2025 •

edited

Loading

scholzj Apr 3, 2025

ShubhamRwt Apr 4, 2025 •

edited

Loading

im-konge commented Apr 4, 2025

azure-pipelines bot commented Apr 4, 2025

ppatierno commented Apr 7, 2025

im-konge commented Apr 7, 2025

ppatierno commented Apr 7, 2025

ppatierno commented Apr 7, 2025

azure-pipelines bot commented Apr 7, 2025

ppatierno left a comment

ppatierno Mar 26, 2025

ShubhamRwt Apr 8, 2025

Bump the Cruise Control version to 2.5.142 #11288

Bump the Cruise Control version to 2.5.142 #11288

Conversation

ShubhamRwt commented Mar 25, 2025 • edited Loading

Type of change

Description

Checklist

kyguy left a comment

Choose a reason for hiding this comment

ppatierno commented Mar 25, 2025

azure-pipelines bot commented Mar 25, 2025

ShubhamRwt commented Mar 26, 2025 • edited Loading

im-konge commented Mar 27, 2025

ShubhamRwt commented Mar 27, 2025 • edited Loading

im-konge commented Mar 28, 2025

ShubhamRwt commented Apr 1, 2025

ShubhamRwt commented Apr 1, 2025 • edited Loading

ppatierno commented Apr 1, 2025

azure-pipelines bot commented Apr 1, 2025

ShubhamRwt commented Apr 2, 2025 • edited Loading

kyguy commented Apr 2, 2025

ShubhamRwt commented Apr 2, 2025

kyguy commented Apr 3, 2025

ShubhamRwt commented Apr 3, 2025

im-konge commented Apr 3, 2025

ShubhamRwt commented Apr 3, 2025

scholzj Apr 3, 2025

Choose a reason for hiding this comment

ShubhamRwt Apr 3, 2025 • edited Loading

Choose a reason for hiding this comment

ShubhamRwt Apr 3, 2025 • edited Loading

Choose a reason for hiding this comment

scholzj Apr 3, 2025

Choose a reason for hiding this comment

ShubhamRwt Apr 4, 2025 • edited Loading

Choose a reason for hiding this comment

im-konge commented Apr 4, 2025

azure-pipelines bot commented Apr 4, 2025

ppatierno commented Apr 7, 2025

im-konge commented Apr 7, 2025

ppatierno commented Apr 7, 2025

ppatierno commented Apr 7, 2025

azure-pipelines bot commented Apr 7, 2025

ppatierno left a comment

Choose a reason for hiding this comment

ppatierno Mar 26, 2025

Choose a reason for hiding this comment

ShubhamRwt Apr 8, 2025

Choose a reason for hiding this comment

ShubhamRwt commented Mar 25, 2025 •

edited

Loading

ShubhamRwt commented Mar 26, 2025 •

edited

Loading

ShubhamRwt commented Mar 27, 2025 •

edited

Loading

ShubhamRwt commented Apr 1, 2025 •

edited

Loading

ShubhamRwt commented Apr 2, 2025 •

edited

Loading

ShubhamRwt Apr 3, 2025 •

edited

Loading

ShubhamRwt Apr 3, 2025 •

edited

Loading

ShubhamRwt Apr 4, 2025 •

edited

Loading