[AUTOCUT] Gradle Check Flaky Test Report for WorkloadManagementIT

[opensearch-ci-bot]

Flaky Test Report for WorkloadManagementIT

Noticed the WorkloadManagementIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
4a48b4b	17147	52716	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}}
8fd9aeb	18631	59881	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}
bc04657	17276	53165	org.opensearch.wlm.WorkloadManagementIT.testHighMemoryInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"false"}}
bf8696e	18548	59389	org.opensearch.wlm.WorkloadManagementIT.testHighCPUInEnforcedMode {p0={"search.concurrent_segment_search.enabled":"true"}}

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the WorkloadManagementIT class are:

• 16818
• 17591
• 17043
• 17137
• 17190
• 17299
• 17390
• 17446
• 17555
• 17575
• 17631
• 17635
• 17714
• 17777
• 17817
• 18054
• 18215
• 18470
• 18497
• 18573

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[kaushalmahi12]

@ruai0511 Can you look into it ?

[krisfreedain]

Catch All Triage - 1

[andrross]

@ruai0511 @kaushalmahi12 If I'm reading the code correctly, it looks like there is a race condition in the "testHighMemory" case here:

OpenSearch/server/src/internalClusterTest/java/org/opensearch/wlm/WorkloadManagementIT.java

Lines 414 to 418 in df3e645

	case "MEMORY":
	int bytesToAllocate = (int) (Runtime.getRuntime().totalMemory() * 0.01);
	Byte[] bytes = new Byte[bytesToAllocate];
	int[] ints = new int[bytesToAllocate];
	break;

The code will continuously loop while allocating new on heap arrays, but those arrays immediately go out of scope and are eligible for garbage collection. It seems possible to observe the resources when the doWork() loop is holding no references and the GC has freed everything, in which case the memory threshold might not be breached. It seems like it would be better to allocate the memory and hold on to the references until the task is canceled to guarantee the resource utilization.