[RFC] Isolating CPU for specific workloads(search or others)

[sgup432]

Is your feature request related to a problem? Please describe

I want to propose this idea and looking for some opinions from the community. Hopefully it doesn't sound terrible.

Context:
As of now, we have search and indexing workload(and others as well) running on the same node under the same process with their own threadpool defined. A lot of times certain "expensive" search queries cause CPU to spike till 100% on some/all nodes in a cluster and causes cluster instability. This can happen when all search threads seems to occupy majority/all CPU cores, also considering that we define number of search threads greater than available CPU cores on a node for performance reasons (formula: ((allocatedProcessors * 3) / 2) + 1).

We don't have a nice way as of today for users to isolate a search workload(for example) to not take up more than 70%(lets say) CPU, thereby providing resiliency and avoid node drops.

We have had discussion around defining "search" only nodes but it is complex and might be expensive for users to maintain a different node type. Existing Search backpressure feature also doesn't solve this holistically and plus relies on a cancellation mechanism(for search tasks after CPU goes beyond X%) which has its own limitations and may not work necessarily.

Unfortunately there is no way in JAVA to isolate heap for specific group of threads, otherwise this could have been extended to JVM heap as well.

Use case:
User can say that they want to allocate 70% of CPU to search workloads via some cluster setting. And we will accordingly map search threadpool to X% CPU cores or something like that.

Describe the solution you'd like

In Java, a concept called Java affinity exists where you can map of group of threads to certain CPU cores. And it is mostly used in cases where OS's thread scheduling does not provide optimal performance. But we can use it for isolating a group of threads to only run on X cpu cores on a Y core node(for example).
There also exists a library around this which provides a way to assign a group of threads to certain CPU cores(by reading /proc/cpuinfo) using a ThreadFactory.

I haven't done a POC yet as not sure of the feasibility.

Related component

Search:Resiliency

Describe alternatives you've considered

Have different processes for search/indexing but this seems to be way more complex.

Additional context

No response

[andrross]

we define number search threads greater than available CPU cores on a node for performance reasons

Can you expand on these reasons? It would probably be helpful context. Naively, if all search tasks were 100% CPU-bound then you could accomplish "User can say that they want to allocate 70% of CPU to search workloads" by defining the search thread pool size as allocated processors * 0.7.

[sgup432]

@andrross

Can you expand on these reasons? It would probably be helpful context.

Sure, I can try. Usually when we have systems like OpenSearch which does a lot of I/O bound tasks(network, file I/O), we can have a situation where a lot of threads are not doing much apart from waiting and in this case other threads(which needs to do some computation) can be scheduled(internally by Operating system) to use the CPU instead. This achieves much better CPU utilization. In case we had a service which is only meant to heavy CPU work(like computations), then it would have made sense to keep number of threads equal to the number of cores.

So I think we shouldn't solve this by reducing the thread pool size to allocated processors * 0.7, it will degrade the performance compared to the approach I have mentioned.

[andrross]

Thanks @sgup432. The question behind my question is whether there is opportunity to structure the code such that we aren't doing blocking operations on "search" threads, which would make your scaling decisions about that thread pool much simpler and avoid needing to over-provision relative to the physical compute resources.

[jainankitk]

The question behind my question is whether there is opportunity to structure the code such that we aren't doing blocking operations on "search" threads

@andrross - The blocking operations on "search" threads are for reading data from lucene segment files on disk. Hence IMO, unless we fundamentally change lucene or way opensearch works, we cannot avoid blocking of threads.

[sgup432]

In addition to what @jainankitk said, we can never avoid a situation where thread won't be interrupted or made space for another until unless we have an ideal system. This stackoverflow thread probably explains it better.

[peternied]

[Triage - attendees 1 2 3 4 5]
@sgup432 Thanks for creating this RFC, it might be useful to tie this to the #11061 effort that looks related

[reta]

Unfortunately there is no way in JAVA to isolate heap for specific group of threads, otherwise this could have been extended to JVM heap as well.

There some interesting options here that Graalvm explores, see please https://medium.com/graalvm/isolates-and-compressed-references-more-flexible-and-efficient-memory-management-for-graalvm-a044cc50b67e

In addition to what @jainankitk said, we can never avoid a situation where thread won't be interrupted or made space for another until unless we have an ideal system. This stackoverflow thread probably explains it better.

So this picture is a bit simplified and does not reflect the state we are now. Since we have concurrent search in GA (as of 2.12): there is a shared thread pool to manage segment traversals in parallel. In happy path, search threads do not do any I/O anymore but indexer search threads do.

We don't have a nice way as of today for users to isolate a search workload(for example) to not take up more than 70%(lets say) CPU, thereby providing resiliency and avoid node drops.

We cannot limit CPUs/heap on a thread pool efficiently (at least I do think so and would appreciate correction here), we could certainly try to estimate the pool's CPU/heap usage or limit the number of active threads to some % of cores, but all of those would not be very effective I think, we would end up with underutilization (fe many blocking threads would tell that the pool is doing nothing) or overutilization (once those blocking thread become runnable at the same time).

[sgup432]

There some interesting options here that Graalvm explores, see please https://medium.com/graalvm/isolates-and-compressed-references-more-flexible-and-efficient-memory-management-for-graalvm-a044cc50b67e

Thanks for the reference, I will take a look.

We cannot limit CPUs/heap on a thread pool efficiently (at least I do think so and would appreciate correction here)

Accordingly to my limited knowledge, the library I mentioned above provides a way to create a specific thread factory which is pinned to certain CPU cores. I am guessing we could use that(maybe with some tweak) to achieve this. As eventually search threadpool is essentially created by initializing a thread factory which is passed to ExecutorService(correct me if I am wrong here). Though I need to do more deep dive on the library part but first wanted to see if the idea makes any sense and initiate some discussions around it.

we could certainly try to estimate the pool's CPU/heap usage or limit the number of active threads to some % of cores, but all of those would not be very effective I think, we would end up with underutilization (fe many blocking threads would tell that the pool is doing nothing) or overutilization..

Yeah I agree that we might end up either over-utilizing or under-utilizing the CPU/heap usage. But this RFC accepts that caveat and is more focussed towards providing resiliency by isolating underlying resources for search(or indexing) workloads.