[BUG] response header TASK_RESOURCE_USAGE should be optional

[nicolasfranck]

Describe the bug

I noticed just recently that opensearch adds the header TASK_RESOURCE_USAGE to every response,
which contains, in json, a full report of the resource usage during the request (memory, cpu cycles, and where this happened).

e.g.

TASK_RESOURCE_USAGE: {"action":"indices:data/read/search[phase/query+fetch/scroll]","taskId":12451,"parentTaskId":29630,"nodeId":"Fu39EBN3T7u6YK7LxNCJEw","taskResourceUsage":{"cpu_time_in_nanos":4520542,"memory_in_bytes":826600}}
TASK_RESOURCE_USAGE: {"action":"indices:data/read/search[phase/query]","taskId":12447,"parentTaskId":29623,"nodeId":"Fu39EBN3T7u6YK7LxNCJEw","taskResourceUsage":{"cpu_time_in_nanos":22117710,"memory_in_bytes":853696}}

For simple search query I mostly see one such header, but in (for example) _delete_by_query calls I see hundreds of them. I created an index with 200.000 items, and deleted the documents with a delete-by-query, and saw around 400 of such headers.

Apparently, this can be disabled with setting task_resource_tracking.enabled=false, but shouldn't this be disabled by default? This looks like a debugging tool?

I only started noticing this when a client library (Mojo::Client in perl) refused to process these responses because of the exceeded amount of headers.

Related component

No response

To Reproduce

1. Create an index of around 200K documents
2. Execute a delete by query (match-all)
3. See the hundreds of headers TASK_RESOURCE_USAGE

Expected behavior

Setting task_resource_tracking.enabled should be disabled by default

Additional Details

No response

[andrross]

@ansjcy @jainankitk Can one of you take a look at this?

[ansjcy]

Hi @nicolasfranck ! Thanks for submitting the ticket. There was a PR to fixed a similar issue: #16532, may I know what version of OpenSearch are you running?

[nicolasfranck]

version 2.18.0. Maybe I am reporting an outdated issue, but it was hard to find anything about it on the internet.

[jainankitk]

@kaushalmahi12 - I am wondering if this is caused by Workload Management, and should be disabled in response by default?