[Feature Request] Support "search-pipeline" in Multi-Search API

[DougTrajano]

Is your feature request related to a problem? Please describe

When using Hybrid Search, we must configure a Search Pipeline with a normalization-processor and inform it in the Search API operation using the search_pipeline=<search-pipeline-name> query parameter.

The Multi-search does not support this parameter, which means that we need to specify the Search Pipeline as a default search pipeline in the index. It's ok for some use cases, but it limits our capability to experiment with multiple normalization-processor.

Describe the solution you'd like

I want to have the search_pipeline in the URL parameters and metadata options of the Multi-search API operation.

Related component

Search

Describe alternatives you've considered

The current way to run multiple hybrid search objects using the Multi-search API operation is by defining the search pipeline as a default search pipeline in the index.

Additional context

No response

[epugh]

I just stumbled across the same situation. In my case, I'm using a ReactiveSearch library that assumes you are using the _msearch end point. I don't want to redo the entire communication layer just to use a pipeline, and I can't assume that a default pipeline makes sense since I want to experiment with different settings.

[msfroh]

@martin-gaievski and I were discussing this very idea a few weeks ago. Also, it came up in a chat with @joellerobinson at Haystack this year after her talk about combining results from multiple indices.

Multi-search opens up some interesting options for search pipelines. You could apply the same pipeline to all requests or apply different pipelines to each individual request. On top of that, you could do a "multi-request" or "multi-response" processor that transforms the full set of requests or responses. In particular, a "multi-response" processor could do result interleaving.

That said, @DougTrajano and @epugh -- I'm assuming that being able to specify a single search pipeline in the URL and having it apply to all requests (and their responses) would be sufficient. Is that correct?

[epugh]

A single search pipeline would at least bring the overall "feel" of the Multi-search more in line with regular search. It was quite a surprise to me. It would at least solve my specific problem!

Having said that, being able to pass these types of parameters as either url OR as part of the query seems like a "good thing". The search pipeline is a core part of how you express your query, it isn't just an add on ;-). Especially in the direction that we appear to be going in with more and more ranker type features enabled via the pipeline!

[DougTrajano]

@msfroh specifying it in the URL meets my use case, but I believe it will not cover additional use cases when different queries are performed.

To complete what @epugh said, since we will need to implement this, I believe it's more beneficial to add it at the query level.

[msfroh]

[Search Triage] This might actually be a "Good First Issue". We have the ability to specify an ad-hoc pipeline as a JSON object. I believe that when we parse the search request, we could check for a JSON string instead of an object. If it's a string, move it into the search request (as though it was a URL parameter).

[owaiskazi19]

@DougTrajano @epugh @msfroh raised #15923. Let me know what you all think.

[DougTrajano]

@DougTrajano @epugh @msfroh raised #15923. Let me know what you all think.

Hey buddy! It looks great. Good work!