[BUG] Unexpected Ranking Behavior in Hybrid Query with Min-Max Normalization and Arithmetic Mean Combination

[rohantilva]

Opensearch Version: 2.15
Environment: AWS OpenSearch

Issue Description

I am executing hybrid queries with three sub-queries on a large dataset containing tens to hundreds of thousands of documents. The queries are weighted as follows: [0.9998, 0.0001, 0.0001], with the first query having the highest weight. However, I am seeing unexpected results where a document with a high score from the first query is missing from the top results in the final ranking, while documents with lower scores from the same query are included.

Example:

• Documents: A, B, C, D
• Query 1 Scores (when run independently):
  • Document A: 1200
  • Document B: 1000
  • Document C: 300
  • Document D: 100

However, in the hybrid query, Document B does not appear in the top results, but Document C does, despite the heavily skewed weighting toward the first query (0.9998).

Pipeline Configuration:

{
  "phase_results_processors": [
    {
      "normalization-processor": {
        "combination": {
          "parameters": {
            "weights": [
              0.9998,
              0.0001,
              0.0001
            ]
          },
          "technique": "arithmetic_mean"
        },
        "normalization": {
          "technique": "min_max"
        }
      }
    }
  ]
}

Observations:

Essentially, even if Document C returns the highest possible scores from queries 2 and 3, it cannot score higher than Document B. Given this, it seems impossible for Document B to not appear in the final results, and Document C should not rank higher.

Question:

How is it possible for Document B to be excluded from the top results while Document C is included, given the heavily skewed weights and expected normalization?

Related component

Search:Relevance

Expected behavior

I would expect Document B to appear in the hybrid query search results no matter what, given the weight we've assigned to the first query.

[getsaurabh02]

Thanks @rohantilva. We will move this to the Neural Search repository.

[vibrantvarun]

Hi @rohantilva,thanks for creating this issue. Can you share some insights on the shape of search request? As per the example you shared in the PR description, B should not be missed. To further investigate the issue, I need to reproduce the issue on my end. It would be great if you share the steps to reproduce it.

[martin-gaievski]

Thank for for the feedback @rohantilva, I'd like to add one more ask to the previous request - can you please also share the exact requests used to obtain those raw scores mentioned in the header:

Documents: A, B, C, D
Query 1 Scores (when run independently):
Document A: 1200
Document B: 1000
Document C: 300
Document D: 100

[rohantilva]

@vibrantvarun @martin-gaievski Thanks for jumping on this. Some details below/attached.

Hybrid query request: this is an example of a similar looking request (I trimmed some of the extraneous fields to remove sensitive information). It's a hybrid query executing 2 queries (keyword match + knn semantic search), where the weight of the first query is set to 1 and the weight of the second query is set to 0 (note: I set these weights intentionally to illustrate the bug in full effect). Also note: I've removed the actual embeddings from the request (hence "vector": []).

hybrid_query.json

Screenshots of results: there are three "sections" in the screenshot, which show a couple things:

• first section: shows results from Opensearch when the keyword query (first subquery within the hybrid query) is executed individually
• second section: shows results from Opensearch when the knn/semantic query (second subquery) is executed individually
• third section: shows results from the hybrid query - you can clearly see that even though the weights of the queries are set to [1, 0], the hybrid query results do not match the results from the keyword query.

Btw, I am using the AWS managed Opensearch, version 2.15. I know there could be some drift between that and opensource version 2.15, so just wanted to point that out.

[vibrantvarun]

Hey @rohantilva I will look into this today.

[vibrantvarun]

Hey, @rohantilva I just verified that hybrid query works as expected. I have some concerns regarding the screenshot you shared. The subqueries when ran individually yields 5 results and hybrid query result count is 8, considering 2 duplicate documents which are part of result set from both subqueries. However, I can clearly see those duplicate results are not even present in the hybrid query result set? I think there is some other issue which is impacting this?

How come documents starting with

Checkissuing ...

are part of hybrid result when either of the subqueries do not return it?

[vibrantvarun]

Also I just wanted to confirm that you ran the subquery individually by running it in standalone manner, not under the hybrid clause.

[rohantilva]

@vibrantvarun

• yes, I executed the queries individually
• pretty much this bug relates to why the results from the hybrid query do not match the results from the subqueries - so your question of How come documents starting with 'Checkissuing...' are part of hybrid result when either of the subqueries do not return it? is exactly what my question is. I'm not sure why, when weights are set to [1, 0], the hybrid query results do not match the results from the first query when executed individually

The reason why the individual subqueries yielded 5 results was because I explicitly requested 5 documents for the subqueries (again, executed individually), and >5 documents for the hybrid query. If I request only 5 documents for the hybrid query, I get the same results, excluding the last 3 pictured in the screenshot I sent.

[vibrantvarun]

What is URL you guys are hitting for hybrid query? Can you share that.