[Feature Request] Expose result window being exceeded as a metric

[sander-bol]

Is your feature request related to a problem? Please describe

During a recent incident, we found that an important search query was exceeding the result window (index.max_result_window), leading to unexpected truncation of results.

In our interaction with AWS Support we suggested this would be a great metric to track + include in the default dashboards, as silently not returning some results during the matching phase represents a sudden shift in behaviour triggered by the passing of a threshold. This shift might come as a surprise to developers using OpenSearch. Support asked we file this as a ticket against the OpenSearch project, so here we are.

Describe the solution you'd like

A metric being reported to CloudWatch Metrics whenever a search query is truncated because the amount of documents it gathered during the matching phase was too large. Seeing this metric jump from 0 to "significant" in the dashboard would have given us an early clue that our queries were doing something naughty and needed finetuning.

Related component

Search:Resiliency

Describe alternatives you've considered

With total hits tracking enabled, we could inspect whether the hits.total.relation === 'gte' in the responses, and track this client-side. Now that we know about this issue, we can do this. The problem was that our team was not aware of this limitation in the first place, and therefor did not put this measure in place.

Alternatively, we could do this in the SDK, and emit a warning in case the above is detected.

Additional context

Based on an AWS case. Case number can be provided, if that's helpful. Not sure if case numbers are considered sensitive information.

[sandeshkr419]

Thanks @sander-bol for bringing it here.
This is related to AWS OpenSearch Service and not OpenSearch, so you may want to reach out to AWS support directly for this.

[sander-bol]

Fair enough! That was my own assessment as well, but support asked me to file the request here :) I'll report your feedback back. Thanks!