[ignoreFilterIfFieldNotInIndex] Build separate Timeline filters by index

[neodescis]

Description

When ignoreFilterIfFieldNotInIndex is enabled:

1. Parse the Timeline visualization expression for indexes
2. For each index in the expression, build a separate OpenSearch query from the current filters
3. Pass the filters for each index to the Timeline request handler
4. In the request handler, use the index-specific query instead of the single index-agnostic query

Issues Resolved

closes #6184

Testing the changes

1. Enable ignoreFilterIfFieldNotInIndex
2. Create dashboard with visualizations for more than one index
3. Create timeline visualization for one of those indexes and add it to the dashboard
4. Create filter that applies only to the index not referenced by the timeline visualization
5. The timeline visualization should no longer flatline, because the filter is no longer being applied to it

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[codecov]

Codecov Report

Attention: Patch coverage is 42.85714% with 4 lines in your changes missing coverage. Please review.

Project coverage is 67.55%. Comparing base (87627af) to head (8b0d331).

Files	Patch %	Lines
...r/series_functions/opensearch/lib/build_request.js	42.85%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6190      +/-   ##
==========================================
- Coverage   67.55%   67.55%   -0.01%     
==========================================
  Files        3469     3469              
  Lines       68479    68485       +6     
  Branches    11130    11130              
==========================================
+ Hits        46264    46266       +2     
- Misses      19512    19516       +4     
  Partials     2703     2703              

Flag	Coverage Δ
Linux_1	33.13% <42.85%> (+<0.01%)	⬆️
Linux_2	55.26% <ø> (ø)
Linux_3	45.31% <ø> (+0.01%)	⬆️
Linux_4	34.71% <ø> (ø)
Windows_1	33.15% <42.85%> (+<0.01%)	⬆️
Windows_2	55.21% <ø> (ø)
Windows_3	45.32% <ø> (ø)
Windows_4	34.71% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ananzh]

@neodescis Found an issue when I play with in. So in OSD, we can typically omit the explicit index= when specifying the index directly after the .es()/.os()/.opensearch() function. So the regex function const indexRegEx = /index=['"]?([\sA-z0-9_\-\*]*?)[^\sA-z0-9_\-\*]/g; will fail to capture the index.

For example, in the following expression, for opensearch_dashboards_sample_data_logs we have index= but for opensearch_dashboards_sample_data_ecommerce we omit

.es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes).holt(0.8, 0.2, 0.8, 2h).color(#c6c6c6).lines(10).label('Prediction Value'), .es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes).color(red).lines(1).label('Actual Value'), .es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes).lines(1).holt(0.8, 0.2, 0.8, 2h).subtract(.es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes)).abs().if(lt, 500, null, .es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes)).points(5,2,1.5).color(green).label('Anomaly Value').title('max types anomaly'),
.es(opensearch_dashboards_sample_data_ecommerce, metric=max:taxful_total_price).label('Maximum Price'),
.es(opensearch_dashboards_sample_data_ecommerce, metric=avg:taxful_total_price).movingaverage(window=7).label('7-day Avg Price Moving Average').color(green)

If we add a filter to opensearch_dashboards_sample_data_ecommerce, then in your handler, because the regex function will only capture one index pattern

and this will cause the filterByIndex not apply the filter to the opensearch_dashboards_sample_data_ecommerce

[ananzh]

@neodescis could you check my comment and fix the expression? Feel without index= is hard to capture. Maybe we could add /\.(es|elasticsearch|os|opensearch)\(\s*['"]?([\w\-\*]+)['"]?/ since.es, .elasticsearch, .os, .opensearch can help to capture and index name is always the first one. But I am worried about potential edge cases, like could index name not be the first string in the expression or if there is any special characters in index names.

[neodescis]

Thanks for looking at this, @ananzh. I appreciate the analysis. I didn't know that index= was optional. I understand the issue, and I will see what I can do.

[neodescis]

@neodescis Found an issue when I play with in. So in OSD, we can typically omit the explicit index= when specifying the index directly after the .es()/.os()/.opensearch() function. So the regex function const indexRegEx = /index=['"]?([\sA-z0-9_\-\*]*?)[^\sA-z0-9_\-\*]/g; will fail to capture the index.

For example, in the following expression, for opensearch_dashboards_sample_data_logs we have index= but for opensearch_dashboards_sample_data_ecommerce we omit

.es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes).holt(0.8, 0.2, 0.8, 2h).color(#c6c6c6).lines(10).label('Prediction Value'), .es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes).color(red).lines(1).label('Actual Value'), .es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes).lines(1).holt(0.8, 0.2, 0.8, 2h).subtract(.es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes)).abs().if(lt, 500, null, .es(index=opensearch_dashboards_sample_data_logs,metric=max:bytes)).points(5,2,1.5).color(green).label('Anomaly Value').title('max types anomaly'),
.es(opensearch_dashboards_sample_data_ecommerce, metric=max:taxful_total_price).label('Maximum Price'),
.es(opensearch_dashboards_sample_data_ecommerce, metric=avg:taxful_total_price).movingaverage(window=7).label('7-day Avg Price Moving Average').color(green)

I've done some digging, and I've determined that the .os() function does not function as you stated. When omitting an argument's name, it seems the argument must be at a particular place in the list of arguments. In the case of the index argument, it must be supplied as the 4th argument to .os(). The first argument is for "query". I have verified this is how it works, as I get completely different results when using .os(indexname) and .os(index=indexname). The order is determined by repositionArguments() and Datasource's config args list. Furthermore, I found that this is documented in Kibana, though I cannot find the same in the documentation for OpenSearch Dashboards.

That said, I do have a solution that supports the unnamed argument as the 4th parameter. I did end up changing the logic a fair amount to do so. I will get it pushed to this pull request soon.

[neodescis]

Any chance anyone can look at this? @ananzh ?