[FEATURE] Add full entity information for AD monitor

[amitgalitz]

Is your feature request related to a problem?
When you create an OpenSearch Monitor on top of an Anomaly Detector, the alert output will generally contain structured information about the detected anomaly. However currently we only look for the max anomaly grade to display in that interval without too much additional information.

Our current search is for the sorted top entity and thats what we use for threshold comparison and what is available for users to pull into the context.

Currently also the default message pre-filling is:

Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
  - Trigger: {{ctx.trigger.name}}
  - Severity: {{ctx.trigger.severity}}
  - Period start: {{ctx.periodStart}} UTC
  - Period end: {{ctx.periodEnd}} UTC

What solution would you like?
I would like to improve the output capability of a monitor that is created on top of an anomaly detector so we can see at least 10-20 entities in a give interval that have an anomaly and also show the actual and expected value for that an anomaly along with any other information. The output would look like this:

We can add an additional toggle on the frontend to either show just the anomaly with the max grade

Anomaly 1:
Entity: {"name": "error_type", "value": "error16"}, Timestamp: <10:02pm>, actual value: 100, expected value: 10, anomaly grade: 0.7, anomaly confidence: 0.8
Anomaly 2:
Entity: {"name": "error_type", "value": "error17"}, Timestamp: <10:10pm>, actual value: 80, expected value: 11, anomaly grade: 0.7, anomaly confidence: 0.8
Anomaly 3:
Entity: {"name": "error_type", "value": "error18"}, Timestamp: <10:12pm>, actual value: 70, expected value: 12, anomaly grade: 0.7, anomaly confidence: 0.8

[kaituo]

Can you make the entity more reader friendly?

Say error_type:error18

This would make it easier to read when we have multiple categorical fields.

Also, the timestamp is a range, not a point.

[AWSHurneyt]

so we can see at least 10-20 entities in a give interval

I'm not too familiar with the AD plugin, but something to consider is how large the message size will be if it includes all 10-20 enties.

When we added the sample_documents variables to the ctx object (documentation link), we limited the number of sample docs to 10 (though we'd like to eventually enhance that limit to an action-level setting) as the plugins wouldn't inherently have access to the payload size restrictions of the destination webhook; different webhook providers may have different restrictions.

[krisfreedain]

Catch All Triage - 1 2