feat(messages): polish pagination and filtering

[alex-mcgovern]

Builds on @yrobla's work in #1186 for pagination for /v1/workspaces/:name/messages

• fixes issue with duplicates in the response
• adds filtering by alert trigger_type
• attempts to generally refactor this code

We're not configured yet to run integration tests on workspace crud (but will be after #1170) so I've just manually verified the changes.

TODO:

• total count is broken, so in the UI there are buttons to nonexistent pages
• ability to filter/search by text is not implemented yet

Usage

Without filtering:

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=2" | jq '[.data[].chat_id]'
[
  "aa85b6a5-423a-4cd0-a27f-51088b5fe30c",
  "5ac437ff-6b1c-4aa4-b276-d43bce33f28e"
]

By `prompt_id

Filtering by a single ID:

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=2&filter_by_ids=aa85b6a5-423a-4cd0-a27f-51088b5fe30c" | jq '[.data[].chat_id]'
[
  "aa85b6a5-423a-4cd0-a27f-51088b5fe30c"
]

Filtering by multiple IDs:

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=2&filter_by_ids=aa85b6a5-423a-4cd0-a27f-51088b5fe30c&filter_by_ids=5ac437ff-6b1c-4aa4-b276-d43bce33f28e" | jq '[.data[].chat_id]'
[
  "aa85b6a5-423a-4cd0-a27f-51088b5fe30c",
  "5ac437ff-6b1c-4aa4-b276-d43bce33f28e"
]

Filtering by non-existent ID:

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=2&filter_by_ids=2F2FDF94-429D-45E4-9D61-DFA4DB3E6D2F" | jq
{
  "data": [],
  "limit": 2,
  "offset": 0,
  "total": 63
}

Filtering by non-existent ID:

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=2&filter_by_ids=2F2FDF94-429D-45E4-9D61-DFA4DB3E6D2F" | jq
{
  "data": [],
  "limit": 2,
  "offset": 0,
  "total": 63
}

By trigger_type

(Note: I only have secrets detected in this DB)

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=10&filter_
by_alert_trigger_types=codegate-context-retriever" | jq
{
  "data": [],
  "limit": 10,
  "offset": 0,
  "total": 63
}

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=10&filter_
by_alert_trigger_types=codegate-pii" | jq
{
  "data": [],
  "limit": 10,
  "offset": 0,
  "total": 63
}

curl -s "http://localhost:8989/api/v1/workspaces/default/messages?page=1&page_size=10&filter_
by_alert_trigger_types=codegate-secrets" | jq '[.data[].chat_id]'
[
  "131a028d-9e00-4565-8796-40cef7cd3b83",
  "00916b98-a3d3-424d-8e1f-d24da8c51afc",
  "3fae75e6-e32b-4d24-9856-f0b0cc88785e",
  "5c7d8bc9-716d-4106-aafc-890564cb212f",
  "cf50f1e5-4064-42e5-bf26-15249dc78e46",
  "0fdd2a27-821a-427f-8bf6-cca39bfde838",
  "5ac437ff-6b1c-4aa4-b276-d43bce33f28e",
  "aa85b6a5-423a-4cd0-a27f-51088b5fe30c",
  "f397d262-07f6-4724-ac94-f0d45a09eb01",
  "9aa71a60-ab7f-4d76-b9a3-13ec25d65661"
]

[alex-mcgovern]

I've been doing some manual testing — it's a bit of a mess.

Some of the issues I've found are:

• same record across different pages
• ordering issues (newer last) — this is due to using the loop to query more results until we have a full page

I think a lot of the issues originate from the behaviour in v1_processing.parse_messages_in_conversations where we group related prompts+outputs into a single conversation, or discard results in some cases where we can't parse a user request.

[alex-mcgovern]

@yrobla is taking over on this one with the approach mentioned here: #1223