Skip to content

Commit 8706f3c

Browse files
author
Philip de Nier
committed
adr: Add ADR for updating webhook events and filters
1 parent c83de83 commit 8706f3c

File tree

2 files changed

+168
-0
lines changed

2 files changed

+168
-0
lines changed

‎docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ For more information on how we use ADRs, see [here](./adr/README.md).
3434
| [0022](./adr/0022-flow-bit-rate-properties.md) | Definitions for Flow bit rate properties and additional properties |
3535
| [0023](./adr/0023-filter-segment-get-urls.md) | Add query option to filter Flow Segment `get_urls` by `label` |
3636
| [0025](./adr/0025-flow-property-updates.md) | Options for updating Flow properties |
37+
| [0026](./adr/0026-updated-webhook-events-and-filters.md) | Updates to the webhook event structures and filters |
3738

3839
\* Note: ADR 0004a was the unintended result of a number clash in the early development of TAMS which wasn't caught before publication
3940

‎docs/adr/0026-updated-webhook-events-and-filters.md

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
status: "proposed"
3+
---
4+
# Updated Webhook Events and Filters
5+
6+
## Context and Problem Statement
7+
8+
TAMS includes an endpoint to register webhooks for receiving Flow, Flow Segment and Source events.
9+
The API specification for the webhook was intended to provide an example of a basic webhook with filtering capabilities.
10+
It was assumed that developers would implement and use the basic webhook that is available in implementations and that later iterations of the TAMS specification would incorporate changes based on learnings and to better match real-world requirements.
11+
12+
The webhook as specified was found to be useful to users of the current set of TAMS implementations, but they have asked for some updates.
13+
This ADR proposes some updates to the webhook specification in TAMS.
14+
15+
## Decision Drivers
16+
17+
* Users have asked to filter on Flow Collections such that all events associated with Flows in the collection are included.
18+
This avoids needing to query the list of Flows in the collection when registerering the webhook.
19+
This allows new Flows to be added to the collection without requiring a new webhook with updated filters to be registered, avoiding disruption to the event streams.
20+
* Users have asked to be able to filter Flow and Flow Segment events by Source ID.
21+
* Similar to filtering on Flow Collection, it would be useful to be able to filter Flow and Flow Segment events by Source Collections as well.
22+
* Users have asked to have the Flow Segment events contain the same information that is present in the API endpoint response.
23+
This avoids needing additional API queries to get the missing information.
24+
* Users have asked to be able to filter the `get_urls` in Flow Segments in the same way that is currently possible in API endpoint requests.
25+
26+
## Considered Options
27+
28+
* Option 1: Leave the webhook specification unchanged
29+
* Option 2a: Make the event filter and complete Flow Segment information requested by users
30+
* Option 2b: Extend option 2a to add support for filtering by Source Collections
31+
* Option 3: Make the event transformation updates requested by users
32+
33+
## Decision Outcome
34+
35+
Chosen option: Option 2b, because that provides most of what users have explicitly requested.
36+
Also, filtering on Source Collections is likely to be needed in systems that operate on Sources.
37+
The assumption is that the extra queries to get the Flow and Source Collection IDs is worth the benefits of having more advanced event filtering on collections.
38+
39+
Option 3 requires additional event processing to transform events for webhook endpoints.
40+
It is unclear whether the additional processing warrants specification at this time.
41+
42+
There may be other ways to achieve a requirement to include or not include presigned `get_urls` in Flow Segments.
43+
44+
* A TAMS implementation may include a configure option to enable inclusion of presigned URLs.
45+
* A Flow tag may be used to indicate where Flow Segment events should include presigned URLs.
46+
An application that requires presigned URLs for performance reasons (e.g. to avoid an API request) can decide to enable it for the Flows it creates.
47+
A permissions system sitting in front of a TAMS could decide whether users can enable presigned URLs for Flows using the tag.
48+
49+
### Implementation
50+
51+
TBD - in this PR.
52+
53+
## Pros and Cons of the Options
54+
55+
### Option 1: Leave the webhook specification unchanged
56+
57+
* Good, because it avoids making breaking changes to the specification
58+
* Good, because it has been proven to work using existing implementations
59+
* Bad, because it doesn't fulfill known user requirements which may lead to future divergence in implementations to meet those requirements
60+
61+
### Option 2a: Make the event filter and complete Flow Segment information requested by users
62+
63+
The `flow_ids` and `source_ids` filter option in the webhook registration now refer to just the Flow and/or Source entity directly associated with the event.
64+
The Flow and Flow Segment events have a Flow (Flow `id` property) and Source (Flow `source_id` property) directly associated with it.
65+
Note that the Flows and Sources could be collection Flows and Sources.
66+
67+
The `flow_collected_by_ids` filter option in the webhook registration refers to the Flow Collections referenced through the Flow `collected_by` property.
68+
69+
The `flows/segments_added` event is changed to add a `segments` property and remove the `timerange` property.
70+
The `segments` property that contains a list of Flow Segment data structures that contain the same information as returned by the segments API endpoint.
71+
Note that a TAMS implementation may limit the information returned in events, e.g. omit the presigned `get_urls` for security reasons.
72+
73+
The `timerange` property is removed from the `flows/segments_added` event as that information is now available in the union of `timerange` properties in the `segments`.
74+
75+
* Good, because it fulfills the known user requirements
76+
* Good, because it extends the current specifiction rather than being a complete overhaul
77+
* Neutral, because it requires an extra query to get the Flow Collection IDs (Flow's `collected_by` property) associated with the Flow
78+
* Neutral, because more information is included in Flow Segment events which may increase the transmission and processing costs
79+
* Neutral, because including presigned URLs in the Flow Segment information may compromise security as the webhook receiver is not directly involved in the authentication process
80+
81+
### Option 2b: Extend option 2a to add support for filtering by Source Collections
82+
83+
A system that prefers dealing with Sources rather than Flows would benefit from the ability to filter events based on Source Collections.
84+
A filter based on Source Collections would also support additions to the underlying Flows without requiring the webhook to be re-registered with an updated list of Flows to filter on.
85+
86+
The `source_collected_by_ids` filter option is added to the webhook registration.
87+
The property lists the Source Collections that collect the Source in the `collected_by` property.
88+
The Source in the Flow and Flow Segment event case is the Source referenced by the Flow's `source_id` property.
89+
90+
* Good, because it provides a filtering capability at the Source level
91+
* Good, because new Flows or Sources in a collection wouldn't require changes to the webhook registration
92+
* Neutral, because it requires an extra query to get the Source Collection IDs (Source's `collected_by` property) associated with the Flow's Source
93+
94+
### Option 3: Make the event transformation updates requested by users
95+
96+
The webhook options could be extended with a `accept_get_urls` property that functions the same way as the query parameter on the segment access API endpoint.
97+
This requires the event processing pipeline to transform the `segments` in the Flow Segment events by filtering the Flow Segment `get_urls` using the value set in `accept_get_urls`.
98+
99+
A `segment_timerange_only` webhook option could be used to reduce the size of Flow Segment events by returning just a `timerange` rather than the `segments`.
100+
101+
* Good, because it provides some control to webhook users of how much information is sent, which may help reduce transmission and processing costs.
102+
* Bad, because it requires additional processing in the event transmission pipeline to transform the events
103+
104+
## More Information
105+
106+
### Event filter logic
107+
108+
The `events` webhook property lists the event types that are passed through the filter.
109+
The `events` cannot be empty as an empty events is used to delete the webhook.
110+
111+
If `flow_ids` is set then Flow and Flow Segment events are only passed through if they refer directly to a Flow in the `flow_ids` list.
112+
The `flows_ids` has no filter effect if not set or for Source events.
113+
114+
If `source_ids` is set then Flow, Flow Segment and Source events are only passed through if they refer directly to a Source in the `source_ids` list.
115+
The `source_ids` has no filter effect if not set.
116+
117+
If `flow_collected_by_ids` is set then Flow and Flow Segment events are only passed through if they are `collected_by` a Flow Collection in the `flow_collected_by_ids` list.
118+
The `flow_collected_by_ids` has no filter effect if not set or for Source events.
119+
120+
If `source_collected_by_ids` is set then Flow, Flow Segment and Source events are only passed through if they collected by a Source Collection in the `source_collected_by_ids` list.
121+
The Flow is collected by a Source Collection if the Source referenced by the Flow is collected by the Source Collection.
122+
The `source_collected_by_ids` has no filter effect if not set.
123+
124+
### Some use cases and associated webhook options
125+
126+
Here are some general use cases along with the webhook options:
127+
128+
* Allow users to filter which events types to receive
129+
* Set the `events` property to the list of event types to receive
130+
* Allow users to filter Flow and / or Flow Segment events for a specific Flow
131+
* Set the `events` to include the Flow and / or Flow Segment event types as required
132+
* Add the Flow ID to the `flow_ids` property
133+
* Allow users to filter Flow and / or Flow Segment events for all Flows of a Source
134+
* Set the `events` to include the Flow and / or Flow Segment event types as required
135+
* Add the Source ID to the `source_ids` property
136+
* Allow users to filter Flow and / or Flow Segment events for all Flows collected by a Flow Collection
137+
* Set the `events` to include the Flow and / or Flow Segment event types as required
138+
* Add the Flow Collection ID to the `flow_collected_by_ids` property
139+
* Allow users to filter Flow and / or Flow Segment events for a Flow Collection as well as all Flows collected by that Flow Collection
140+
* Set the `events` to include the Flow and / or Flow Segment event types as required
141+
* Add the Flow Collection ID to the `flow_ids` property
142+
* Add the Flow Collection ID to the `flow_collected_by_ids` property
143+
* Allow users to filter Source events for a Source
144+
* Set the `events` to include the Source event types as required
145+
* Add the Source ID to the `source_ids` property
146+
* Allow users to filter Source events for all Sources collected by a Source Collection
147+
* Set the `events` to include the Source event types as required
148+
* Add the Source ID to the `source_collected_by_ids` property
149+
* Allow users to filter Source events for all Sources for a Source Collection as well as all Sources collected by that Source Collection
150+
* Set the `events` to include the Source event types as required
151+
* Add the Source Collection ID to the `source_ids` property
152+
* Add the Source ID to the `source_collected_by_ids` property
153+
* Allow users to filter Flow and / or Flow Segment events for all Flows that have the Source collected by a Source Collection
154+
* Set the `events` to include the Flow and / or Flow Segment event types as required
155+
* Add the Source Collection ID to the `source_collected_by_ids` property
156+
* Allow users to filter events for multiple Flows
157+
* Set the `events` to include Flow and / or Flow Segment event types as required
158+
* Add the Flow IDs to the `flow_ids` property
159+
* Allow users to filter events for multiple Flow Collections
160+
* Set the `events` to include Flow and / or Flow Segment event types as required
161+
* Add the Flow Collection IDs to the `flow_collected_by_ids` property
162+
* Allow users to filter events for multiple Sources
163+
* Set the `events` to include the event types as required
164+
* Add the Source IDs to the `source_ids` property
165+
* Allow users to filter events for multiple Source Collections
166+
* Set the `events` to include the event types as required
167+
* Add the Source Collection IDs to the `source_collected_by_ids` property

0 commit comments

Comments
 (0)