Outside-in compression: scaling MOQT for high capacity ingest

[wilaw]

Forgive the Pied Piper Reference. The purpose of this issue is to initiate a design discussion around making MOQT efficient for many-to-one distribution.

Much of the preceding design of MOQT assumes either few-to-few (e.g. video conferencing) or one-to-many (live sports) distribution. We can leverage caching to enable scalable egress, combined with a fan-out architecture for non-cacheable real-time content.

But what about the use-case of "fan-in" - millions of sensors providing real-time telemetry to a single monitoring point? In this case, caching does no good from a scale perspective, as the content is only ever going to one subscriber. The fan-out architecture works in reverse to collapse all upstream data onto downstream pipes.

What features do we need in MOQT to make high capacity ingest efficient?:

1. We want a way for a publisher to signal to all relays that a track should not be cached. Currently the MAX CACHE DURATION Parameter can only be present in a SUBSCRIBE_OK, FETCH_OK or TRACK_STATUS message. It cannot be sent by a publisher. Setting the DELIVERY TIMEOUT Parameter to zero is not equivalent to do-not-cache.
2. Keep-alives for long running sessions. Imagine a security camera that sends sparse data once every 5min. We can rely on application level pings to keep the connection alive, or should web build in a mechanism into the protocol to keep connections alive (with auth protection so untrusted clients cannot keep connections open forever).
3. Can the central collectors use the relays as configurable filters? What if we do want to leverage the cache to "batch" the collection of telemetry? For example, sensors output temperature data every second, but the collector only wants to retrieve the last 5mi of data every 5min. We could create a FORWARD_FILTER Parameter which the subscriber can send during subscription, which specifies a key parameter and a threshold value. For example, I subscribe only to objects for which the "foo" parameter is > 10.
4. Much telemetry data is timeline-based. Can we have a standard mode in which a track is identified as a timeline and make sure that works efficiently for our major actions (SUBSCRIBE, FETCH etc). Is setting each group number to be a millisecond epoch timestamp sufficiently robust?
5. Publisher authentication and authorization: is our current token scheme sufficient for high-capacity ingest?
6. End-to-end encryption: we allow the payload to be encrypted, but is this sufficient for sensitive IoT data at city-wide scale?

What else might be important for high-capacity ingest that we can address now before we fail at scale?

Cheers
Will

[LPardue]

For 2, my own experiences on keep alive in various shapes would lean towards doing nothing in the protocol itself and leave this for the application layer to drive it (or alternatively, implementation-defined choices)

QUIC negotaties connection idle times and silent close for various reasons, including saving batteries. Even in cases where there are NATS, the nature of QUIC lends itself to "quiet" connections.

Furthermore, session resumption and possibly 0-RTT is a general solution to avoid needing to keep active state. I don't know off hand how well that works for WebTransport and MoQ given all the bootstrapping needed. I suspect it might not work at all but happy to hear from other people that have though about it harder.

[suhasHere]

I am interested in exploring further on points 3/4 above too

[afrind]

In general, I'd like to push as much of this as possible to either applications of moqt or extensions, rather than the base protocol.

1. • MAX_CACHE_DURATION should be allowed in PUBLISH, see Indicate which params can appear PUBLISH* messages #1071
2. See Timeout behavior for publisher with no subscribers #805, which is related. I'm mostly inclined to +1 Lucas and let applications solve timeouts at the QUIC layer, though I wonder if that works for browsers.
3. Relays can only filter or batch objects on moqt metadata - group ID, object ID, etc. Maybe filter/batch are a good extension candidates; if you don't understand the extension, it doesn't break it's just less performant.
4. I don't think any changes are required to moqt for applications that want to use timestamps as group IDs?
5. I'm not an auth expert - do you foresee challenges with the current scheme?
6. I'm not sure what you are driving at -- do you want to preclude e2e encryption for these cases to get more relay functionality? If so, then it sounds like you are designing a republisher rather than a relay, since the opacity of payloads to relays is fundamental to moqt.

[wilaw]

So for [3] I propose a new FILTER parameter on SUBSCRIBE. It works numerically, because the majority of filter operations are either natively numeric, or can be cast to be numeric. It takes 3 arguments

1. The operator : one of >, >=, <, <=, =, !=o
2. The operand: one of groupID, object ID, or a extension header ID.
3. The threshold value

The extension ID must point at an extension header with an even type i.e one with a single integer value. It allows applications to supply custom data in an extension header and then filter on that.

Examples:

1. Only forward me the first object in every group
   FILTER ( =, object ID, 0 )

2. Only forward me Group IDs larger than 1752138462
   FILTER ( >, group ID, 1752138462)

3. Only forward me security camera groups where the activity level is greater than or equal to 25%. The activity level is added to each object as the custom extension with a type of 400.
   FILTER (>=, extension 400, 25)

4. Only forward me temperature data from zip code 94925. The zip code is stored as the custom extension header 1444.
   FILTER (=, extension 1444, 94925)

5. Only forward me log data from the media player with subscriber ID 847362511. The subscriber ID is sent in extension header 768 by the media player.
   FILTER (=, extension 768, 847362511)

One detail to work out is where the filtering happens. Ideally we want it to happen at the edge, to prevent unwanted traffic from moving through the network. One consequence of this is that the subscriptions with different filters MUST NOT be aggregated. Imagine a publisher producing a track and three subscribers on the other side of the network, issuing subscriptions with filters, one for groups < 1000, the other for groups > 1000 and the third for groups > 2000. The edge relay will receive 3 subscriptions for the same track. The first two are mutually exclusive, however the second two, for groups numbers > 2000, result in duplicate traffic being forwarded. This duplication is acceptable, as attempting to decide which subscriptions are children of others would be fragile and unstable if a parent subscription suddenly disappears. Additionally in practice subscribers within a real-world application would likely apply non-overlapping filters.

[afrind]

@wilaw is this proposal different from #441

[wilaw]

@wilaw is this proposal different from #441

Hmm, forgot I wrote #441 :). So yes, #441 proposes only an Object filter (which I still think is valuable). This current proposal allows you to filter on object IDs, but also on groups IDs as well as extension headers, which I think has greater utility. To combine the two, we could use the range convention in place of the threshold value and also remove the operand, as its implicit in how we define our range.

So FILTERv2 has two arguments

1. The operand: one of groupID, object ID, or a extension header ID.
2. An array of values, with the following meaning

[0] - only send items with a value of 0
[0,5] - only send items with an value of 0 or 5.
[0..5] - only send items in the range 0 to 5 inclusive.
[0..2, 16..20] - only send items in the range 0 to 5 inclusive, or 16..20 inclusive.
[5..] - only send items >= 5
[..5] - only send items <= 5

1. Only forward me the first object in every group
   FILTER ( object ID, [0] )

2. Only forward me Group IDs larger than 1752138462
   FILTER ( group ID, [1752138462..])

3. Only forward me security camera groups where the activity level is greater than or equal to 25%. The activity level is added to each object as the custom extension with a type of 400.
   FILTER (extension 400, [25..])

4. Only forward me temperature data from zip codes 94925 and 23554. The zip codes are stored as the custom extension header 1444.
   FILTER (extension 1444, [94925,23554])

5. Only forward me log data from the media player with subscriber ID 847362511, 657372 and 8251132 The subscriber ID is sent in extension header 768 by the media player.
   FILTER (extension 768, [847362511,657372,8251132])

[ianswett]

+1 to @LPardue comment that I don't want MoQ to do anything about keepalives.

On [3], there's also the topic of batching. I do think that might be an optimization worth considering at the transport layer, similar to Nagle's algorithm. Sending <100 byte payloads immediately incurs quite a bit of overhead. Possibly the reverse of a delivery timeout? For example batch_timeout = 100ms could mean batch up to 100ms of data before sending.

[wilaw]

+1 to the idea of a reverse delivery timeout for very small payloads that are not latency sensitive.