How to handle packets encoded with VP8/VP9 or AV1?

[Xoffio]

I created a simple program that receives packets from UltraGrid, removes the first 36 bytes (12 bytes for RTP + 24 bytes for the UltraGrid video payload), and then forwards the modified packets to ffmpeg. When using H.264, ffmpeg is able to process the stream without any issues.

However, when I try the same approach with VP8, VP9, or AV1, ffmpeg fails to read the stream.

Is there anything I can do to the packets to make ffmpeg recognize them?

Any help would be greatly appreciated.
Thanks!

[MartinPulec]

Hi, I think that it works with H.264 because the frames are both self contained and self-delimiting (if concatenated, it creates Annex-B format).

I've played played a bit around and it seems to me that AV1 might work? Provided that the below is the correct simulation of the situation. I've tried:

uv -t testcard -c lavc:e=libaom-av1 --record=rec # press Ctrl-C after few secs
cd rec
cat *.av1 > out.av1
ffplay out.av1

However, it doesn't work the same way for VP9, which perhaps means that it needs some container. What if you add the frames to a Duck IVF container? I've created a simple muxer mux_ivf.c.txt that seem to work on recorded files. Usage is simple: mux_ivf <width> <height> <fps> *.vp9 > out.vp9.ivf (replacing the line 3 above).

Using it at packet level may be, however, slightly challenging:

1. number of frames would perhaps need to be set to 0 (because you might not know)
2. you'll need to buffer the packets to assemble whole frames (because each frame in the IVF has own header) - this can be done by the detecting RTP M-bit (indicating last packet of a frame, AFAIK 9th bit of RTP header)
3. if you don't know width/height, you might parse it from UG headers, but setting to 0 may work (it may be deduced from the compressed data), fps seemingly need to be inevitably set, but you can pick there 30 or so

[Xoffio]

@MartinPulec Thank you so much for taking the time!

I have tested your av1 test and it works but it doesn't work when doing UDP... I am trying something like this:

uv -t testcard -c lavc:e=libaom-av1 -P 6004 192.168.8.8
topoxo -s 0.0.0.0:6004 -t 192.168.8.8:6006 --insecure -p udp --plugin ug_h264
ffplay udp://192.168.8.8:6006

Note: topoxo is the program I made that removes the 32bytes. I can make the project public if you want to test it. This works with H.264.

Between VP8, VP9, and AV1, which one should be the easier to work with in your experience?

2. you'll need to buffer the packets to assemble whole frames (because each frame in the IVF has own header) - this can be done by the detecting RTP M-bit (indicating last packet of a frame, AFAIK 9th bit of RTP header)

The next is a sample of what I am getting.. How do you know which packet is part of which frame?

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 0, payload_type: 20, sequence_number: 11500, timestamp: 1509751420, ssrc: 1082604833 }  
UgVideoPayload { offset: 4308, length: 11617, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 0, payload_type: 20, sequence_number: 11501, timestamp: 1509751420, ssrc: 1082604833 }  
UgVideoPayload { offset: 5744, length: 11617, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 0, payload_type: 20, sequence_number: 11502, timestamp: 1509751420, ssrc: 1082604833 }  
UgVideoPayload { offset: 7180, length: 11617, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 0, payload_type: 20, sequence_number: 11503, timestamp: 1509751420, ssrc: 1082604833 }  
UgVideoPayload { offset: 8616, length: 11617, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 0, payload_type: 20, sequence_number: 11504, timestamp: 1509751420, ssrc: 1082604833 }  
UgVideoPayload { offset: 10052, length: 11617, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 1, payload_type: 20, sequence_number: 11505, timestamp: 1509751420, ssrc: 1082604833 }  
UgVideoPayload { offset: 11488, length: 11617, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 1, payload_type: 20, sequence_number: 11506, timestamp: 1509755020, ssrc: 1082604833 }
UgVideoPayload { offset: 0, length: 238, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 1, payload_type: 20, sequence_number: 11507, timestamp: 1509758620, ssrc: 1082604833 }
UgVideoPayload { offset: 0, length: 218, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

RtpPacket { version: 2, padding: 0, extension: 0, csrc_count: 0, marker: 1, payload_type: 20, sequence_number: 11508, timestamp: 1509762220, ssrc: 1082604833 }
UgVideoPayload { offset: 0, length: 213, horizontal_res: 640, vertical_res: 480, four_cc: 1635135537 }

[MartinPulec]

I have tested your av1 test and it works but it doesn't work when doing UDP...

Interesting but possible. I'd expect it to behave in the same way with the same FFmpeg. Maybe it can take different decode route programmatically than ffmpeg run from command-line - eg. passing the data directly to libavcodec vs through libavformat demuxer.

The next is a sample of what I am getting.. How do you know which packet is part of which frame?

It is very simple - marker: 1 means last packet of the frame. In your snippet, obviously the first frame is an I-frame, because it spans 6 packets while the later ones just one small.

Alternatively you can watch timestamp change - all packets from one frame have the same one. A difference is that will detect missing frame with M-bit, but you'll need to wait until start of the next frame. Otherwise it should be the same.

[Xoffio]

I decided to use AV1 and wrapped it in an IVF container as you suggested. Now it's working! Thanks, @MartinPulec!