ffmpeg-cli & Safari WebCodec decoder requires additional frames enqueued for some videos in order to produce frames + spurious errors

[Wumpf]

Some video material, for instance https://github.com/user-attachments/assets/726c1803-9b4b-4dfb-a6ee-29a23a2353a9
(source https://huggingface.co/datasets/pollen-robotics/apple_storage) requires us to enqueue up to 16 more video samples than what expect.
Practically what happens is that as we keep queuing in GOPs/samples, we eventually get frames from ffmpeg, but with a 16 frame delay - i.e. we never get the frames back that we'd expect under normal circumstances.

Note that our WebCodec implementation has no issues with this video in Firefox & Chrome. However, the sample issue arises on Safari!

As of [TODO PR LINK] we now make sure that we have at least 16 samples enqueued for the ffmpeg cli decoder at any point in time to work around this.

---

Furthermore, even with a more aggressive enqueuing strategy that takes this into account we are seeing spurious errors like these which recover either after rendering more frames.

It also has been observed that when viewing several of these videos at once errors only disappear after seeking:

---

As of now it's unclear what is different about this video material than others we tried in the past. The video example linked above is an "easy" case: It has only GOPs with length 2, each consisting of an IDR frame (marked as such both in the NAL header & mp4 container) and a P frame. Did not find any special properties in the SPS (also looked into parsing the PPS but didn't succeed).

Already experimented among other things with....

• various flags to ffmpeg to reduce latency further or override timings
• insert EndStream/EndStream NAL units at the end of GOPs
• inserting sleeps to feed ffmpeg slower
• insert fill data on stdin
• convert to h264 and back to mp4 with ffmpeg - both play fine and no change if done with copy flags to avoid re-encoding (this should exclude most mp4 specific issues, but note that the AVCC box holding PPS & SPS should be completely replicated)
• extract single frames to image with ffmpeg works fine
• other video players handle this just fine (tried Windows Media Player, VLC, Firefox, Chrome)

Things we should still look into:

• try to build our own "problematic video" in order to understand better what causes these issues (notably the video was chunked from a larger video)
• read-up more on WebCodec implementation in Firefox & Chrome -> prototype with directly interfacing libav
• ?