fix: Data levels issues

[github-actions]

Problem

I'm a little confused about this. Not sure what's going on.

If data levels are set to auto (interpreted as "full" I guess), proxies are still being exported with limited range and they appear to have much more contrast; crushed blacks and blown-out whites. It doesn't happen for proxy media generated internally.

Add support

If we can get ffmpeg to encode with the correct data levels, we still need to know what the input levels are. Since clip properties returned through the api can have levels set to auto, we'll have to alert the user that they should probably set the clips explicitly to "full" or "video".

ProRes support hurdles

There's also a concern about using ProRes. ProRes doesn't have a flag to tell the player how the colour data should be interpreted. I believe Resolve 18 allows for ProRes proxies, so maybe we should see how they're handling it? Some additional custom metadata? Some internal forced level interpretation? A new implementation for 18 altogether?

If special handling is required for some codecs like ProRes, we may need to add some data-levels settings to the proposed render-presets. Something like:

- proxy_settings:
    - data_levels: ["inherit", "video", "full"] 
    - filters:
        - data_levels_whitelist: ["full", "video", "auto"]

Additional Resources

• https://superuser.com/questions/1405177/ffmpeg-how-to-handle-color-range-conversions#1405333
• https://www.reddit.com/r/ffmpeg/comments/a1b6vv/how_to_add_data_level_in_the_output_metadata/

closes #200

[in03]

A lot of research going into this issue when I have the time!

Check out:
https://www.thepostprocess.com/2019/09/24/how-to-deal-with-levels-full-vs-video/

So far it seems like the issue is inconsistent across projects. It's not an 8-bit / 10-bit issue like I originally thought it was. My bad. Internally generated 8-bit footage data levels interpreted correctly, but superwhites and superblacks clamped. It appears to be that proxies generated through ffmpeg are being given video-levels when they should be full. Likely one of our cameras has had its levels set to full (0-255) instead of (16-235) so the ranges written differ from the metadata and that's why it's inconsistent? Just a guess. Will continue testing and see.

[in03]

TL;DR

Alright! Finally figured it out....
Looks like Resolve is interpreting input files correctly. If a camera is set to use full range levels (0-255) with 10-bit, colours are interpreted correctly by Resolve, if the levels are set to limited (16-235) the colours are interpreted incorrectly. This isn't Resolve's fault, it's reading the metadata correctly, the metadata is just not helping.

Ffmpeg on the other hand is not interpreting the input levels correctly and sets them to limited every time. What also doesn't help is that Resolve will only interpret the levels of proxies as limited. To get around this we can set input levels manually using color_range as reported by ffprobe:

• color_range: pc is equivalent to in_range: full
• color_range: tv is equivalent to in_range:limited

Finally, out_range should always be set to limited to match Resolve's expectations of proxies.
So for 10-bit full levels h2.65 we can set, -vf scale=in_range=full:out_range=limited.
If our source file levels were set incorrectly in metadata or interpreted incorrectly by Resolve for some reason, we can get Proxima to render proxies matching data levels as set in Resolve's clip attributes.

The inconsistency I was experiencing was due to two cameras shooting with different levels settings. The one set to limited levels always matched the limited proxies. Bit obvi in hindsight.

Testing

I tested so many codecs... Resolve, Adobe Media Encoder, raw ffmpeg, Shutter Encoder, etc, greped and diffed metadata and colour metadata was identical between files that were interpreted differently by Resolve. A very confusing time!

Encode

This is essentially the command I was testing with Proxima under the hood. Using ProRes 422 Proxy instead of DNxHR SQ to ensure 10-bit and low bitrate. I was concerned that ProRes's Quicktime container can't contain colour metadata and so maybe incorrect interpretation was due to that, but 10/12-bit DNxHR variants were just as bad:

ffmpeg -y -hide_banner -stats -i "Desktop\infile.mov" -c:v dnxhd -profile:v dnxhr_sq -vf scale=-2:720 -vf scale=in_range=full:out_range=full -vf format=yuv422p -c:a pcm_s16le -ar 48000 -movflags +write_colr "Desktop\outfile.mov"

Check bits per Raw Sample

Just to ensure the bit-depth was being set correctly I tested files with this command:

ffprobe -loglevel panic -show_entries stream=bits_per_raw_sample -select_streams v "videofile.mov"

Check Y Min Max

This checks the minimum and maximum luminance levels for YUV. This was when I realised real-world levels were different between files that otherwise had identical metadata.

ffprobe -f lavfi movie="Desktop/infile.mov",signalstats -show_entries frame_tags=lavfi.signalstats.YMAX,lavfi.signalstats.YMIN

Check Colour

I ran this on each file to check color_range, color_space and color_transfer were as expected.

ffprobe -v error -show_streams "Desktop/outfile.mov" | grep color_

Results

Variant	Additional args	Bit Depth	Chroma SS	Observed Colour Range	Colour space, transfer	Colour range	Display Correct ?
Resolve DNxHR SQ	None	8-bits	4:2:2	15-245	bt709	TV	Yes
Ffmpeg ProRes 422 Proxy	None	10-bits	4:2:2	4-1019	bt709	TV	No
Ffmpeg DNxHR SQ	None	8-bits	4:2:2	0-255	bt709	PC	No
Adobe ProRes 422 Proxy	None	10-bits	4:2:2	4-1019	bt709	TV	Yes
Dan's Panasonic H.265	None	10-bits	4:2:0	0-1023	bt709	PC	Yes
Dave's Panasonic H.265	None	10-bits	4:2:0	0-1023	bt709	TV	No
Ffmpeg ProRes 422 Proxy (full, limited)	in_range=full, out_range=limited	10-bits	4:2:2	5-1013	bt709	TV	Yes

After adding that last entry to the table, I figured out the relationship ffmpeg video filter settings in_range and out_range have with color_range metadata reported. bit depth and chroma subsampling have no pull on levels interpretation and bt709 is consistent across the board. So I started another two tables:

Dan's Camera - colour_level = PC

Variant	In Range	Out Range	Display Correct?	Group
Ffmpeg ProRes 422 Proxy	Auto	Auto	No	1
Ffmpeg ProRes 422 Proxy	Auto	Limited	No	1
Ffmpeg ProRes 422 Proxy	Full	Limited	Yes	0
Ffmpeg ProRes 422 Proxy	Full	Full	No	1
Ffmpeg ProRes 422 Proxy	Limited	Auto	No	1
Ffmpeg ProRes 422 Proxy	Limited	Full	No	2
Ffmpeg ProRes 422 Proxy	Limited	Limited	No	1

Dave's Camera - colour_level = TV

Variant	In Range	Out Range	Display Correct?	Group
Ffmpeg ProRes 422 Proxy	Auto	Auto	No	1
Ffmpeg ProRes 422 Proxy	Auto	Limited	No	1
Ffmpeg ProRes 422 Proxy	Full	Limited	Yes	0
Ffmpeg ProRes 422 Proxy	Full	Full	No	1
Ffmpeg ProRes 422 Proxy	Limited	Auto	No	1
Ffmpeg ProRes 422 Proxy	Limited	Full	No	2
Ffmpeg ProRes 422 Proxy	Limited	Limited	No	1

I compared each shot with difference set as blending mode and overlaid them. Any interpretations that matched existing ones joined a group. Hence the group column. The only incorrect variant that didn't share the same interpretation as the majority was set to in_range=limited:out_range:full this compounded the inaccurate interpretation by squashing ranges even more.

Interestingly, setting levels to auto in these tests seemed to be equivalent to limited. Not sure of the logic behind ffmpeg determining levels?

Our only correct ffmpeg interpretation was in_range=full:out_range=limited. Settingin_range=limited for Dave's camera yielded matching, yet inaccurate results. After a little more testing I developed my proposal and started an implementation.

Proposal

If data levels are set to auto in Resolve, we ffprobe the file and find the color_range.
If pc, set in_range=full.
If tv, set in_range=limited.

If data levels are set specifically to full or video, we pull that from the clip and treat it like the color_range. If full, set in_range=full.
If video, set in_range=limited.

Always set out_range=limited, unless codec preset specifies otherwise.

Conclusion

Confirmed data-level matching working in latest tests. Proxima now attempts to match Resolve's interpretation as closesly as possible, even if metadata is set incorrectly and Resolve's interpretation will be wrong. Incorrect interpretations can be fixed by setting data levels manually and re-encoding as needed to match new settings (just as clip-attribute level orientation settings work).

[in03]

Fixed an issue where the main video stream may not be the first stream probed by ffprobe. This caused a key error for color_range and would crash the worker.

[in03]

Color range issues