[smolvlm] fix video inference

[zucchini-nlp]

What does this PR do?

Fixes #39006. The model actually had default values for sampling and thus the flag has to be set to True for BC

Added a small tests, we had no video tests and thus the bug went unnoticed

[pcuenca]

Nice! Generation works now, although greedy generation does not match the results from v4.51.3. I assume this may be because we are now defaulting to bilinear bicubic in the fast processor, whereas we were using Lanczos resampling before?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[pcuenca]

We still read all the video frames here (shape using the repro above: (559, 730, 1920, 3)). I think that was not the case in previous versions.

[pcuenca]

Shape in previous (v4.51.3) read_video_pyav (was in image_utils.py): (9, 730, 1920, 3).

[zucchini-nlp]

We still read all the video frames here (shape using the repro above: (559, 730, 1920, 3)). I think that was not the case in previous versions.

Yes, this is expected though I didn't think about extra RAM usage when we decode the whole video. I will open a subsequent PR, allowing video processors to directly accept url/paths at input and decode/sample videos before preprocessing

[pcuenca]

Also unrelated to this PR: even when we use use_fast=False, we get the following warning from here:

            logger.warning_once(
                "You have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. "
                "BICUBIC resample will be used as an alternative. Please fall back to image processor if you "
                "want full consistency with the original model."
            )

I'm not sure if this is actionable, how can the user fall back to using the image processor instead?

[pcuenca]

this is expected

Noting that this broke in v4.52. Previously, we passed the sampling function here, and then frames were filtered.

[qubvel]

logger.warning_once(
"You have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. "
"BICUBIC resample will be used as an alternative. Please fall back to image processor if you "
"want full consistency with the original model."
)

Does it mean we should use slow ImageProcessor instead of fast VideoProcessor? Probably a code snippet in the warning should clarify it.
processor = *VideoProcessor.from_pretrained -> processor = *ImageProcessor.from_pretrained(..., use_fast=False)

[zucchini-nlp]

I'm not sure if this is actionable, how can the user fall back to using the image processor instead?

Yeah, this is one thing that is not really actionable since video have no slow processing option, and we'll be defaulting to using only fast processors slowly. Maybe we can delete that, I remember @yonigozlan wanted to ask smolVLM team about defaulting to bilinear interpolation for models

[pcuenca]

Does it mean we should use slow ImageProcessor instead of fast VideoProcessor? Probably a code snippet in the warning should clarify it.

How? The only solution would be to instantiate the image processor and tokenizer manually, and then build SmolVLMProcessor with them. Is there another way? This is probably far from what the user was doing (AutoProcessor.from_pretrained, in my case).

[zucchini-nlp]

The only solution would be to instantiate the image processor and tokenizer manually, and then build SmolVLMProcessor with them. Is there another way?

+1, i think we either just remove it (including in image processor) or create a non-very-efficient resizing by forcing PIL based lanczos. Personally, I prefer the first option as we have only 3 such models (SmolVLM, Flava, Chameleon), though that means model's will not match to their original implementations fully

[pcuenca]

cc @andimarafioti for info on lanczos vs bicubic.

(This doesn't have to be decided for this PR in my opinion)

[pcuenca]

Looks good to merge and fix inference!

Remaining work for followup PRs could include:

• Go back to using less memory when reading videos.
• Reword bicubic vs Lanczos warning (it's not easily actionable), or provide a way to fallback to Lanczos.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: smolvlm

[zucchini-nlp]

run-slow: smolvlm

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/smolvlm']
quantizations: [] ...