smolvlm

[pcuenca]

No description provided.

[pcuenca]

This is a proof-of-concept example for video generation with smolvlm. It's much simpler than trying to integrate within the existing video_generate script: there are differences in the chat template, and processing is handled automatically by the processor.

[pcuenca]

System prompt is important, so added as an option.

[pcuenca]

This is a recent addition to transformers: apply_chat_template can now produce inputs in a single step, including for VLMs.

We could potentially leverage this as well in the video_generate example (to be tested with various models).

[Blaizzy]

This is great news!

I will refactor this in the next release and test the models.

From which version? Because we need to pin 📌 that version of transformers in the requirements.txt

[pcuenca]

I think 4.49.0, released yesterday. SmolVLM2 requires main or this tagged release: https://github.com/huggingface/transformers/releases/tag/v4.49.0-SmolVLM-2

[Blaizzy]

Awesome 🙌🏽

Thanks Pedro!

Just two things, whilst I review the code:

1. Run pre-commit
2. Bump the version to v0.1.14

[Blaizzy]

LGTM!

I will approve once the 3 changes request are done.

[pcuenca]

This includes the processor changes, but SmolVLM2 will require main or this release: https://github.com/huggingface/transformers/releases

[Blaizzy]

LGTM!

[jrp2014]

With the latest transformers Requirement already satisfied: transformers in /opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages (4.49.0)

I get:

Running mlx-community/SmolVLM2-2.2B-Instruct-mlx
Fetching 12 files: 100%|███████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 24093.66it/s]
Fetching 12 files: 100%|███████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 29995.02it/s]
==========
Files: ['/Users/xxx/Pictures/Processed/20250215-161252_DSC02535_openWith-1-Edit.jpg'] 

Prompt: <|im_start|>User:<image>Provide a factual caption, description and comma-separated keywords or tags for this image so that it can be searched for easily.<end_of_utterance>
Assistant:
Failed to generate output for model at mlx-community/SmolVLM2-2.2B-Instruct-mlx: GPT2TokenizerFast has no attribute tokenizer

One of the other models spits out

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.

so there must be some fiddling to be done with aligning parameters / configs.

[pcuenca]

Hello!

You need to install transformers from main, or from this stable branch:

pip install git+https://github.com/huggingface/[email protected]