[Add] transformers text and image models

[ariG23498]

This PR adds the provision of evaluating models using transformers.

One can start a run by the following command

python clip_benchmark/cli.py eval \
--model_type "transformers" \
--model "openai" \
--pretrained "clip-vit-base-patch32" \
--dataset "imagenet1k" \
--seed 40 \
--task "zeroshot_classification" \
--output "result_{model}_{pretrained_full_path}_{task}_{dataset}_{language}.json" \
--dataset_root "{dataset}" \
--model_cache_dir "."

[qubvel]

ckpt = f"{model_name}/{pretrained}" may confusing, it's better to provide model_name as checkpoint on the hub, and hardcode pretrained as True IMO. Otherwise it's going to be like

model_name = "openai"
pretrained = "clip-..."

[ariG23498]

I had to choose this option for better verbosity.

CLIP_benchmark/clip_benchmark/cli.py

Line 247 in a230282

print(f"Running '{task}' on '{dataset_name}' with the model '{args.pretrained}' on language '{args.language}'")

[qubvel]

Interestingly, why should we remove 1-st dim?

[ariG23498]

I wanted to brain storm on this bit, thanks for asking this question.
The load function should return a model, transforms, and a tokenizer. In the load function I have written it returns the model, the image processor, and the tokenizer.

The transform is used in the collation function while building the dataloader, which is where it adds another dimension to the tensors. So I noticed that the images["pixel_values].shape=(b, 1, c, h, w).

Is there a way I could extract the transform function from an image_processor?

[qubvel]

AFAIR it might be good to be able to pass additional args for the tokenizer, e.g. for siglip we should specify
padding="max_length", max_length=64

[ariG23498]

I would have loved to pass extra parameters too, but there is no way to pass extra parameters to the load function. The only way we pass parameters is by arguments provided to the script.

[ariG23498]

Worth mentioning with the current integration to transformers I ran zero shot evaluation with open_clip siglip and transformers siglip

python clip_benchmark/cli.py eval \
--model_type "open_clip" \
--model "ViT-SO400M-14-SigLIP-384" \
--pretrained "webli" \
--dataset "imagenet1k" \
--seed 40 \
--task "zeroshot_classification" \
--output "result_{model}_{pretrained_full_path}_{task}_{dataset}_{language}.json" \
--dataset_root "{dataset}" \
--model_cache_dir "."

python clip_benchmark/cli.py eval \
--model_type "transformers" \
--model "google" \
--pretrained "siglip-so400m-patch14-384" \
--dataset "imagenet1k" \
--seed 40 \
--task "zeroshot_classification" \
--output "result_{model}_{pretrained_full_path}_{task}_{dataset}_{language}.json" \
--dataset_root "{dataset}" \
--model_cache_dir "."

Results:

dataset	model	pretrained	task	language	metrics.acc1	metrics.acc5	metrics.mean_per_class_recall
imagenet1k	google	siglip-so400m-patch14-384	zeroshot_classification	en	0.83066	0.97036	0.831
imagenet1k	ViT-SO400M-14-SigLIP-384	webli	zeroshot_classification	en	0.83084	0.97036	0.83102

[mehdidc]

Awesome @ariG23498 ! this is very helpful.

Sorry for late answer @ariG23498 @qubvel looking into this now.
Wouldn't it be possible to get the max_length directly from the weights or HF config, to avoid hardcoding it ?
otherwise we can add an option in the CLI.