Model checkpoints

[namin-an]

Hello,

Are the checkpoints (e.g., CLOOB - ResNet50) that you provided the models pretrained on CC or YFCC?
When I ran your demo notebook, I got the zero-shot classification performance of 3.66% for CLOOB - ResNet50, which seems to be similar to results in Table 1 (3.06, CLOOB trained on CC) than Table 2 (28.9 CLOOB trained on YFCC).

[fuersta]

Hello,
the checkpoints are trained on YFCC. You should get the correct performance.
Your results seem to be wrong. Did you try the same for other zeroshot datasets? What are the results?

[namin-an]

Hello,

Thank you for your reply.

The attached images are the code that I used to evaluate your pre-trained model (CLOOB—ResNet50) on the Birdsnap test dataset (1,855 images). I have also tried the Country211 dataset, which gave me lower results than the one reported in the paper.

Could you please go through my code and see if there are mistakes that I made? :(
I just changed the file name from clip to clip_cloob and downloaded the model checkpoint (https://ml.jku.at/research/CLOOB/downloads/checkpoints/cloob_rn50_yfcc_epoch_28.pt) and placed it under my checkpoint directory.

[namin-an]

BTW, I got the same score (18.8) as the one in the paper for CIFAR100.

[fuersta]

Did you try cloning the repo again and without changing anything running the notebook again?
I did the experiments with a clean repo and could get the same results as stated in the paper.