[Model] Add varco-vision-2.0-1.7b

[cyr0930]

Hi again
We've just uploaded our small model varco-vision-2.0-1.7b.
This is our HF url: https://huggingface.co/NCSOFT/VARCO-VISION-2.0-1.7B
Thanks :)

[cyr0930]

We found that some of VQA benchmarks doesn't work with our previous prompt as they use exact match for evaluation.
Could you re-evaluate our model "varco-vision-2-14b", please?

[cyr0930]

Sorry, now that I think about it, it's better to be a draft for now.
I will change status after I check the prompt thoroughly.
Thanks :)

[cyr0930]

It's good to go I think. Thanks for waiting!

[FangXinyu-0913]

Hi @cyr0930, thanks for your contribution to our community! It seems that after the prompt change, the MMVet score gets lower, and the OCRBench score and MathVista Acc remains the same.

Before this prompt change (MMVet):
"Category","tot","acc"
"rec","187","66.84491978609631"
"ocr","108","81.8518518518519"
"know","84","50.23809523809524"
"gen","80","49.12499999999998"
"spat","75","76.39999999999999"
"math","26","87.69230769230768"
"Overall","218","69.17431192660554"

After this prompt change (MMVet):
"Category","tot","acc"
"rec","187","66.36363636363643"
"ocr","108","81.9444444444445"
"know","84","49.642857142857146"
"gen","80","49.249999999999986"
"spat","75","76.39999999999999"
"math","26","87.69230769230768"
"Overall","218","68.85321100917436"

If you confirm this change after reviewing the results, I'll help you merge it. If you need more detailed results, you can also bring them up, and I will upload them here.

[cyr0930]

@FangXinyu-0913
Thank you for sharing the results. I'll check that as soon as possible :)

[cyr0930]

I think this would be better and cover more datasets.
Sorry for bothering you :)

[FangXinyu-0913]

I think this would be better and cover more datasets. Sorry for bothering you :)

After your latest fix, the MMVet score seems to get lower:

"Category","tot","acc"
"rec","187","66.36363636363643"
"ocr","108","81.38888888888894"
"know","84","48.80952380952379"
"gen","80","48.24999999999998"
"spat","75","76.26666666666665"
"math","26","87.69230769230768"
"Overall","218","68.66972477064225"

[cyr0930]

Thanks for evaluating our model.
That is bad news for us, but I think it's just a variation in judging model then.
I just reverted the code as it looks cleaner.
Lower score in MMVet is okay to us, thanks!