ONNXConfig: Add a configuration for all available models

[chainyo]

ISSUE TRANSFER: Optimum repository -> huggingface/optimum#555

This issue is about the working group specially created for this task. If you are interested in helping out, take a look at this organization, or add me on Discord: ChainYo#3610

We want to contribute to HuggingFace's ONNX implementation for all available models on HF's hub. There are already a lot of architectures implemented for converting PyTorch models to ONNX, but we need more! We need them all!

Feel free to join us in this adventure! Join the org by clicking here

Here is a non-exhaustive list of models that all models available:

• Albert
• BART
• BeiT
• BERT
• BigBird
• BigBirdPegasus
• Blenderbot
• BlenderbotSmall
• BLOOM
• CamemBERT
• CANINE
• CLIP
• CodeGen
• ConvNext
• ConvBert
• CTRL
• CvT
• Data2VecText
• Data2VecVision
• Deberta
• DebertaV2
• DeiT
• DecisionTransformer
• DETR
• Distilbert
• DPR
• DPT
• ELECTRA
• FNet
• FSMT
• Flaubert
• FLAVA
• Funnel Transformer
• GLPN
• GPT2
• GPTJ
• GPT-Neo
• GPT-NeoX
• Hubert
• I-Bert
• ImageGPT
• LED
• LayoutLM
• 🛠️ LayoutLMv2
• LayoutLMv3
• LayoutXLM
• LED
• LeViT
• Longformer
• LongT5
• 🛠️ Luke
• Lxmert
• M2M100
• MaskFormer
• mBart
• MCTCT
• MPNet
• MT5
• MarianMT
• MegatronBert
• MobileBert
• MobileViT
• Nyströmformer
• OpenAIGPT-2
• 🛠️ OPT
• OWLViT
• PLBart
• Pegasus
• Perceiver
• PoolFormer
• ProphetNet
• QDQBERT
• RAG
• REALM
• 🛠️ Reformer
• RemBert
• ResNet
• RegNet
• RetriBert
• RoFormer
• RoBERTa
• SEW
• SEW-D
• SegFormer
• Speech2Text
• Speech2Text2
• Splinter
• SqueezeBERT
• Swin Transformer
• T5
• TAPAS
• TAPEX
• Transformer XL
• TrOCR
• UniSpeech
• UniSpeech-SAT
• VAN
• ViT
• Vilt
• VisualBERT
• Wav2Vec2
• WavLM
• XGLM
• XLM
• XLMProphetNet
• XLM-RoBERTa
• XLM-RoBERTa-XL
• 🛠️ XLNet
• YOLOS
• Yoso

🛠️ next to a model suggests that the PR is in progress. If there is nothing next to a model, it means that ONNX does not yet support the model, and thus we need to add support for it.

If you need help implementing an unsupported model, here is a guide from HuggingFace's documentation.

If you want an example of implementation, I did one for CamemBERT months ago.

[chainyo]

• GPT-J: add GPT-J ONNX config to Transformers #16274
• FlauBERT: Add Flaubert OnnxConfig to Transformers #16279

[chainyo]

• LayoutLMv2: Add LayoutLMv2 OnnxConfig #16309

[vumichien]

Let me try with BigBird

[vumichien]

• Bigbird add Bigbird ONNX config #16427

[LysandreJik]

Love the initiative here, thanks for opening an issue! Added the Good First Issue label so that it's more visible :)

[chainyo]

Love the initiative here, thanks for opening an issue! Added the Good First Issue label so that it's more visible :)

Thanks for the label. I don't know if it's easy to begin, but it's cool if more people see this and can contribute!

[aakashb95]

I would like to try with Luke. However, Luke doesn't support any features apart from default AutoModel. It's main feature is LukeForEntityPairClassification for relation extraction. Should I convert luke-base to Onnx or LukeForEntityPairClassification which has a classifier head?

[xiadingZ]

Data2vecAudio doesn't have ONNXConfig yet. I write its ONNXConfig according to Data2VecTextOnnxConfig but it throws error. Can anyone help me?

from typing import Mapping, OrderedDict
from transformers.onnx import OnnxConfig

from transformers import AutoConfig
from pathlib import Path
from transformers.onnx import export
from transformers import AutoTokenizer, AutoModel

class Data2VecAudioOnnxConfig(OnnxConfig):
    @property
    def inputs(self):
        return OrderedDict(
            [
                ("input_values", {0: "batch", 1: "sequence"}),
                ("attention_mask", {0: "batch", 1: "sequence"}),
            ]
        )
    
config = AutoConfig.from_pretrained("facebook/data2vec-audio-base-960h")
onnx_config = Data2VecAudioOnnxConfig(config)



onnx_path = Path("facebook/data2vec-audio-base-960h")
model_ckpt = "facebook/data2vec-audio-base-960h"
base_model = AutoModel.from_pretrained(model_ckpt)
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)

onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, onnx_config.default_onnx_opset, onnx_path)

errors

ValueError                                Traceback (most recent call last)
/var/folders/2t/0w65vdjs2m32w5mmzzgtqrhw0000gn/T/ipykernel_59977/667985886.py in <module>
     27 tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
     28 
---> 29 onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, onnx_config.default_onnx_opset, onnx_path)

~/miniconda3/lib/python3.9/site-packages/transformers/onnx/convert.py in export(tokenizer, model, config, opset, output)
    255 
    256     if is_torch_available() and issubclass(type(model), PreTrainedModel):
--> 257         return export_pytorch(tokenizer, model, config, opset, output)
    258     elif is_tf_available() and issubclass(type(model), TFPreTrainedModel):
    259         return export_tensorflow(tokenizer, model, config, opset, output)

~/miniconda3/lib/python3.9/site-packages/transformers/onnx/convert.py in export_pytorch(tokenizer, model, config, opset, output)
    112 
    113             if not inputs_match:
--> 114                 raise ValueError("Model and config inputs doesn't match")
    115 
    116             config.patch_ops()

ValueError: Model and config inputs doesn't match

[chainyo]

I would like to try with Luke. However, Luke doesn't support any features apart from default AutoModel. It's main feature is LukeForEntityPairClassification for relation extraction. Should I convert luke-base to Onnx or LukeForEntityPairClassification which has a classifier head?

When you implement the ONNX Config for a model it's working for all kind of task, because the base model and the ones pre-packaged for fine-tuning have the same inputs.

So you can base your implementation on the base model and other tasks will work too.

[chainyo]

• LUKE Add Luke to Onnx #16562 from @aakashb95 👍

[jorabara]

Still learning

[pikaqqqqqq]

Issue description

Hello, thank you for supporting GPTJ with ONNX. But when I exported an ONNX checkpoint using transformers-4.18.0, I got the issue like below.

(venv) root@V100:~# python -m transformers.onnx --model=gpt-j-6B/ onnx/
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/lvenv/lib/python3.8/site-packages/transformers/onnx/__main__.py", line 99, in <module>
    main()
  File "/data/venv/lib/python3.8/site-packages/transformers/onnx/__main__.py", line 62, in main
    raise ValueError(f"Unsupported model type: {config.model_type}")
ValueError: Unsupported model type: gptj

I found GPTJ with ONNX seems supported when I checked your document transformers-4.18.0 [https://huggingface.co/docs/transformers/serialization#exporting-a-model-for-an-unsupported-architecture] and code [src/transformers/onnx/features.py etc.]. But I still got this issue. And then, I checked the parameter of config.model_type in File "/data/venv/lib/python3.8/site-packages/transformers/onnx/main.py", which is related to two parameters [from ..models.auto.feature_extraction_auto import FEATURE_EXTRACTOR_MAPPING_NAMES, from ..models.auto.tokenization_auto import TOKENIZER_MAPPING_NAMES]. I did not find GPTJ's config in these configs. It seems not sensible.

Environment info

• Platform: Ubuntu 20.04.2
• python: 3.8.10
• PyTorch: 1.10.0+cu113
• transformers: 4.18.0
• GPU: V100

[chainyo]

Hello, thank you for supporting GPTJ with ONNX. But when I exported an ONNX checkpoint using transformers-4.18.0, I got the issue like below.

Hello @pikaqqqqqq, thanks for reporting the problem. I opened a PR with a quick fix to avoid this problem, check #16780