Fail to reproduce tf2 bert-large f1-score on SQuADv1.1

[zhuango]

System Info

transformers=4.20.1, python=3.7, tensorflow-gpu=2.9.1.

Who can help?

@Rocketknight1 @sgugger @patil-suraj

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Download bert-large tf2 pretrained models, .h5 file from huggingface.
2. Run pytorch question-answering example for reference. My script is:

python run_qa.py \
  --model_name_or_path $BERT_DIR \
  --dataset_name $SQUAD_DIR \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 128 \
  --doc_stride 48 \
  --output_dir $OUTPUT \
  --save_steps 10000 \
  --overwrite_cache \

I got an f1-score 90.3953%. Note that the pretrained model are loaded from tf2 .h5 checkpoints.

3. Run tf2 question-answering example with the same setting of pytorch example. My script is:

python run_qa.py \
  --model_name_or_path $BERT_DIR \
  --dataset_name $SQUAD_DIR \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 128 \
  --doc_stride 48 \
  --output_dir $OUTPUT \
  --save_steps 10000 \
  --overwrite_cache \

I only got an f1-score 88.5672%, which is much lower than expected and pytorch results (90.3953%).

Expected behavior

TF2 question-answering example should achieve similar f1-score with results of corresponding pytroch example.
Or you guys could provide script examples that achieves the target F1-score. Thanks.

[Rocketknight1]

Hi @zhuango, it's possible there's a bug here, but there is also always some variation between runs. Can you run the training a couple of times for both PyTorch and TensorFlow and report the range of F1-scores you get for both?

[zhuango]

@Rocketknight1 Sure, I willl try more runs with exactly the same hyper-parameters. But the hyper-parameter setting I showed above is the one that can achieve the highest f1-score. I have run the tf2 bert-large fine-tuning on SQuADv1.1 with different hyper-parameter setting combinations, like total batchsize=12, 24/warmup_rate=0, 0.1/learning rate=3e-5, 1e-5/sequence length=384, 128/doc_stride=128, 48/Adam, AdamW optimizers. All the results are lower than 87% except the one I showed above.
So I doubt that maybe I miss something in the hyper-parameter setting if there is no bug in the example. And Is it pissible that you share the setting that you guys used to reproduced the F1-score?
Note that I modified the example code to make sure the model's inputs are complete, Please have a check on #18223 for details.

[Rocketknight1]

Hi @zhuango - I just saw your other issue, I don't know how I missed it the first time. You're completely right - that example is hardcoding the input columns to the model, which is totally wrong and will degrade performance for models with token_type_ids inputs. I'll work on a PR and let you know when it's ready.

[zhuango]

@Rocketknight1 Thanks, I tried more runs with the same setting (and fixed #18223 ) and still got 88% like F1-socre.

But, I tried a linear decay learning rate scheduler plus 0.1 * total training steps warmup and AdamW optimizers (from tensorflow_addons), and I finally got 90.247% F1-score. Here is my optimizers:

            # Calculate the total trainnig steps.
            total_steps = int(training_args.num_train_epochs * ( len(processed_datasets["train"]) / (training_args._n_gpu * training_args.per_device_train_batch_size) ))

            # Linear decay learning rate.
            linear_decay = tf.keras.optimizers.schedules.PolynomialDecay(
                training_args.learning_rate,
                total_steps,
                end_learning_rate=0.0,
                power=1.0,
                cycle=False,
                name=None
            )

            # 0.1 * total_steps warmup.
            warmup_schedule = tfm.optimization.lr_schedule.LinearWarmup(
                warmup_learning_rate = 0,
                after_warmup_lr_sched = linear_decay,
                warmup_steps = total_steps * 0.1,
            )

            # AdamW optimizers with no weight decay.
            optimizer = tfa.optimizers.AdamW(
                weight_decay=0.0,
                learning_rate=warmup_schedule,
                beta_1=training_args.adam_beta1,
                beta_2=training_args.adam_beta2,
                epsilon=1e-06,
                amsgrad=False,
                name='AdamW',
                clipnorm=training_args.max_grad_norm,
            )

Maybe you would like to try the above setting with the tf example code.

[Rocketknight1]

@zhuango Thank you for that! It's reminding me that all of our examples need to be overhauled, honestly. I'm going to make a PR to modernize all of them - I'll ping you when it's ready if you want to review it.

[zhuango]

@Rocketknight1 Sure, Thank you.

[Rocketknight1]

Hi @zhuango, I've made a PR to rewrite all of our examples. If you get a chance, please confirm that the SQUaD performance using the new examples matches the PyTorch performance!