About reproduced results

[7Anonym]

Thanks for your great job in reproducing these works with Pytorch!

I have a question about the reproduced results. I am running the code with suggested scripts with a single GPU in readme:

python -m torch.distributed.launch \ --nproc_per_node=1 \ --use_env main.py \ cifar100_dualprompt \ --model vit_base_patch16_224 \ --batch-size 16 \ --data-path /local_datasets/ \ --output_dir ./output

this shall undergo the training process of CIFAR100, with a single GPU. The full log is reported here dualprompt.txt
The final performance is listed as:

`
Averaged stats: Lr: 0.001875 Loss: -0.7786 Acc@1: 93.7500 (93.3000) Acc@5: 100.0000 (99.3200)
Test: [Task 1] [ 0/63] eta: 0:00:17 Loss: 0.9931 (0.9931) Acc@1: 75.0000 (75.0000) Acc@5: 93.7500 (93.7500) time: 0.2747 data: 0.1619 max mem: 2112
Test: [Task 1] [30/63] eta: 0:00:03 Loss: 0.6518 (0.5845) Acc@1: 81.2500 (83.6694) Acc@5: 100.0000 (97.5806) time: 0.1127 data: 0.0001 max mem: 2112
Test: [Task 1] [60/63] eta: 0:00:00 Loss: 0.4332 (0.5457) Acc@1: 81.2500 (83.4016) Acc@5: 100.0000 (97.9508) time: 0.1190 data: 0.0001 max mem: 2112
Test: [Task 1] [62/63] eta: 0:00:00 Loss: 0.3826 (0.5352) Acc@1: 87.5000 (83.8000) Acc@5: 100.0000 (98.0000) time: 0.1163 data: 0.0001 max mem: 2112
Test: [Task 1] Total time: 0:00:07 (0.1176 s / it)

• Acc@1 83.800 Acc@5 98.000 loss 0.535
  Test: [Task 2] [ 0/63] eta: 0:00:16 Loss: 0.8936 (0.8936) Acc@1: 68.7500 (68.7500) Acc@5: 100.0000 (100.0000) time: 0.2681 data: 0.1567 max mem: 2112
  Test: [Task 2] [30/63] eta: 0:00:03 Loss: 0.7908 (0.8057) Acc@1: 68.7500 (74.3952) Acc@5: 100.0000 (95.9677) time: 0.1146 data: 0.0007 max mem: 2112
  Test: [Task 2] [60/63] eta: 0:00:00 Loss: 0.5982 (0.7659) Acc@1: 75.0000 (76.2295) Acc@5: 100.0000 (96.7213) time: 0.1126 data: 0.0001 max mem: 2112
  Test: [Task 2] [62/63] eta: 0:00:00 Loss: 0.5738 (0.7533) Acc@1: 81.2500 (76.5000) Acc@5: 100.0000 (96.8000) time: 0.1098 data: 0.0001 max mem: 2112
  Test: [Task 2] Total time: 0:00:07 (0.1156 s / it)
• Acc@1 76.500 Acc@5 96.800 loss 0.753
  Test: [Task 3] [ 0/63] eta: 0:00:18 Loss: 0.5311 (0.5311) Acc@1: 75.0000 (75.0000) Acc@5: 100.0000 (100.0000) time: 0.3011 data: 0.1866 max mem: 2112
  Test: [Task 3] [30/63] eta: 0:00:03 Loss: 0.6168 (0.6531) Acc@1: 81.2500 (79.6371) Acc@5: 100.0000 (97.7823) time: 0.1147 data: 0.0002 max mem: 2112
  Test: [Task 3] [60/63] eta: 0:00:00 Loss: 0.5880 (0.6590) Acc@1: 75.0000 (79.9180) Acc@5: 100.0000 (97.5410) time: 0.1131 data: 0.0002 max mem: 2112
  Test: [Task 3] [62/63] eta: 0:00:00 Loss: 0.5880 (0.6539) Acc@1: 81.2500 (80.1000) Acc@5: 100.0000 (97.6000) time: 0.1104 data: 0.0002 max mem: 2112
  Test: [Task 3] Total time: 0:00:07 (0.1163 s / it)
• Acc@1 80.100 Acc@5 97.600 loss 0.654
  Test: [Task 4] [ 0/63] eta: 0:00:16 Loss: 1.2372 (1.2372) Acc@1: 62.5000 (62.5000) Acc@5: 81.2500 (81.2500) time: 0.2606 data: 0.1482 max mem: 2112
  Test: [Task 4] [30/63] eta: 0:00:03 Loss: 0.7294 (0.8557) Acc@1: 75.0000 (73.9919) Acc@5: 100.0000 (95.3629) time: 0.1152 data: 0.0004 max mem: 2112
  Test: [Task 4] [60/63] eta: 0:00:00 Loss: 0.6991 (0.7561) Acc@1: 75.0000 (76.6393) Acc@5: 100.0000 (96.3115) time: 0.1123 data: 0.0001 max mem: 2112
  Test: [Task 4] [62/63] eta: 0:00:00 Loss: 0.7169 (0.7534) Acc@1: 75.0000 (76.6000) Acc@5: 100.0000 (96.3000) time: 0.1096 data: 0.0001 max mem: 2112
  Test: [Task 4] Total time: 0:00:07 (0.1155 s / it)
• Acc@1 76.600 Acc@5 96.300 loss 0.753
  Test: [Task 5] [ 0/63] eta: 0:00:16 Loss: 0.3555 (0.3555) Acc@1: 81.2500 (81.2500) Acc@5: 100.0000 (100.0000) time: 0.2636 data: 0.1497 max mem: 2112
  Test: [Task 5] [30/63] eta: 0:00:03 Loss: 0.6456 (0.7171) Acc@1: 75.0000 (77.0161) Acc@5: 100.0000 (96.7742) time: 0.1149 data: 0.0002 max mem: 2112
  Test: [Task 5] [60/63] eta: 0:00:00 Loss: 0.6275 (0.6793) Acc@1: 75.0000 (77.1516) Acc@5: 100.0000 (97.2336) time: 0.1126 data: 0.0001 max mem: 2112
  Test: [Task 5] [62/63] eta: 0:00:00 Loss: 0.6424 (0.6989) Acc@1: 75.0000 (76.7000) Acc@5: 100.0000 (97.1000) time: 0.1099 data: 0.0001 max mem: 2112
  Test: [Task 5] Total time: 0:00:07 (0.1157 s / it)
• Acc@1 76.700 Acc@5 97.100 loss 0.699
  Test: [Task 6] [ 0/63] eta: 0:00:16 Loss: 0.4319 (0.4319) Acc@1: 81.2500 (81.2500) Acc@5: 100.0000 (100.0000) time: 0.2672 data: 0.1561 max mem: 2112
  Test: [Task 6] [30/63] eta: 0:00:03 Loss: 1.1000 (0.9934) Acc@1: 62.5000 (67.5403) Acc@5: 93.7500 (95.1613) time: 0.1134 data: 0.0005 max mem: 2112
  Test: [Task 6] [60/63] eta: 0:00:00 Loss: 0.9254 (1.0341) Acc@1: 68.7500 (65.8811) Acc@5: 100.0000 (95.4918) time: 0.1147 data: 0.0005 max mem: 2112
  Test: [Task 6] [62/63] eta: 0:00:00 Loss: 0.8768 (1.0284) Acc@1: 68.7500 (66.0000) Acc@5: 100.0000 (95.6000) time: 0.1119 data: 0.0005 max mem: 2112
  Test: [Task 6] Total time: 0:00:07 (0.1160 s / it)
• Acc@1 66.000 Acc@5 95.600 loss 1.028
  Test: [Task 7] [ 0/63] eta: 0:00:17 Loss: 1.4151 (1.4151) Acc@1: 62.5000 (62.5000) Acc@5: 87.5000 (87.5000) time: 0.2841 data: 0.1698 max mem: 2112
  Test: [Task 7] [30/63] eta: 0:00:03 Loss: 0.7989 (0.9311) Acc@1: 75.0000 (72.1774) Acc@5: 93.7500 (94.1532) time: 0.1148 data: 0.0002 max mem: 2112
  Test: [Task 7] [60/63] eta: 0:00:00 Loss: 0.9859 (0.9544) Acc@1: 68.7500 (72.1311) Acc@5: 93.7500 (93.9549) time: 0.1122 data: 0.0001 max mem: 2112
  Test: [Task 7] [62/63] eta: 0:00:00 Loss: 1.0050 (0.9652) Acc@1: 68.7500 (72.1000) Acc@5: 93.7500 (93.9000) time: 0.1095 data: 0.0001 max mem: 2112
  Test: [Task 7] Total time: 0:00:07 (0.1156 s / it)
• Acc@1 72.100 Acc@5 93.900 loss 0.965
  Test: [Task 8] [ 0/63] eta: 0:00:16 Loss: 0.7402 (0.7402) Acc@1: 75.0000 (75.0000) Acc@5: 87.5000 (87.5000) time: 0.2654 data: 0.1541 max mem: 2112
  Test: [Task 8] [30/63] eta: 0:00:03 Loss: 1.0145 (0.9946) Acc@1: 68.7500 (71.1694) Acc@5: 93.7500 (92.3387) time: 0.1139 data: 0.0002 max mem: 2112
  Test: [Task 8] [60/63] eta: 0:00:00 Loss: 0.9531 (0.9975) Acc@1: 68.7500 (70.7992) Acc@5: 93.7500 (92.8279) time: 0.1128 data: 0.0002 max mem: 2112
  Test: [Task 8] [62/63] eta: 0:00:00 Loss: 0.8923 (0.9979) Acc@1: 68.7500 (70.9000) Acc@5: 93.7500 (92.8000) time: 0.1101 data: 0.0002 max mem: 2112
  Test: [Task 8] Total time: 0:00:07 (0.1153 s / it)
• Acc@1 70.900 Acc@5 92.800 loss 0.998
  Test: [Task 9] [ 0/63] eta: 0:00:16 Loss: 0.4951 (0.4951) Acc@1: 81.2500 (81.2500) Acc@5: 100.0000 (100.0000) time: 0.2616 data: 0.1483 max mem: 2112
  Test: [Task 9] [30/63] eta: 0:00:03 Loss: 0.4687 (0.5289) Acc@1: 81.2500 (83.8710) Acc@5: 100.0000 (98.3871) time: 0.1149 data: 0.0003 max mem: 2112
  Test: [Task 9] [60/63] eta: 0:00:00 Loss: 0.4225 (0.5093) Acc@1: 87.5000 (85.5533) Acc@5: 100.0000 (98.3607) time: 0.1125 data: 0.0001 max mem: 2112
  Test: [Task 9] [62/63] eta: 0:00:00 Loss: 0.4110 (0.5027) Acc@1: 87.5000 (85.7000) Acc@5: 100.0000 (98.4000) time: 0.1098 data: 0.0001 max mem: 2112
  Test: [Task 9] Total time: 0:00:07 (0.1163 s / it)
• Acc@1 85.700 Acc@5 98.400 loss 0.503
  Test: [Task 10] [ 0/63] eta: 0:00:17 Loss: 0.7950 (0.7950) Acc@1: 68.7500 (68.7500) Acc@5: 100.0000 (100.0000) time: 0.2822 data: 0.1709 max mem: 2112
  Test: [Task 10] [30/63] eta: 0:00:03 Loss: 0.7083 (0.6928) Acc@1: 75.0000 (77.8226) Acc@5: 100.0000 (96.1694) time: 0.1128 data: 0.0002 max mem: 2112
  Test: [Task 10] [60/63] eta: 0:00:00 Loss: 0.6565 (0.6841) Acc@1: 75.0000 (77.1516) Acc@5: 100.0000 (97.1311) time: 0.1124 data: 0.0001 max mem: 2112
  Test: [Task 10] [62/63] eta: 0:00:00 Loss: 0.6565 (0.6851) Acc@1: 75.0000 (77.3000) Acc@5: 100.0000 (97.1000) time: 0.1096 data: 0.0001 max mem: 2112
  Test: [Task 10] Total time: 0:00:07 (0.1154 s / it)
• Acc@1 77.300 Acc@5 97.100 loss 0.685
  [Average accuracy till task10] Acc@1: 76.5700 Acc@5: 96.3600 Loss: 0.7574 Forgetting: 7.9000 Backward: -7.9000
  Total training time: 0:53:13`

It seems the final acc@1 is 77.3 instead of the reported 86.51. Did I miss anything? Thanks!

[7Anonym]

A similar problem occurs in the ImageNet-R dataset, where I got (with the suggested scripts)

• Acc@1 57.600 Acc@5 77.867 loss 2.098
  [Average accuracy till task10] Acc@1: 60.8711 Acc@5: 79.9315 Loss: 1.9111 Forgetting: 4.5296 Backward: -4.3146
  Total training time: 1:03:48

which is also less than the reported values.

[JH-LEE-KR]

Hi,
thanks for your comment.

It's my fault for writing the README incorrectly.

In the README batch_size was written as 16, but looking at the script file (tranin_cifar100_dualprompt.sh and train_imr_dualmprompt.sh) used for the actual train, it is 24.
Confused with L2P.

README will be modified and,
Sorry for the confusion once again.

Experiment with batch_size 24 and feel free to comment if there is anything strange.

Best,
Jaeho Lee.

[7Anonym]

Hi Jaeho,

Thanks for your timely reply. After changing the batch size, I obtain almost the same performance, i.e., final accuracy as 77.22.

In fact, I found a way to reproduce the results by changing the pre-trained model in Line 41 into vit_base_patch16_224_in21k.

In my opinion, the original L2P and DualPrompt are using the imagenet21k pretrained Vit model, since the link in the original repo is also directed to it (see this issue)

By changing the pre-trained model, I obtain the competitive results to the original paper.

[JH-LEE-KR]

Hi,
thanks for the comment.

I found that timm's vit_base_patch16_224_in21k and the model used in the official code are different.
It uses the augmented version of vit_base_patch16_224_in2lk url.

So I modified the url of vit_base_patch16_224 to be the same as the model used in the official code.
Check out my code here.

I still reproduce well.
Please let me know if there is anything I missed and
provide detailed information such as detailed environment, GPU information, etc.

Let's discuss about this issue together.

Best,
Jaeho Lee.

[7Anonym]

Hi Jaeho,

Thanks for your comment! I noticed your modification in the pretrained model, which is consistent with the original repo.

After the clarification, I am not quite sure about the reason for the performance gap. I will create a same environment to yours for evaluation to check whether the gap is from different torch version, etc.

BTW, could you provide the training log of yours in your environment? I am still curious about the reason for such performance gap. Thanks!

[JH-LEE-KR]

Sorry for the delayed reply.
I had busy jobs.

The training log is here. Please check it.
dualprompt_result.txt
This is the result of randomly giving only seed (--seed=$RANDOM)in train_cifar100_dualprompt.sh

I still don't know why the performance degradation is occurring.
More information and experiments are needed.

I will create a same environment to yours
Does this mean that the previous experiment was conducted in a different environment than mine?
If so, you also need to consider environmental issues.

Please let me know the results after experimenting in the same environment as me.

Best,
Jaeho Lee.

[zhaoedf]

Sorry for the delayed reply. I had busy jobs.

The training log is here. Please check it. dualprompt_result.txt This is the result of randomly giving only seed (--seed=$RANDOM)in train_cifar100_dualprompt.sh

I still don't know why the performance degradation is occurring. More information and experiments are needed.

I will create a same environment to yours Does this mean that the previous experiment was conducted in a different environment than mine? If so, you also need to consider environmental issues.

Please let me know the results after experimenting in the same environment as me.

Best, Jaeho Lee.

In your log, you use batch_size=24 and e-prompt length=5. i don't this setting matches to the official code?

[xvrong]

I encountered a similar problem where the performance dropped significantly just by changing the batch size during testing.

When the batch size is 128, the result is
[Average accuracy till task10] Acc@task: 57.7611 Acc@1: 75.6217 Acc@5: 89.3090 Loss: 1.0701 Forgetting: 3.3095 Backward: -2.6626

When the batch size is 1, the result is
[Average accuracy till task10] Acc@task: 57.7611 Acc@1: 65.1856 Acc@5: 81.7701 Loss: 1.5652 Forgetting: 5.6082 Backward: -5.4771

When I decrease the batch size, such as 128 64 32 16 8 4 2 1, the performance continuously drops. Are there any possible explanations for this? Thank you!"

python -m torch.distributed.launch \
        --nproc_per_node=4 \
        --use_env main.py \
        imr_hideprompt_5e \
        --original_model vit_base_patch16_224 \
        --model vit_base_patch16_224 \
        --batch-size 128 \
        --data-path ./datasets/ \
        --output_dir ./output/HiDe_imr_vit_covariance_mlp_2_seed42 \
        --epochs 20 \
        --ca_storage_efficient_method covariance \
        --sched constant \
        --seed 42 \
        --train_inference_task_only \
        --lr 0.0005 \
        --shuffle True \

python -m torch.distributed.launch \
        --nproc_per_node=4 \
        --use_env main.py \
	imr_hideprompt_5e \
	--model vit_base_patch16_224 \
	--original_model vit_base_patch16_224 \
	--batch-size 128 \
	--epochs 150 \
	--data-path ./datasets \
	--ca_lr 0.005 \
	--crct_epochs 30 \
	--seed 42 \
	--prompt_momentum 0.01 \
	--reg 0.5 \
	--length 20 \
	--sched cosine \
	--larger_prompt_lr \
    --ca_storage_efficient_method covariance \
	--trained_original_model ./output/HiDe_imr_vit_covariance_mlp_2_seed42 \
	--output_dir ./output/HiDe_imr_vit_pe_seed42 \
    --shuffle True  \


# test with batch size 128
python main.py \
	imr_hideprompt_5e \
	--model vit_base_patch16_224 \
	--original_model vit_base_patch16_224 \
	--batch-size 128 \
	--epochs 150 \
	--data-path ./datasets \
	--ca_lr 0.005 \
	--crct_epochs 30 \
	--seed 42 \
	--prompt_momentum 0.01 \
	--reg 0.5 \
	--length 20 \
	--sched cosine \
	--larger_prompt_lr \
    --ca_storage_efficient_method covariance \
	--trained_original_model ./output/HiDe_imr_vit_covariance_mlp_2_seed42 \
	--output_dir ./output/HiDe_imr_vit_pe_seed42 \
    --shuffle True  \
	--eval

# test with batch size 1
python main.py \
	imr_hideprompt_5e \
	--model vit_base_patch16_224 \
	--original_model vit_base_patch16_224 \
	--batch-size 1 \
	--epochs 150 \
	--data-path ./datasets \
	--ca_lr 0.005 \
	--crct_epochs 30 \
	--seed 42 \
	--prompt_momentum 0.01 \
	--reg 0.5 \
	--length 20 \
	--sched cosine \
	--larger_prompt_lr \
    --ca_storage_efficient_method covariance \
	--trained_original_model ./output/HiDe_imr_vit_covariance_mlp_2_seed42 \
	--output_dir ./output/HiDe_imr_vit_pe_seed42 \
    --shuffle True  \
	--eval