Need help in reproducing the results

[amitdoda1983]

Hi,

First of all, thanks for sharing the code and details here. I am exploring the continual learning space and going through this paper / code for better understanding.

I ran the code for few configurations and have the below observations/ queries.

1. I believe the results table in the paper uses Avg_End_Acc that we get after 10 runs.

2. I could reproduce the results (approx. almost same as table) for PCR (cifar10 & cifar100).

3. I am facing issues running PCR for mini_imagenet.
   Error :
   File "/content/PCR/models/resnet.py", line 85, in forward
   cos_dist = torch.mm(x_normalized,weight_normalized.transpose(0,1))
   RuntimeError: mat1 and mat2 shapes cannot be multiplied (20x640 and 160x100)

4. I could reproduce the results for SCR (all datasets) to somewhat very close to the table results.

5. However i am getting very different results for ER, GSS, MIR & ASER for all the three datasets and buffer sizes.

6. for example, below are logs/results for my results for few experiments.

Please review if my configurations/ parameters are right for respective model experiment.

ER cifar10

!python general_main.py --num_runs 10 --data cifar10 --cl_type nc --agent ER --retrieve random --update random --mem_size 100 --num_tasks 5

Namespace(num_runs=10, seed=0, val_size=0.1, num_val=3, num_runs_val=3, error_analysis=False, verbose=True, store=False, save_path=None, agent='ER', update='random', retrieve='random', optimizer='SGD', learning_rate=0.1, epoch=1, batch=10, test_batch=128, weight_decay=0, num_tasks=5, fix_order=False, plot_sample=False, data='cifar10', cl_type='nc', ns_factor=(0.0, 0.4, 0.8, 1.2, 1.6, 2.0, 2.4, 2.8, 3.2, 3.6), ns_type='noise', ns_task=(1, 1, 2, 2, 2, 2), online=True, mem_size=100, eps_mem_batch=10, lambda_=100, alpha=0.9, fisher_update_after=50, subsample=50, gss_mem_strength=10, gss_batch_size=10, k=5, aser_type='asvm', n_smp_cls=2.0, stm_capacity=1000, classifier_chill=0.01, log_alpha=-300, minlr=0.0005, clip=10.0, mem_epoch=70, labels_trick=False, separated_softmax=False, kd_trick=False, kd_trick_star=False, review_trick=False, ncm_trick=False, mem_iters=1, min_delta=0.0, patience=0, cumulative_delta=False, temp=0.07, buffer_tracker=False, warmup=4, head='mlp', cuda=True)

---

Avg_End_Acc (0.19796999999999998, 0.008131261957618655) Avg_End_Fgt (0.67701, 0.020675848339865154)
Avg_Acc (0.4240406666666667, 0.011397132721618845) Avg_Bwtp (0.0, 0.0) Avg_Fwt (0.0, 0.0)

MIR cifar10

!python general_main.py --num_runs 10 --data cifar10 --cl_type nc --agent ER --retrieve MIR --update random --mem_size 100 --num_tasks 5

Namespace(num_runs=10, seed=0, val_size=0.1, num_val=3, num_runs_val=3, error_analysis=False, verbose=True, store=False, save_path=None, agent='ER', update='random', retrieve='MIR', optimizer='SGD', learning_rate=0.1, epoch=1, batch=10, test_batch=128, weight_decay=0, num_tasks=5, fix_order=False, plot_sample=False, data='cifar10', cl_type='nc', ns_factor=(0.0, 0.4, 0.8, 1.2, 1.6, 2.0, 2.4, 2.8, 3.2, 3.6), ns_type='noise', ns_task=(1, 1, 2, 2, 2, 2), online=True, mem_size=100, eps_mem_batch=10, lambda_=100, alpha=0.9, fisher_update_after=50, subsample=50, gss_mem_strength=10, gss_batch_size=10, k=5, aser_type='asvm', n_smp_cls=2.0, stm_capacity=1000, classifier_chill=0.01, log_alpha=-300, minlr=0.0005, clip=10.0, mem_epoch=70, labels_trick=False, separated_softmax=False, kd_trick=False, kd_trick_star=False, review_trick=False, ncm_trick=False, mem_iters=1, min_delta=0.0, patience=0, cumulative_delta=False, temp=0.07, buffer_tracker=False, warmup=4, head='mlp', cuda=True)

----------- Avg_End_Acc (0.20265999999999998, 0.009694146294515506) Avg_End_Fgt (0.67465, 0.030499070146613624) Avg_Acc (0.42342783333333334, 0.013827498535359988) Avg_Bwtp (0.0, 0.0) Avg_Fwt (0.0, 0.0)-----------

ASER cifar10

!python general_main.py --num_runs 10 --data cifar10 --cl_type nc --agent ER --retrieve ASER --update ASER --mem_size 100 --num_tasks 5

Namespace(num_runs=10, seed=0, val_size=0.1, num_val=3, num_runs_val=3, error_analysis=False, verbose=True, store=False, save_path=None, agent='ER', update='ASER', retrieve='ASER', optimizer='SGD', learning_rate=0.1, epoch=1, batch=10, test_batch=128, weight_decay=0, num_tasks=5, fix_order=False, plot_sample=False, data='cifar10', cl_type='nc', ns_factor=(0.0, 0.4, 0.8, 1.2, 1.6, 2.0, 2.4, 2.8, 3.2, 3.6), ns_type='noise', ns_task=(1, 1, 2, 2, 2, 2), online=True, mem_size=100, eps_mem_batch=10, lambda_=100, alpha=0.9, fisher_update_after=50, subsample=50, gss_mem_strength=10, gss_batch_size=10, k=5, aser_type='asvm', n_smp_cls=2.0, stm_capacity=1000, classifier_chill=0.01, log_alpha=-300, minlr=0.0005, clip=10.0, mem_epoch=70, labels_trick=False, separated_softmax=False, kd_trick=False, kd_trick_star=False, review_trick=False, ncm_trick=False, mem_iters=1, min_delta=0.0, patience=0, cumulative_delta=False, temp=0.07, buffer_tracker=False, warmup=4, head='mlp', cuda=True)

----------- Avg_End_Acc (0.20358, 0.006487462886215012) Avg_End_Fgt (0.6760400000000001, 0.025897505237588537) Avg_Acc (0.42569850000000004, 0.009808121280926051) Avg_Bwtp (0.0, 0.0) Avg_Fwt (0.0, 0.0)-----------

This is the full table showing the reproduced results.

Request you to help me with these queries.

Regards
Amit