Code-Llama-7B-Python 4 Bit Error on HumanEval

[wilyub]

Hi All,

I've been trying to run the HumanEval benchmark on some CodeLlama models. However, I get an error when using this code. Any idea what's causing the error? I also tested this code on "meta-llama/CodeLlama-7b-hf" (still 4-bit) and had no errors.

Code
accelerate launch main.py
--model "meta-llama/CodeLlama-7b-Python-hf"
--max_length_generation 1024
--tasks humaneval
--temperature 0.2
--n_samples 200
--batch_size 100
--allow_code_execution
--load_in_4bit
--save_generations
--save_generations_path /scratch/wpy2004/coding_compression_llms/eval_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json
--metric_output_path /scratch/wpy2004/coding_compression_llms/result_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json
--limit 10

Error
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 1
--num_machines was set to a value of 1
--mixed_precision was set to a value of 'no'
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
Selected Tasks: ['humaneval']
Loading model in 4bit
Loading checkpoint shards: 100%|██████████████| 2/2 [00:16<00:00, 8.42s/it]
number of problems for this task is 10
10%|████ | 2/20 [00:27<04:09, 13.87s/it]
Traceback (most recent call last):
File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/main.py", line 414, in
main()
File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/main.py", line 398, in main
results[task] = evaluator.evaluate(
^^^^^^^^^^^^^^^^^^^
File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 95, in evaluate
generations, references = self.generate_text(task_name, intermediate_generations=intermediate_generations)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 69, in generate_text
generations = parallel_generations(
^^^^^^^^^^^^^^^^^^^^^
File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/generation.py", line 141, in parallel_generations
generations = complete_code(
^^^^^^^^^^^^^^
File "/scratch/wpy2004/coding_compression_llms/bigcode-evaluation-harness/bigcode_eval/utils.py", line 294, in complete_code
generated_tokens = accelerator.unwrap_model(model).generate(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ext3/miniconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/ext3/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 2223, in generate
result = self._sample(
^^^^^^^^^^^^^
File "/ext3/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 3257, in _sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: probability tensor contains either inf, nan or element < 0
Traceback (most recent call last):
File "/ext3/miniconda3/bin/accelerate", line 8, in
sys.exit(main())
^^^^^^
File "/ext3/miniconda3/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/ext3/miniconda3/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1097, in launch_command
simple_launcher(args)
File "/ext3/miniconda3/lib/python3.11/site-packages/accelerate/commands/launch.py", line 703, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/ext3/miniconda3/bin/python', 'main.py', '--model', 'meta-llama/CodeLlama-7b-Python-hf', '--max_length_generation', '1024', '--tasks', 'humaneval', '--temperature', '0.2', '--n_samples', '200', '--batch_size', '100', '--allow_code_execution', '--load_in_4bit', '--save_generations', '--save_generations_path', '/scratch/wpy2004/coding_compression_llms/eval_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json', '--metric_output_path', '/scratch/wpy2004/coding_compression_llms/result_save/CodeLlama-7b-Python-hf-humaneval_bnb4.json', '--limit', '10']' returned non-zero exit status 1.