Select device: GPU for model eval #297
Description
has the error：
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 23.69 GiB of which 2.53 GiB is free. Process 3238833 has 3.43 GiB memory in use. Process 3238831 has 2.53 GiB memory in use. Process 3238830 has 2.75 GiB memory in use. Process 3238832 has 2.61 GiB memory in use. Including non-PyTorch memory, this process has 2.41 GiB memory in use. Process 3509629 has 2.41 GiB memory in use. Process 3509627 has 2.38 GiB memory in use. Process 3509630 has 2.41 GiB memory in use. Of the allocated memory 2.16 GiB is allocated by PyTorch, and 1.92 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
How can I eval the model edumunozsala/llama-2-7b-int4-python-code-20k with cuda:0 , beacause that the cuda:0 has been occupied by other tasks. Shall I change the config by (accelerate config) ?