Big batchsize cause OOM in bloom-ds-inference.py, how to adjust max_split_size_mb value

[tohneecao]

OutOfMemoryError: CUDA out of memory. Tried to allocate 62.00 MiB (GPU 6; 79.19 GiB total capacity; 66.51 GiB already allocated; 61.56 MiB free; 67.77 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFtorch.cuda
.OutOfMemoryError : return forward_call(*input, **kwargs)CUDA out of memory. Tried to allocate 62.00 MiB (GPU 4; 79.19 GiB total capacity; 66.51 GiB already allocated; 61.56 MiB free; 67.77 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[mayank31398]

max_split_size_mb won't work with deepspeed inference I think.
This is only for pure pytorch native code.