How to Fix CUDA out of memory error during KV cache allocation (vLLM)

Quick Answer: This error occurs when vLLM attempts to allocate GPU memory for the Key-Value (KV) cache, but the available VRAM is exhausted due to high concurrency, long context lengths, or a large default memory utilization fraction. The fastest fix is to reduce the GPU memory utilization parameter or decrease the maximum model sequence length.

What Causes This Error

• The gpu_memory_utilization parameter is set too high, leaving insufficient headroom for model weights and activation buffers.
• The max_model_len is configured larger than what your GPU VRAM can support alongside the KV cache.
• Too many concurrent requests are being processed, demanding more KV cache blocks than available.
• Other applications or background processes are consuming a portion of the GPU's VRAM.

Step-by-Step Fixes

Fix 1: Fix 1: Lower GPU Memory Utilization

Locate your vLLM startup command or configuration file.,Add or modify the --gpu-memory-utilization flag (e.g., set it to 0.85 or 0.8 instead of the default 0.90).,Restart the vLLM server to apply the new memory allocation limit.

Fix 2: Fix 2: Reduce Maximum Model Length

Identify the current max_model_len setting in your vLLM deployment script.,Decrease the value (e.g., from 8192 to 4096) to reduce the per-request KV cache memory footprint.,Restart the vLLM service and test with your workload.

Fix 3: Fix 3: Enable Tensor Parallelism

Check if you have multiple GPUs available on your node.,Add the --tensor-parallel-size flag matching your available GPU count (e.g., --tensor-parallel-size 2).,Restart the server to distribute the model weights and KV cache across multiple devices.

Advanced Fixes

Advanced Fix 1: Advanced: Tune KV Cache Block Size and Swap Space

Configure --swap-space to offload KV cache blocks to CPU memory when GPU VRAM runs tight.,Adjust block size configurations if supported by your vLLM version to minimize internal fragmentation.

FAQs

Q: What is the default gpu_memory_utilization in vLLM?

A: The default value is 0.90, meaning vLLM tries to pre-allocate 90% of the total GPU VRAM for the KV cache and model weights, leaving 10% as safety headroom.

Q: How can I check current GPU VRAM usage before starting vLLM?

A: You can use the 'nvidia-smi' command in your terminal to monitor real-time GPU memory consumption and identify any background processes hogging VRAM.