vLLM v0.34.0: Extensible KV Cache Enabled by Default
vLLM v0.34.0 is out. The release adds ten new models, including Mistral Large 4, MiniMax-M3.1-Flash and Nemotron-3.5-Transcribe. Model Runner V1 has been removed, and Model Runner V2 is now the only GPU runner. Speedups are claimed on Blackwell and MI355X: up to 1.95x on decoding, up to 1.6x in throughput, and 1.24—1.40x faster TTFT for long prompts.
What it means
The main change for local deployment is that the extensible KV cache is now enabled by default. vLLM determines the cache size itself from the memory actually free after warmup, so most models get a larger cache and better throughput. The gpu_memory_utilization parameter defaults to 1.0. To restore the previous behavior, pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.
The update may break existing configurations. Model Runner V1 has been removed entirely: the VLLM_USE_V2_MODEL_RUNNER=0 variable now causes an error at startup. The default block size has changed from 16 to 64; for the old behavior, pass --block-size 16. ROCm 7.2 is no longer published; ROCm 10.0 images are required. huggingface_hub 2.0.0 or newer, tokenizers 0.23.3 or newer, and timm 1.0.30 or newer are also required. QuTLASS and fp_quant support has been removed, and fp_quant checkpoints no longer load.
Before updating, it is worth checking your current launch scripts: some parameters have changed their defaults, and the old runner is no longer available. More on tools for running neural networks locally — in the CPU3D news feed.How the method works. The diagram is drawn based on this news note.