vLLM v0.32.0: structured outputs, low-precision KV cache, and Model Runner V2

The vLLM inference tool has released v0.32.0. In the release notes, the authors note 814 commits from 363 contributors, 121 of whom are new. Key changes include a new POST /v1/systemone endpoint for structured decisions, universal tool and reasoning parsers based on the response_template from the checkpoint, an expansion of low-precision KV cache to new architectures, and a switch to Model Runner V2 by default on ROCm.

What it means

For those running models through vLLM on their own hardware, the release brings several practical changes. Low-precision KV cache now works more broadly: NVFP4 is available on SM8x, SM12x, SM90 and in combination with XQA decode on SM12x, while GLM-5.x uses FP8 KV cache by default on SM100, which halves KV memory per rank at the cost of a slight increase in first-token latency. On ROCm, a 4-bit ultraquant_4bit KV cache has appeared with roughly twice the density compared to FP8. Model Runner V2 is now standard on ROCm, with model-specific exceptions removed. Added: a GPU n-gram speculator with variable-length drafts, Elastic EP, pipeline parallelism with prefill context parallelism for MLA models, and a stock torch.compile mode. The universal hf parser reads the response_template from the checkpoint, so models that contain it no longer require a model-specific parser — this works in both the Python and Rust frontends. More news about neural network inference tools is in the CPU3D news feed.