Bookmarks

    Nothing saved yet. Use the bookmark on any article. Bookmarks are kept on this device only: they stay private and work without an account.

    Settings

    Appearance

    Reading mode

    Font size

    Content width

    Line spacing

    Keyboard shortcuts

    /
    Search
    j k
    Next / previous story
    ← →
    Previous / next section (columns)
    b
    Bookmarks
    s
    Settings
    r
    Columns / scroll
    t
    Light / dark

    Saved in this browser only.

    Ping2Prompt
    Subscribe to the Weekly Brief Sign in
    A white circuit-board pattern shaped like a human brain on a dark blue background, with glowing colored grid tiles beneath it
    Photo by Steve A Johnson on Unsplash

    vLLM v0.34.0 adds Mistral Large 4, drops Model Runner V1, changes defaults

    vLLM

    What changed

    vLLM v0.34.0 is out, with 1,061 commits from 412 contributors, 120 of them new. It adds 10 models led by Mistral Large 4, makes extensible KV cache the default and removes Model Runner V1.

    What it does

    Mistral Large 4 is listed at 1T total and 52B active parameters, in FP8 and NVFP4, and runs on NVIDIA Blackwell/Hopper and AMD MI350. Extensible KV cache sizes the cache on CUDA and ROCm from the memory actually free after CUDA graph capture and warmup. Model Runner V2 is now the only GPU runner. Nemotron-3.5-Transcribe, a speech model, is served through /v1/audio/transcriptions.

    What is verified

    All performance numbers below are vLLM's own claims from its release notes. vLLM says extensible KV cache grows the MiniMax-M3 KV cache on H200 from 433k to 672k tokens, at a cost of 6-18% higher TPOT on some H200 models at concurrency 128. vLLM reports up to 1.95x decode for Step5 NVFP4 MoE on TRTLLM-Gen on 4xGB200. vLLM lists a known regression: openai/gpt-oss-120b on AMD MI355X is slower than in v0.33.0. At 8k-input prefill and concurrency 128, throughput is about 24% lower and TTFT about 3x higher. ROCm correctness fixes caused it.

    Running it

    The commands and flags in this piece come from the release notes in the sources list, so check them against your own install before running them. Images include vllm/vllm-openai:v0.34.0 (CUDA 13.0), vllm/vllm-openai-rocm:v0.34.0, vllm/vllm-openai-cpu:v0.34.0 and vllm/vllm-openai-xpu:v0.34.0. For CUDA 13.0 wheels, run pip install vllm.

    ROCm 7.2 artifacts are gone; use the ROCm 10.0 wheels and images. You need huggingface_hub >= 2.0.0, tokenizers >= 0.23.3 and timm >= 1.0.30.

    • Default --block-size is now 64, up from 16. Pass --block-size 16 to keep the old behaviour.
    • gpu_memory_utilization now defaults to 1.0. To keep old sizing, pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.
    • Remove VLLM_USE_V2_MODEL_RUNNER=0; it now fails at startup.

    What it means for you

    If you run vLLM in production, test this release on a staging box first, because the new memory and block-size defaults change how a server starts and behaves. AMD users running the affected gpt-oss model on that GPU should benchmark their own workload before moving.

    What the sources don't cover

    The release notes give no release date, licence, price or hardware minimums for the new models, and no independent benchmarks exist yet.

    Sources

    Image: Photo by Steve A Johnson on Unsplash


    This story was produced with AI assistance from the sources listed above and fact-checked against them before publication. Read our editorial methodology and the corrections log. Spotted an error? Email [email protected].

    Continue

    Share on Bluesky · Mastodon · Email ·

    Discussion