Bookmarks

    Nothing saved yet. Use the bookmark on any article. Bookmarks are kept on this device only: they stay private and work without an account.

    Settings

    Appearance

    Reading mode

    Font size

    Content width

    Line spacing

    Keyboard shortcuts

    /
    Search
    j k
    Next / previous story
    ← →
    Previous / next section (columns)
    b
    Bookmarks
    s
    Settings
    r
    Columns / scroll
    t
    Light / dark

    Saved in this browser only.

    Ping2Prompt
    Subscribe to the Weekly Brief Sign in
    Illustration for: Hugging Face Transformers PR streams MoE experts from SSD on a 32GB Mac

    Hugging Face Transformers PR streams MoE experts from SSD on a 32GB Mac

    Hugging Face Hugging Face Transformers

    Transformers PR adds expert streaming from SSD

    Expert streaming is now a from_pretrained option in Transformers pull request huggingface/transformers#50717; the source does not state that it is merged. The post uses it to run Qwen3.8-Flash-Next, a model that is 121 GB in FP8, on a 32G MacBook M5.

    How the slot pool streams experts from disk

    According to the PR, experts mapped to "disk" in device_map skip Accelerate offload and take the streaming path. The expert cache becomes a slot pool: one preallocated device tensor shared by all layers, plus a per-layer table from expert id to slot. Router expert ids are remapped to slots, so the pool acts as a normal expert bank and grouped_mm runs on it without a custom expert matmul.

    Cache misses are read by 8 I/O threads, one contiguous read per expert. The routers of the next two layers run early on current hidden states to guess which experts to prefetch. Experts can be quantized offline to shrink reads, and the approach works for MoE models with the standard experts layout.

    The stock path was a poor fit: Accelerate offloads a layer's whole expert bank, so every token copies every expert of that layer to use 8 or 10 of them. On a Mac, CPU and GPU share memory, so offloading to CPU frees nothing. The author also says vllm-metal cannot run this model on 32 GB, because its n-gram tables, 32 GB in the MLX checkpoints on the Hub, stay in memory.

    Author's benchmarks: Transformers versus Strata on a MacBook M5

    All figures here are the post author's own measurements. They ran Strata 0.1.60 with IQ3_XXS experts, greedy decoding, thinking, MTP and speculative drafts off, and 1,024-token prefill chunks.

    • Decode, 200 tokens: Transformers 10.0 tok/s against Strata's 14.8.
    • Prefill, 4K prompt: 68 tok/s against 220.
    • Decode after a 6K document: 7.3 tok/s against 13.7.
    • Memory: about 18 GiB after loading, peaking at 25.4 GiB on a 6K prompt.

    On an A10G, the author measured Transformers with Accelerate at 0.16 tok/s decode, against 20.1 for vLLM FP8 with --cpu-offload-gb 16 --cpu-offload-params experts. On OLMoE-1B-7B, which fits in memory, the slot-pool version reached 34.6 tok/s against 38.1 for vanilla. The author's rule: if the experts fit in RAM, llama.cpp is hard to beat; if not, stream only what the router picks through a bounded cache.

    Running Qwen3.8-Flash-Next on 32 GB: setup and cost

    According to the PR's documentation, you install Transformers from the PR branch with pip install "git+https://github.com/huggingface/transformers.git@refs/pull/50717/head". The commands come from the documentation listed under Sources; confirm them against the PR before running them. The PR's example uses device_map={"": "mps", "experts.*": "disk"}, MetalConfig(bits=8) and experts_offload with cache_bytes of 12 << 30.

    The 32 GB setup quantizes experts to 3 bits (49 GB on disk, 12 GiB cached on the device) and dense layers to 8 bits (about 5.4 GB, +0.2% perplexity versus BF16). The 51 GB n-gram embedding table stays on disk and rows are read when needed. Conversion takes about 5 hours, and the author has to close the browser or macOS starts swapping. The converted model is on the Hub as tarekziade/Qwen3.8-Flash-Next-expert-streaming; code and raw results are in the Space tarekziade/moe-expert-streaming.

    Who should try the Transformers streaming branch

    If you run large MoE models on a Mac with limited memory, try the PR branch on a test machine first, since it is unreleased code. Keep a margin for memory: close other apps, and expect a long one-time conversion before the first run.

    What the post leaves open about speed and the API

    The subtitle claims streaming gets within 6% of Strata on token generation, but the visible text is truncated before the final measurement, so that figure is unverified here. The Transformers API is still under discussion with maintainers, and quantized experts will probably go through the per-layer quantization plan of #50245.

    Sources


    This story was produced with AI assistance from the sources listed above and fact-checked against them before publication. Read our editorial methodology and the corrections log. Spotted an error? Email [email protected].

    Continue

    Share on Bluesky · Mastodon · Email ·

    Discussion