> ## Content Index
> Fetch the complete content index at: https://ping2prompt.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Hugging Face Transformers PR streams MoE experts from SSD on a 32GB Mac
- URL: https://ping2prompt.com/hugging-face-transformers-ssd-expert-streaming-32gb-mac/
- Published: 2026-11-18T17:55:17.000Z
- Updated: 2026-11-18T17:55:17.000Z
- Description: Transformers pull request huggingface/transformers#50717 adds SSD expert streaming, not stated as merged. The author ran the 121 GB Qwen3.8-Flash-Next on a 32G MacBook M5 at 10.0 tok/s decode. Install from the PR branch on a test machine first.
- Author: Ping2Prompt Newsroom
- Tags: Hugging Face, Hugging Face Transformers, AI, #short-ping

## Transformers PR adds expert streaming from SSD

Expert streaming is now a `from_pretrained` option in Transformers pull request huggingface/transformers#50717; the source does not state that it is merged. The post uses it to run Qwen3.8-Flash-Next, a model that is 121 GB in FP8, on a 32G MacBook M5.

## How the slot pool streams experts from disk

According to the PR, experts mapped to "disk" in `device_map` skip Accelerate offload and take the streaming path. The expert cache becomes a slot pool: one preallocated device tensor shared by all layers, plus a per-layer table from expert id to slot. Router expert ids are remapped to slots, so the pool acts as a normal expert bank and `grouped_mm` runs on it without a custom expert matmul.

Cache misses are read by 8 I/O threads, one contiguous read per expert. The routers of the next two layers run early on current hidden states to guess which experts to prefetch. Experts can be quantized offline to shrink reads, and the approach works for MoE models with the standard experts layout.

The stock path was a poor fit: Accelerate offloads a layer's whole expert bank, so every token copies every expert of that layer to use 8 or 10 of them. On a Mac, CPU and GPU share memory, so offloading to CPU frees nothing. The author also says vllm-metal cannot run this model on 32 GB, because its n-gram tables, 32 GB in the MLX checkpoints on the Hub, stay in memory.

## Author's benchmarks: Transformers versus Strata on a MacBook M5

All figures here are the post author's own measurements. They ran Strata 0.1.60 with IQ3\_XXS experts, greedy decoding, thinking, MTP and speculative drafts off, and 1,024-token prefill chunks.

- **Decode, 200 tokens**: Transformers 10.0 tok/s against Strata's 14.8.
- **Prefill, 4K prompt**: 68 tok/s against 220.
- **Decode after a 6K document**: 7.3 tok/s against 13.7.
- **Memory**: about 18 GiB after loading, peaking at 25.4 GiB on a 6K prompt.

On an A10G, the author measured Transformers with Accelerate at 0.16 tok/s decode, against 20.1 for vLLM FP8 with `--cpu-offload-gb 16 --cpu-offload-params experts`. On OLMoE-1B-7B, which fits in memory, the slot-pool version reached 34.6 tok/s against 38.1 for vanilla. The author's rule: if the experts fit in RAM, llama.cpp is hard to beat; if not, stream only what the router picks through a bounded cache.

## Running Qwen3.8-Flash-Next on 32 GB: setup and cost

According to the PR's documentation, you install Transformers from the PR branch with `pip install "git+https://github.com/huggingface/transformers.git@refs/pull/50717/head"`. The commands come from the documentation listed under Sources; confirm them against the PR before running them. The PR's example uses `device_map={"": "mps", "experts.*": "disk"}`, `MetalConfig(bits=8)` and `experts_offload` with `cache_bytes` of `12 << 30`.

The 32 GB setup quantizes experts to 3 bits (49 GB on disk, 12 GiB cached on the device) and dense layers to 8 bits (about 5.4 GB, +0.2% perplexity versus BF16). The 51 GB n-gram embedding table stays on disk and rows are read when needed. Conversion takes about 5 hours, and the author has to close the browser or macOS starts swapping. The converted model is on the Hub as `tarekziade/Qwen3.8-Flash-Next-expert-streaming`; code and raw results are in the Space `tarekziade/moe-expert-streaming`.

## Who should try the Transformers streaming branch

If you run large MoE models on a Mac with limited memory, try the PR branch on a test machine first, since it is unreleased code. Keep a margin for memory: close other apps, and expect a long one-time conversion before the first run.

## What the post leaves open about speed and the API

The subtitle claims streaming gets within 6% of Strata on token generation, but the visible text is truncated before the final measurement, so that figure is unverified here. The Transformers API is still under discussion with maintainers, and quantized experts will probably go through the per-layer quantization plan of #50245.

## Sources

- [huggingface.co/blog/moe-expert-streaming](https://huggingface.co/blog/moe-expert-streaming?ref=ping2prompt.com)

---

This story was produced with AI assistance from the sources listed above and fact-checked against them before publication. Read our [editorial methodology](https://ping2prompt.com/about-how-we-work/) and the [corrections log](https://ping2prompt.com/corrections/). Spotted an error? Email [editor@ping2prompt.com](mailto:editor@ping2prompt.com).