Hugging Face shipped direct GGUF support in Transformers on 22 September 2026, letting the library load a quantized .gguf checkpoint straight from the Hub and run it through optimized ggml Metal kernels. The post, written by Marc Sun, Arthur Zucker and Lysandre Debut, benchmarks three Qwen checkpoints on a MacBook Pro M2 Max with 32GB of unified memory and reports Transformers landing within roughly 2% to 14% of llama.cpp depending on the model. The headline everyone will take from it is parity. The more useful detail is a one-sentence footnote under the chart, plus four constraints that decide whether any of this is usable in your setup today.
What Hugging Face shipped
Until now, running a GGUF quant meant leaving the Python ecosystem. You prototyped in Transformers and served with llama.cpp, Ollama or LM Studio, which meant two toolchains, two config formats and two sets of sampling defaults. The new release collapses that into one call: pass a gguf_file argument to from_pretrained() and the quantized weights load directly, without an intermediate dequantization step to BF16.
The practical size difference is the reason anyone bothers. For Qwen3.5-4B, the unquantized BF16 reference is 8.42GB. The Q4_K_M quant is 2.74GB, which is 67% smaller. Q5_K_M sits at 3.14GB and Q6_K at 3.53GB. On a 16GB Mac, that is the difference between a model that swaps and a model that runs, and it is why the GGUF format became the default distribution route for local inference in the first place.
A second piece ships alongside it. Quantized models can now be served through transformers serve behind an OpenAI-compatible API, so an application already pointed at a chat-completions endpoint can be repointed at a local GGUF checkpoint without touching the client code.
The benchmark numbers, and the footnote that reframes them
Three checkpoints were measured, all at Q4_K_M, all on the same M2 Max. Transformers reached roughly 23 to 24 tokens per second on Qwen3.5-4B against llama.cpp's 24. On the Qwen3.8-32B mixture-of-experts checkpoint it hit 18 against 19. On the dense Qwen3.5-32B it managed 9 to 10 against 11.
Read as a straight table, that is a small and consistent deficit. But the post attaches a caveat that most coverage will drop, and it points the other way: "The chart uses the same measurements described above; it does not imply identical benchmark conditions, since the Transformers measurement includes prefill while llama-bench reports decode-only throughput."
That distinction matters. Prefill is the pass over your input prompt before the first output token appears. Folding that time into a tokens-per-second figure drags the number down, because you are dividing generated tokens by a total that includes work which produced none of them. llama.cpp's figure excludes it. So the two columns are not measuring the same thing, and the bias runs in llama.cpp's favour. Transformers is being scored more harshly than its competitor, and still lands close. Nobody has yet published a like-for-like decode-only comparison, so the true gap is unknown, and it is probably smaller than the chart implies rather than larger. This is a rare case where the vendor's own numbers understate the vendor's result, and they flagged it themselves.
Where the gap actually widens
The three results are not equally close, and the pattern is worth naming because it predicts which of your own models will behave well. The percentages below are our arithmetic on the published figures, using the midpoint where a range was given.
| Checkpoint (Q4_K_M) | Transformers | llama.cpp | Gap |
|---|---|---|---|
| Qwen3.5-4B (dense, small) | 23-24 tok/s | 24 tok/s | about 2% |
| Qwen3.8-32B (mixture-of-experts) | 18 tok/s | 19 tok/s | about 5% |
| Qwen3.5-32B (dense, large) | 9-10 tok/s | 11 tok/s | about 14% |
The deficit is negligible on the small dense model, modest on the sparse 32B, and widest on the dense 32B. The MoE checkpoint activates a fraction of its parameters per token, so it behaves closer to a small model than its parameter count suggests, which is consistent with the 5% result sitting between the other two. If you are running a large dense model, expect the largest penalty. If you are running a small model or a sparse one, expect the difference to be invisible in normal use.
The four limits that decide whether this is for you
This is the part the announcement is honest about and the aggregators will compress away.
First, and most restrictive: "The packed inference path is MPS-only for now." MPS is Apple's Metal Performance Shaders backend. On an NVIDIA or AMD machine, this specific fast path is not available to you today.
Second, architecture coverage is narrow. In Hugging Face's words, "The packed loader currently covers the Qwen3.5 dense and MoE architectures, including compatible Qwen3.8 checkpoints." Llama, Mistral, Gemma and the rest of the GGUF universe are not in scope yet.
Third, this is not in a tagged release. The install instruction points at the development branch: pip install -U "git+https://github.com/huggingface/transformers.git" kernels, described as "main for now, until the next release". The kernels package it depends on comes from the Hugging Face kernels hub.
Fourth, batching is uneven. Padded batches "cannot take the same shortcut and can have lower performance", so a throughput-oriented pipeline that batches ragged inputs will not see the single-stream numbers above.
Try it in ten minutes
The test below is small enough to abandon cheaply if the constraints rule you out. You need an Apple Silicon Mac and about 3GB of disk.
- Install from the development branch:
pip install -U "git+https://github.com/huggingface/transformers.git" kernels - Pick a checkpoint. unsloth/Qwen3.5-4B-GGUF is the one Hugging Face benchmarks, and it carries 18 GGUF variants, well beyond the four the blog post lists as size references.
- Load it by filename. Pass
gguf_file="Qwen3.5-4B-Q4_K_M.gguf"to bothAutoTokenizer.from_pretrained()andAutoModelForCausalLM.from_pretrained(), using the model idunsloth/Qwen3.5-4B-GGUF. The tokenizer needs the same argument, which is the step most people miss. - Time a generation, then time the same prompt under llama.cpp. Do not compare your number against the blog's chart, because you will be measuring prefill and it will not be.
- If you want it behind an API, install the serving extra and run
transformers serve, then point any OpenAI-compatible client at the local endpoint.
Start at Q4_K_M. If output quality disappoints on your task, step up to Q5_K_M or Q6_K and accept roughly 0.4GB to 0.8GB more on disk before concluding the model is unsuitable. Quantization damage is task-dependent, and a quant that is fine for summarization can fall apart on structured output.
Who should switch, and who should not
Switch if you already live in Python, your target is Apple Silicon, and your model is a Qwen3.5 or compatible Qwen3.8 checkpoint. Collapsing two runtimes into one removes a class of bug that has nothing to do with your product: tokenizer mismatches, divergent sampling defaults and chat templates that render differently in each stack. The GGUF documentation in Transformers covers the loading path in more detail.
Stay on llama.cpp if you ship to Windows or Linux, if you depend on a non-Qwen architecture, if you need the dense-32B throughput ceiling, or if pinning to a tagged release is a requirement rather than a preference. Nothing here deprecates llama.cpp, and the ggml Metal kernels doing the work are its kernels.
The strategic read is narrower than "Transformers replaced llama.cpp" and more interesting. The GGUF format is becoming a neutral interchange layer rather than the property of one runtime, and the same week brought oMLX maintainer Jun Kim joining Hugging Face to work on Apple's MLX stack. Two moves, one direction: local inference on Mac hardware is being pulled into the mainstream Python tooling instead of living beside it.
Frequently asked questions
Does this work on an NVIDIA GPU?
Not on this fast path. Hugging Face states the packed inference path is MPS-only for now, meaning Apple Silicon. CUDA users should continue with llama.cpp or the existing Transformers quantization backends.
Which models can I load today?
The packed loader covers Qwen3.5 dense and mixture-of-experts architectures plus compatible Qwen3.8 checkpoints. Other GGUF families are not yet supported through this path.
Is Transformers now as fast as llama.cpp?
Close, but the published comparison is not like-for-like. The Transformers figure includes prefill while llama.cpp's llama-bench reports decode-only throughput, which penalises the Transformers number. Measured gaps ran from about 2% on a 4B model to about 14% on a dense 32B, and the real decode-only gap is likely smaller.
Which quantization level should I pick?
Q4_K_M is the recommended starting point at 2.74GB for Qwen3.5-4B, against 8.42GB for BF16. Q5_K_M (3.14GB) and Q6_K (3.53GB) trade disk for precision if quality on your specific task is not good enough.
Can I serve a GGUF model through an API?
Yes. Install the serving extra and run transformers serve, which exposes an OpenAI-compatible endpoint, so existing clients work without code changes.
Is it safe to use in production?
Not yet, if you value stability. The feature currently requires installing Transformers from the git main branch rather than a tagged release, and batching behaviour is still uneven for padded inputs.
