Cloudflare Cuts Kimi and GLM Costs 30% on Workers AI

On August 3, 2026, Cloudflare published a technical breakdown of how it now serves two of the largest open-weight models, Moonshot AI's Kimi K2.6 and Z.ai's GLM 5.2, at roughly 30% lower cost per token on Workers AI. The post details three inference optimizations that fit more of each model onto the same GPUs without changing the answers those models produce. The full engineering write-up is on the Cloudflare blog, and it drew a long thread on Hacker News.

For builders running open-weight models in production, this is a rare look at the exact levers that decide whether a 1-trillion-parameter model is affordable to serve. The headline numbers are concrete: doubling usable cache capacity on Kimi K2.6, a 40% smaller checkpoint for GLM 5.2, and single-digit accuracy movement across standard benchmarks. Here is what shipped and what it means if you build on these models.

What Cloudflare shipped

All three changes run inside Workers AI, Cloudflare's inference service that hosts models on GPUs in its own data centers. The serving stack runs on NVIDIA H200 GPUs using the open-source SGLang framework, with the prefill phase (reading your prompt) and the decode phase (writing the response) split into separate resource pools so each can be tuned independently.

The two models in focus are both frontier open-weight releases. Kimi K2.6, from Moonshot AI, is a 1-trillion-parameter mixture-of-experts model with 32 billion active parameters, a 262K context window, vision, and agentic tool use. GLM 5.2, from Z.ai, is a comparably large model whose raw checkpoint weighs in at 705 GB. Serving models this size is expensive, so the entire exercise is about packing more work onto fixed hardware.

The three optimizations, side by side

Each technique targets a different bottleneck. One shrinks the memory the running conversation uses, one shrinks the model file itself, and one guards against a subtle correctness bug that only shows up under heavy batching.

The FP8 KV cache is the biggest single lever for long-context and high-concurrency workloads. Because the key-value cache holds every token of the running conversation, halving its footprint lets the same GPUs hold about twice as many tokens. At 64 concurrent requests, the FP8 build sustained 2,192 tokens per second where the BF16 build simply ran out of memory. Cloudflare keeps prefill in BF16, where precision matters most, and only switches the decode cache to FP8. You can read the model details on the Kimi K2.6 model page.

The INT4 weight compression attacks the other cost driver, the size of the model file loaded onto every GPU. Squeezing GLM 5.2 down to 4-bit integers frees roughly 36 GB per GPU across an 8-way tensor-parallel deployment, which in turn opens room for about 1.18 million tokens of KV cache on the same box. Decode throughput rises 55% at a single request, 21% at eight, and 16% at 64. The catch is that INT4 wins on decode but loses on prefill (8,660 tokens per second versus 10,160 for FP8), so Cloudflare runs INT4 for decode and FP8 for prefill.

The integrity check is the "safer" in the title. When you pack cache pages this tightly and reallocate them across thousands of concurrent requests, a mis-mapped page can silently corrupt a response. Cloudflare tags each page and verifies the mapping before any decode step reads it, running the check as a separate batch to avoid GPU thread-group races. The default is a no-op tracker with no measurable overhead.

Accuracy: did the answers change?

Aggressive quantization is only useful if the model still gives the same answers. Cloudflare put it plainly: "None of this would matter if it changed the model's answers, so we checked." The FP8 KV cache results on Kimi K2.6 land within noise of the BF16 baseline.

The GLM 5.2 INT4 weights held up too, with every benchmark staying within 0.8 percentage points of the FP8 version. Cloudflare describes the two caches as "indistinguishable" across its evaluation suite. In other words, the cost savings come from packing, not from cutting corners on quality.

What this enables for builders

If you are building agents or long-context apps on open-weight models, the practical payoff is capacity. A doubled KV cache means more simultaneous users, longer documents, and deeper agent traces before you hit the memory wall that forces a bigger, pricier GPU allocation. Cheaper decode also lowers the per-token cost of the exact workloads that dominate agent bills: many short tool calls and long reasoning chains.

This is the same economic argument behind the wave of open-weight routing and pooling tools we have covered, such as Echo, which pools open-weight models at about a third of Claude's cost. Serving-side efficiency compounds with routing: if the underlying inference is 30% cheaper, every layer you build on top inherits that savings. It also lowers the bar for self-hosting the newest giants like Kimi K3, the first open 3T-class model.

How to run Kimi and GLM on Workers AI

You do not have to reimplement any of this. The optimizations are already live on Cloudflare's hosted endpoints. To start:

1. Upgrade to a Workers Paid plan. Both Kimi K2.6 and GLM 5.2 require it because of their size.
2. Create an API token in the Cloudflare dashboard and note your account ID.
3. Open the Workers AI model catalog, pick Kimi K2.6 or GLM 5.2, and copy the exact model ID from its model page rather than guessing it.
4. Call the model over the REST API or a Worker binding, passing your prompt as a standard chat completion request.
5. For agent and long-context workloads, send requests concurrently so you capture the throughput headroom the FP8 cache unlocks, and check per-token rates on the Workers AI pricing page before you scale.

Frequently asked questions

Does FP8 or INT4 quantization make Kimi or GLM less accurate?

Not meaningfully. Cloudflare's published tables show FP8 KV cache on Kimi K2.6 moving less than a point on GSM8K, MMLU, and ARC, and GLM 5.2 INT4 weights staying within 0.8 percentage points of FP8. Tool-call validity actually ticked up slightly.

Which optimization matters most for my workload?

The FP8 KV cache helps most if you run long contexts or many concurrent users, because it doubles how much conversation the GPUs can hold. INT4 weight compression helps most when the model file itself is the memory bottleneck, as with GLM 5.2's 705 GB checkpoint.

Do I need to implement quantization myself?

No. These changes run inside Cloudflare Workers AI on its hosted Kimi and GLM endpoints. You call the standard API and inherit the efficiency automatically.

Why does Cloudflare use INT4 for decode but FP8 for prefill?

Because each phase has a different bottleneck. INT4 weights speed up decode by shrinking memory traffic, but prefill runs faster in FP8 (10,160 versus 8,660 tokens per second). Running a hybrid captures the best of both.

What is the KV cache integrity check protecting against?

Under heavy concurrency, cache pages are constantly reallocated. A mis-mapped page could feed one request another request's cached tokens. The tag-and-verify step catches that before decode reads the cache, at under 1% throughput cost.

Are these the same models I can download and self-host?

Yes. Kimi K2.6 and GLM 5.2 are open-weight releases from Moonshot AI and Z.ai. Cloudflare's optimizations are about serving them efficiently; the weights themselves are the public models you can also run on your own hardware.

Related deep dives

For more on running open-weight models affordably, see our breakdown of Kimi K3, the world's first open 3T-class model, and how Echo pools open-weight models at a fraction of Claude's cost.