Somewhere inside an IBM benchmark deck, a mixture-of-experts model served at $0.11 per million tokens. Ninety-eight percent prefix cache hit rate. Five to ten times cheaper than a commercial API. The slide went up, the room nodded, and nobody asked the only question that matters: what network topology is hiding inside that number?
That figure is now the reference price against which every decentralized inference network, every GPU-backed DePIN token, and every verifiable-compute pitch deck gets marked. The project behind it โ llm-d, which shipped 1.0 at PyTorch Conference on October 20 under CNCF Sandbox stewardship from Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA โ is not, on its face, a crypto story. It is a Kubernetes story. But the mechanism that produced $0.11 is precisely the mechanism that decentralized compute structurally cannot copy, and almost nobody in this sector has priced that in.
llm-d is not a model, not an inference engine, and not a serving framework. It is an orchestration layer that sits above vLLM and SGLang, below the cluster scheduler, and anchors itself to the Gateway API Inference Extension maintained under Kubernetes SIG Network. Founded in May 2025 and incubated in CNCF Sandbox, it is Apache-licensed and free. What it actually ships is three components that have existed in research and bespoke production forks for two years but were never standardized into something a platform team could install.
Those three components are: KV cache tiering across GPU memory, CPU, and remote storage; prefill/decode disaggregation across separate replica pools; and prefix-cache-aware routing that sends requests to whichever replica already holds the relevant context. The project's own framing is unusually honest for an infrastructure launch: serving a frontier model fast is no longer the hard part โ running it reliably, for many tenants, through upgrades, under real load, is.
That diagnosis holds up. PagedAttention plus continuous batching pushed single-node throughput to a place where it stopped being the binding constraint. The residual pain really is multi-tenancy, upgrade windows, and burst traffic. And the gap was real, not manufactured: DeepSeek's open inference code was a small-scale reference implementation, never a production service stack. Nothing was withheld. The distance between a reference implementation and a multi-tenant serving tier is the distance between a research paper and a payroll system.
What makes this release worth a crypto audience's attention is the launch cohort. Five independent commercial entities stood up managed layers on the same day: GKE Inference Gateway, Red Hat AI's subscription stack on OpenShift, AWS SageMaker HyperPod, DigitalOcean Serverless Inference, and Mistral's regional inference service. That is not a press release. That is five balance sheets independently concluding the same thing before the announcement went live.
Prefill is compute-bound. Decode is memory-bandwidth-bound. Co-locate them and GPU utilization collapses to whichever profile is weaker at that instant. This is the entire technical justification for disaggregation, and it is sound. The cost is that the KV cache โ gigabytes per session, for a long-context model โ must now migrate between pools at multi-gigabyte-per-second rates.
KV tiering addresses a different wall: context length grows, HBM does not. Offloading cold blocks to CPU or SSD or a remote tier is the only way to serve 128K-token sessions without buying memory you cannot amortize. Prefix-cache routing addresses a third: a multi-turn agent session either reuses its prior context or recomputes it from scratch, and the difference between those two paths is the difference between one prefill and forty.
Individually, each of these is a marginal improvement. Together they change the utilization profile of a cluster. That coupling is why llm-d chose to be an orchestration layer instead of shipping a faster kernel โ and it is also why the layer cannot be evaluated component by component.
Now the part the release notes do not discuss. Cross-node prefill/decode separation is a bandwidth problem before it is a software problem. It requires NVLink, InfiniBand, or a very well-provisioned RoCE fabric. The benefit of llm-d is a function of interconnect quality, not of the code. On commodity Ethernet โ which is what nearly every decentralized GPU marketplace actually runs, because that is what you get when you aggregate heterogeneous machines owned by strangers โ the KV transfer tax eats the gain before the scheduler gets a chance to be clever.
I have spent enough hours staring at distributed systems diagrams to know that topology is the variable people omit when they want a benchmark to look good. A cluster with 400G InfiniBand and a cluster with 10G Ethernet can run identical software and produce numbers that differ by an order of magnitude. The $0.11 slide does not tell you which one it came from. It almost certainly came from the first.
The second omitted constraint is break-even. Disaggregation is net-negative at low query rates, on short prompts, and under low concurrency: you pay for extra GPU instances you cannot amortize and you pay KV transfer overhead for a saving that never materializes. My working estimate, from having sized similar systems, is that the crossover requires a cluster in the dozens of GPUs with either long prompts or high prefix reuse. Nobody has published that threshold table. Its absence is not an oversight; it is a marketing decision.
Which brings us to the numbers themselves. Prefix cache hit rate moving from 35% to 70% on coding agents. TTFT down more than 35%. TPS up 70% on SageMaker HyperPod. Three-times output throughput on Llama 3.1 70B. Ninety-eight percent prefix hit rate. Five to ten times cheaper than commercial APIs. Every one of these is directionally plausible โ double your cache hit rate and you halve your KV recompute, so the latency and throughput gains follow arithmetically. Every one of them is vendor self-reported. Not a single third-party reproduction exists.
In 2019 I spent forty hours inside the Sapling circuit constraints and found a failure in large field element arithmetic that corrupted state silently under specific load. The lesson from that bounty was not that the code was bad. It was that the bug lives in the edge case, and a benchmark is a claim about the average case. A vendor publishing an average-case number is publishing the one measurement least likely to contain the thing that will break you.
There is a harder problem in the source material: the model identifiers. GLM-5.2 at 753B and Kimi K3 at trillion-parameter scale do not match any public lineage I can reconcile โ GLM-4.5 sits at 355B, Kimi K2 is a 1T MoE with 32B active. Those may be typos, internal codenames, or demo-environment artifacts. It does not matter which. Any performance figure attached to an unverifiable model identifier is an unverifiable figure, and this sector has a habit of pricing tokens on announced throughput.
Strip the numbers and what remains is a textbook value-capture structure: open upstream, monetized downstream. Kubernetes to EKS and GKE. Spark to Databricks. Postgres to RDS. Crypto has run the identical pattern โ client diversity monetized as hosted nodes, Cosmos SDK monetized as managed appchains. The margin never lives in the protocol. It lives in the operational gap between what the software does and what your team can keep running at 3 a.m.
Red Hat sells subscriptions against that gap. Google sells reliability. CoreWeave sells GPU-hours that get denser every time the orchestration layer gets smarter. Nobody in that set has an incentive to close the free layer, which is why the free layer will stay free. This is alignment, not coordination โ five parties converging on the same conclusion because the same conclusion pays each of them.
The most commercially loaded line in the entire release is buried in a customer reference: DigitalOcean serving a trillion-parameter mixture-of-experts across a mixed AMD and NVIDIA fleet. That is a live demonstration that CUDA lock-in is negotiable at the inference layer. For any token narrative built on an NVIDIA scarcity premium, it is a slow-acting negative that will take eighteen months to show up in a chart and about six months to show up in a procurement conversation.
And then there is the TorchTPU figure โ duty cycle climbing from 40% to 70% through cooperative time-slicing. That number is doing more work than the article admits. It implies llm-d's value proposition has already expanded from serving online inference to raising accelerator utilization on training-side workloads like RL rollout. If that holds, the addressable market stops being inference orchestration and starts being training orchestration. The throughput cost of time-slicing โ context-switch overhead, memory fragmentation โ is not disclosed anywhere I can find.
Composability isn't a feature flag you toggle on a gateway. It's a ecosystem property, and it only exists when every layer beneath you honors the same contract. That is why the Gateway API Inference Extension matters more than any individual optimization in this release: it puts the orchestration layer inside a standard, which is the only durable moat available to software that anyone can fork.
Now the part that should worry anyone building a verifiable-compute product. Multi-tenant prefix caching means tenant A's KV blocks influence tenant B's timing. Cache hit rates and time-to-first-token become an observable side channel โ an adversary co-located on the same replica pool can infer prompt structure, session length, and workload shape from latency alone. The release discusses tenant isolation as a resource-quota problem. It is not. It is a shared-memory problem with a routing policy on top.
This is the same failure class as a shared mempool, and the crypto industry should recognize it instantly. Information leaks through timing far more readily than through data. A system can be perfectly encrypted and still tell you everything about its neighbors.
Which exposes the gap in most 'verifiable inference' designs on the market. They verify an output hash against a model weight hash. They do not verify that the KV cache was not poisoned by a co-tenant, that the routing decision was honest, or that your decode path was not quietly shared with a higher-paying customer. Verifying a hash is not verifying a computation. It is verifying that a computation happened somewhere and produced this string.
We don't have decentralized inference scheduling. We have a coordinator with a staking requirement. The scheduler โ the component that decides which replica receives which prefix, which is exactly where the economic value now concentrates โ needs a global view of cache state across the pool. A global view is a coordinator. You can federate it, you can shard it, you can attest to it, but the moment you need sub-millisecond decisions on cache locality, you have re-centralized the hardest part and put a token on top of it.
That is the structural reason decentralized inference networks should be reading this release with a specific kind of dread. The free orchestration layer just removed the engineering moat that justified a large share of their valuations, and it did so while simultaneously demonstrating that the replacement layer depends on fabric quality those networks do not have. Both halves of the squeeze are technical, and neither is fixable with incentives.
So here is the question I would put to every team in this category before their next unlock: publish your KV-transfer latency budget, and publish your prefill/decode break-even table. Not your throughput. Not your token price. The two numbers that determine whether your architecture is viable on the hardware your suppliers actually own.
The one development that genuinely changes the picture is a credible KV transport layer outside the NVIDIA stack โ a NIXL equivalent that runs on non-NVIDIA silicon and over ordinary networks. The moment that exists, the topology argument weakens, the fabric assumption stops being a moat, and decentralized compute gets its first real opening. Until then, a free layer is doing what free layers always do: moving the margin somewhere the token cannot reach.