A Response to Apple's "The Illusion of Thinking"
Last week Apple’s machine-learning group released “The Illusion of Thinking,” a forty-page study showing that large reasoning models stumble once problem depth exceeds what their fixed hidden states can juggle.
Working with controlled puzzles like Tower of Hanoi, the authors traced a sudden drop in chain-of-thought tokens and accuracy, even when plenty of the context window remained.
Specifically, Apple states that large language models hit a wall once they have “thought” for a few‑hundred tokens.
While our team can (somewhat) agree, our experience in AI production is more optimistic.
In The Illusion of Thinking, Apple’s researchers show that on many reasoning tasks extra chain‑of‑thought tokens help only up to a point.
Essentially, beyond roughly a few‑hundred tokens, 0.2 × hidden‑width for the models they tested, accuracy stops improving and sometimes falls. They attribute the cliff to a fixed‑width hidden state that must keep compressing its own intermediate reasoning, so every new token steals capacity from the past.
Computation keeps running while quality plateaus.
Atlas Cloud agrees that wider context windows alone cannot buy deeper reasoning forever. Yet we see today’s wall less as a hard law and more as the point where infrastructure cost makes long chains of thought impractical.
Our inference platform is designed to push that point outward. Prefill/Decode (PD) disaggregation keeps the compute‑bound prefill phase on dedicated GPUs while memory‑bound decoding runs elsewhere. This can reduce the back and forth that Apple implicitly assumes in a monolithic server.
Expert‑parallel load balancing and two‑batch overlap raise decode throughput and cut tail‑latency, so a model can afford to “think” longer without timing out.
For example, on a 12‑node H100 cluster we sustain roughly 51.7 k prefill TPS and 22.5 k decode TPS per node, well above published baselines of most reasoning models.
Higher per‑token throughput means a given budget can fund a longer chain of thought before the user notices delay. In practice we see tasks where allowing the model 2 to 3X more reasoning tokens, still well under Apple’s cliff, to lift factual correctness without changing weights.
Atlas Inference is our managed service that bakes these optimizations into the job scheduler, exposes a simple gRPC endpoint, and auto‑scales across heterogeneous GPU pools.
The same PD split that boosts today’s models also slots naturally into upcoming architectures that maintain external episodic memory or retrieve vector‑store chunks mid‑generation.
This means teams can experiment with memory‑augmented or sparse‑moe transformers,
opening space to test whether Apple’s cliff moves when the infrastructure pressure is removed.
Apple has drawn attention to a very real compression problem inside today’s transformers, but we believe that it’s a temporary growing pain of the industry (hot take, we know).
Our recent breakthrough in AI inference already stretches the viable token budget by splitting prefill from decode and keeping every node busy, a design that clocks >50 k prefill TPS and >22 k decode TPS per node in production tests.
As throughput climbs and marginal GPU minutes get cheaper, teams can run longer chains of thought, add external memory, and layer on verifier passes. Each of these step eat away at the collapse Apple measured.
So we see the “inference-time scaling limit” as a growing pain that will quickly become a non-issue within the industry.
The Inference Time Scaling Problem
What Apple means by an “inference‑time scaling limit”
Why we think infrastructure matters
Inference is for tomorrow’s memory‑rich models
So is the “Inference Time Scaling Problem,” a problem?
Technology
