Be a part of the occasion trusted by enterprise leaders for practically 20 years. VB Remodel brings collectively the folks constructing actual enterprise AI technique. Be taught extra
Apple’s machine-learning group set off a rhetorical firestorm earlier this month with its launch of “The Illusion of Thinking,” a 53-page analysis paper arguing that so-called massive reasoning fashions (LRMs) or reasoning massive language fashions (reasoning LLMs) akin to OpenAI’s “o” sequence and Google’s Gemini-2.5 Professional and Flash Pondering don’t really have interaction in unbiased “thinking” or “reasoning” from generalized first rules discovered from their coaching information.
As an alternative, the authors contend, these reasoning LLMs are literally performing a sort of “pattern matching” and their obvious reasoning potential appears to disintegrate as soon as a process turns into too advanced, suggesting that their structure and efficiency is just not a viable path to bettering generative AI to the purpose that it’s synthetic generalized intelligence (AGI), which OpenAI defines as a mannequin that outperforms people at most economically useful work, or superintelligence, AI even smarter than human beings can comprehend.
ACT NOW: Come talk about the newest LLM advances and analysis at VB Remodel on June 24-25 in SF — restricted tickets out there. REGISTER NOW
Unsurprisingly, the paper instantly circulated broadly among the many machine studying group on X and lots of readers’ preliminary reactions had been to declare that Apple had successfully disproven a lot of the hype round this class of AI: “Apple just proved AI ‘reasoning’ models like Claude, DeepSeek-R1, and o3-mini don’t actually reason at all,” declared Ruben Hassid, creator of EasyGen, an LLM-driven LinkedIn put up auto writing device. “They just memorize patterns really well.”
However now at the moment, a brand new paper has emerged, the cheekily titled “The Illusion of The Illusion of Thinking” — importantly, co-authored by a reasoning LLM itself, Claude Opus 4 and Alex Lawsen, a human being and unbiased AI researcher and technical author — that features many criticisms from the bigger ML group concerning the paper and successfully argues that the methodologies and experimental designs the Apple Analysis staff used of their preliminary work are essentially flawed.
Whereas we right here at VentureBeat are usually not ML researchers ourselves and never ready to say the Apple Researchers are mistaken, the talk has actually been a full of life one and the difficulty concerning the capabilities of LRMs or reasoner LLMs in comparison with human pondering appears removed from settled.
How the Apple Analysis research was designed — and what it discovered
Utilizing 4 traditional planning issues — Tower of Hanoi, Blocks World, River Crossing and Checkers Leaping — Apple’s researchers designed a battery of duties that compelled reasoning fashions to plan a number of strikes forward and generate full options.
These video games had been chosen for his or her lengthy historical past in cognitive science and AI analysis and their potential to scale in complexity as extra steps or constraints are added. Every puzzle required the fashions to not simply produce an accurate closing reply, however to clarify their pondering alongside the way in which utilizing chain-of-thought prompting.
Because the puzzles elevated in issue, the researchers noticed a constant drop in accuracy throughout a number of main reasoning fashions. In probably the most advanced duties, efficiency plunged to zero. Notably, the size of the fashions’ inside reasoning traces—measured by the variety of tokens spent pondering by means of the issue—additionally started to shrink. Apple’s researchers interpreted this as an indication that the fashions had been abandoning problem-solving altogether as soon as the duties turned too exhausting, primarily “giving up.”
The timing of the paper’s launch, simply forward of Apple’s annual Worldwide Builders Convention (WWDC), added to the impression. It shortly went viral throughout X, the place many interpreted the findings as a high-profile admission that current-generation LLMs are nonetheless glorified autocomplete engines, not general-purpose thinkers. This framing, whereas controversial, drove a lot of the preliminary dialogue and debate that adopted.
Critics take purpose on X
In a single broadly shared put up, Lisan argued that the Apple staff conflated token price range failures with reasoning failures, noting that “all models will have 0 accuracy with more than 13 disks simply because they cannot output that much!”
For puzzles like Tower of Hanoi, he emphasised, the output dimension grows exponentially, whereas the LLM context home windows stay fastened, writing “just because Tower of Hanoi requires exponentially more steps than the other ones, that only require quadratically or linearly more steps, doesn’t mean Tower of Hanoi is more difficult” and convincingly confirmed that fashions like Claude 3 Sonnet and DeepSeek-R1 usually produced algorithmically appropriate methods in plain textual content or code—but had been nonetheless marked mistaken.
One other put up highlighted that even breaking the duty down into smaller, decomposed steps worsened mannequin efficiency—not as a result of the fashions failed to grasp, however as a result of they lacked reminiscence of earlier strikes and technique.
“The LLM needs the history and a grand strategy,” he wrote, suggesting the true drawback was context-window dimension somewhat than reasoning.
Others echoed that sentiment, noting that human drawback solvers additionally falter on lengthy, multistep logic puzzles, particularly with out pen-and-paper instruments or reminiscence aids. With out that baseline, Apple’s declare of a basic “reasoning collapse” feels ungrounded.
A number of researchers additionally questioned the binary framing of the paper’s title and thesis—drawing a tough line between “pattern matching” and “reasoning.”
Alexander Doria aka Pierre-Carl Langlais, an LLM coach at vitality environment friendly French AI startup Pleias, mentioned the framing misses the nuance, arguing that fashions is perhaps studying partial heuristics somewhat than merely matching patterns.
Okay I suppose I’ve to undergo that Apple paper.
My principal subject is the framing which is tremendous binary: “Are these models capable of generalizable reasoning, or are they leveraging different forms of pattern matching?” Or what in the event that they solely caught real but partial heuristics. pic.twitter.com/GZE3eG7WlM
— Alexander Doria (@Dorialexander) June 8, 2025
Ethan Mollick, the AI centered professor at College of Pennsylvania’s Wharton Faculty of Enterprise, known as the concept that LLMs are “hitting a wall” untimely, likening it to comparable claims about “model collapse” that didn’t pan out.
Briefly, whereas Apple’s research triggered a significant dialog about analysis rigor, it additionally uncovered a deep rift over how a lot belief to put in metrics when the take a look at itself is perhaps flawed.
A measurement artifact, or a ceiling?
In different phrases, the fashions might have understood the puzzles however ran out of “paper” to put in writing the total answer.
“Token limits, not logic, froze the models,” wrote Carnegie Mellon researcher Rohan Paul in a broadly shared thread summarizing the follow-up assessments.
But not everybody is able to clear LRMs of the cost. Some observers level out that Apple’s research nonetheless revealed three efficiency regimes — easy duties the place added reasoning hurts, mid-range puzzles the place it helps, and high-complexity circumstances the place each normal and “thinking” fashions crater.
Others view the talk as company positioning, noting that Apple’s personal on-device “Apple Intelligence” fashions path rivals on many public leaderboards.
The rebuttal: “The Illusion of the Illusion of Thinking”
In response to Apple’s claims, a brand new paper titled “The Illusion of the Illusion of Thinking” was launched on arXiv by unbiased researcher and technical author Alex Lawsen of the nonprofit Open Philanthropy, in collaboration with Anthropic’s Claude Opus 4.
The paper straight challenges the unique research’s conclusion that LLMs fail because of an inherent incapability to cause at scale. As an alternative, the rebuttal presents proof that the noticed efficiency collapse was largely a by-product of the take a look at setup—not a real restrict of reasoning functionality.
Lawsen and Claude show that lots of the failures within the Apple research stem from token limitations. For instance, in duties like Tower of Hanoi, the fashions should print exponentially many steps — over 32,000 strikes for simply 15 disks — main them to hit output ceilings.
The rebuttal factors out that Apple’s analysis script penalized these token-overflow outputs as incorrect, even when the fashions adopted an accurate answer technique internally.
The authors additionally spotlight a number of questionable process constructions within the Apple benchmarks. A number of the River Crossing puzzles, they word, are mathematically unsolvable as posed, and but mannequin outputs for these circumstances had been nonetheless scored. This additional calls into query the conclusion that accuracy failures characterize cognitive limits somewhat than structural flaws within the experiments.
To check their concept, Lawsen and Claude ran new experiments permitting fashions to offer compressed, programmatic solutions. When requested to output a Lua perform that would generate the Tower of Hanoi answer—somewhat than writing each step line-by-line—fashions all of a sudden succeeded on much more advanced issues. This shift in format eradicated the collapse fully, suggesting that the fashions didn’t fail to cause. They merely failed to evolve to a man-made and overly strict rubric.
Why it issues for enterprise decision-makers
The back-and-forth underscores a rising consensus: analysis design is now as vital as mannequin design.
Requiring LRMs to enumerate each step might take a look at their printers greater than their planners, whereas compressed codecs, programmatic solutions or exterior scratchpads give a cleaner learn on precise reasoning potential.
The episode additionally highlights sensible limits builders face as they ship agentic programs—context home windows, output budgets and process formulation could make or break user-visible efficiency.
For enterprise technical resolution makers constructing purposes atop reasoning LLMs, this debate is greater than educational. It raises essential questions on the place, when, and belief these fashions in manufacturing workflows—particularly when duties contain lengthy planning chains or require exact step-by-step output.
If a mannequin seems to “fail” on a fancy immediate, the issue might not lie in its reasoning potential, however in how the duty is framed, how a lot output is required, or how a lot reminiscence the mannequin has entry to. That is significantly related for industries constructing instruments like copilots, autonomous brokers, or decision-support programs, the place each interpretability and process complexity may be excessive.
Understanding the constraints of context home windows, token budgets, and the scoring rubrics utilized in analysis is crucial for dependable system design. Builders might have to contemplate hybrid options that externalize reminiscence, chunk reasoning steps, or use compressed outputs like features or code as a substitute of full verbal explanations.
Most significantly, the paper’s controversy is a reminder that benchmarking and real-world software are usually not the identical. Enterprise groups must be cautious of over-relying on artificial benchmarks that don’t mirror sensible use circumstances—or that inadvertently constrain the mannequin’s potential to show what it is aware of.
Finally, the large takeaway for ML researchers is that earlier than proclaiming an AI milestone—or obituary—be sure that the take a look at itself isn’t placing the system in a field too small to suppose inside.
Each day insights on enterprise use circumstances with VB Each day
If you wish to impress your boss, VB Each day has you lined. We provide the inside scoop on what firms are doing with generative AI, from regulatory shifts to sensible deployments, so you’ll be able to share insights for max ROI.
An error occured.
