Research Brief
How Compute Scaling Forces New Efficiency Designs
How physical limits on compute scaling are forcing design shifts in model design, deployment, and enterprise cost structure.
How physical limits on compute scaling are forcing design shifts in model design, deployment, and enterprise cost structure.
Brute-force compute scaling is hitting three walls simultaneously. Training costs are doubling every five months, with a single frontier model now running upwards of $170 million. The energy required to train these models has reached the equivalent annual carbon output of nearly 500 people for one training run. And the performance returns on that investment are collapsing: the gap between the top-ranked model and the tenth-ranked model on public benchmarks has fallen from 11.9 percent to just 5.4 percent in a single year, with the gap between first and second place now under one percent.
These three forces are not operating in isolation. Their simultaneous convergence creates an inflection point that no amount of capital investment can overcome through scaling alone. The question facing enterprise leaders is not whether this shift will happen, but whether they will recognize it early enough to make better infrastructure and vendor decisions over the next 24 months.
New designs, specifically the Titans design from Google Research and the MIRAS generative framework, provide the first empirical evidence that this transition is not merely theoretical. These systems demonstrate that models capable of continual learning, selective memory, and real-time adaptation can match or exceed the accuracy of scale-dependent transformers while operating at a fraction of the computational cost. Enterprise leaders who understand this design shift will be positioned to evaluate vendors, plan infrastructure, and allocate talent with substantially greater precision than those who continue to equate model quality with parameter count alone.
The economics of training frontier AI models have entered a phase of exponential acceleration that is fundamentally unlike any previous technology cost curve. Training costs are not growing linearly or even at a steady compound rate. They are doubling approximately every five months. To appreciate the magnitude: the original Transformer design in 2017 cost roughly $670 to train. By 2019, training RoBERTa Large required approximately $160,000. Four years later, OpenAI's GPT-4 reportedly cost around $79 million. And Meta's Llama 3.1-405B, released in 2024, required an estimated $170 million in compute alone. That is a cost increase of more than five orders of magnitude in seven years.
This trajectory is not sustainable by any reasonable financial model. Even the most well-capitalized technology companies on Earth are beginning to confront the arithmetic: if training costs continue doubling at this rate, the next generation of frontier models could require budgets approaching or exceeding one billion dollars for a single training run. The capital requirements are concentrating AI development in fewer and fewer hands. According to the Stanford HAI 2025 AI Index Report, nearly 90 percent of notable AI models released in 2024 came from industry rather than academia, a stark reversal from a decade ago when university research labs drove most foundational advances.
The data requirements are scaling in parallel. Large language model training datasets are doubling in size every eight months. Meta's Llama 3.3 was trained on roughly 15 trillion tokens, a number that exceeds the entire digitized text output of human civilization by some estimates. The implication is severe: the cost of acquiring, cleaning, and curating training data is becoming a significant fraction of total training expenditure, and the available supply of high-quality training data is approaching finite limits. Some researchers have begun to refer to this as "peak data," the point at which further scaling of training datasets produces diminishing returns not because the data is unavailable, but because the marginal value of additional data decreases while the cost of processing it does not.
The concentration of training capability creates a vendor dependency that enterprise leaders must reckon with. When only a handful of organizations can afford to train frontier models, every downstream consumer of AI capability, from enterprise software vendors to government agencies, becomes dependent on the design choices and strategic priorities of those few providers. The concentration creates a design dependency that extends far beyond market share: the design decisions embedded in today's frontier models propagate through every application built on top of them. If those foundational designs are optimized primarily for scale rather than efficiency, every downstream deployment inherits that cost structure.
The energy cost of training frontier AI models is scaling even faster than the financial cost, and the environmental implications are becoming impossible for enterprise leaders to ignore. The power required to train a single frontier model now doubles annually. The carbon footprint data tells a story of exponential acceleration: AlexNet in 2012 produced approximately 0.01 tons of CO2. GPT-3 in 2020 generated 588 tons. GPT-4 in 2023 required an estimated 5,184 tons. And Meta's Llama 3.1-405B in 2024 produced approximately 8,930 tons of carbon emissions during training alone.
To translate those numbers into human terms: the average American produces roughly 18 tons of carbon per year across all activities, including transportation, housing, food, and consumption. A single training run of Llama 3.1-405B produced the equivalent annual carbon output of nearly 500 people. This is the carbon cost of building one model, one time. It does not include the energy required to run inference at scale, which for widely deployed models can exceed training costs within months of deployment. Nor does it account for the multiple failed training runs, hyperparameter sweeps, and experimental iterations that precede a successful model release.
The technology industry has recognized this trajectory and is responding with extraordinary measures. Microsoft, Google, and Amazon have all announced or finalized agreements to secure nuclear energy capacity specifically for AI training and inference workloads. These are not incremental energy procurement decisions. They represent a fundamental acknowledgment that the current scaling trajectory cannot be sustained by renewable energy sources alone, and that the energy demands of frontier AI development are large enough to justify the capital expenditure, regulatory complexity, and multi-year timelines associated with nuclear power generation.
For enterprise leaders, the energy dimension of AI scaling introduces a category of risk that most technology evaluation frameworks have not yet been built to assess. Sustainability reporting requirements are tightening across jurisdictions. The European Union's Corporate Sustainability Reporting Directive, the SEC's proposed climate disclosure rules, and similar regulatory frameworks are moving toward requiring organizations to account for the energy consumption and carbon intensity of their technology infrastructure, including cloud-based AI services. Organizations that are heavily dependent on AI capabilities delivered by providers whose training and inference infrastructure carries high carbon intensity may find themselves facing disclosure requirements, reputational risks, and regulatory exposure that were not anticipated when those vendor relationships were established.
The energy equation also introduces a physical constraint that financial capital alone cannot resolve. Unlike compute cost, which can theoretically be addressed by spending more money, energy availability is bounded by the physical infrastructure of power generation and transmission. Data centers cannot consume power that the grid cannot deliver. The race to secure nuclear energy agreements is itself evidence that the scaling trajectory has exceeded what existing power infrastructure can sustainably support.
While training costs and energy consumption are accelerating upward, the performance returns on that investment are compressing downward. This is the third wall, and in many ways the most strategically significant: the diminishing marginal value of scale-based differentiation. The Chatbot Arena Elo rankings, one of the most widely referenced public benchmarks for comparing frontier language models, reveal a striking trend. Over the course of a single year, the Elo skill score difference between the number-one ranked model and the number-ten ranked model fell from 11.9 percent to just 5.4 percent. The gap between the top two models is now only 0.7 percent.
This convergence is not limited to the top of the leaderboard. Open-weight models, those released with publicly available parameters that anyone can fine-tune and deploy, are closing the gap with proprietary frontier models at an accelerating rate. The performance gap between the best proprietary model and the best open-weight model narrowed from 8.04 percent to just 1.70 percent in approximately one year. At this rate of convergence, the distinction between proprietary and open-weight model quality becomes functionally irrelevant for most enterprise applications within the next 12 to 18 months.
Geographic convergence is equally striking. On the MMLU benchmark, a widely used measure of general knowledge and reasoning capability, the gap between the best models from the United States and the best models from China narrowed from 17.5 percentage points to just 0.3 percentage points in 2024. This collapse in the geographic performance gap has significant implications for enterprise leaders evaluating vendor lock-in risk and geopolitical supply chain exposure.
The strategic implication of frontier tightening is profound: if the performance difference between a $170 million model and a $10 million model is measured in fractions of a percent, the investment thesis for continued brute-force scaling collapses. The value proposition shifts from raw capability to deployment efficiency, domain adaptation speed, total cost of ownership, and the ability to update models without full retraining. These are design properties, not scale properties. They cannot be achieved by making models larger. They require fundamentally different design approaches.
Beneath the cost, energy, and performance walls lies a more fundamental design limitation that explains why incremental improvements to the existing paradigm are insufficient. The current AI development cycle relies on building ever-larger models trained on massive, static datasets. Once trained, these models are frozen. Their knowledge is fixed at the point of training. Incorporating new information, correcting errors, or adapting to new domains requires complete, costly, energy-intensive offline retraining. The model cannot learn from experience. It cannot update its understanding as the world changes. It cannot efficiently maintain long-term memory across interactions.
This is the memory wall. The memory wall is fundamentally a design problem — distinct from cost, energy, or benchmark challenges. The transformer design that underlies virtually every frontier model today was not designed for continual learning. It was designed for processing sequences of tokens with a mechanism called self-attention that scales quadratically with sequence length. This quadratic scaling is a direct cause of the compute cost explosion: as models process longer contexts, the computational requirements grow not linearly but exponentially. The design's power comes at the cost of efficiency, and that cost compounds with every increase in model size or context length.
For enterprise applications, the memory wall creates a particularly acute challenge. Organizations that deploy AI models for knowledge-intensive tasks, customer service, research analysis, regulatory compliance, need those models to incorporate new information continuously. A model trained on data from six months ago lacks knowledge of recent regulatory changes, market developments, product updates, and organizational decisions. Under the current paradigm, updating that model's knowledge requires a full retraining cycle that may cost millions of dollars and take weeks to complete. This creates a persistent lag between the state of organizational knowledge and the state of the AI systems deployed to leverage that knowledge.
The fundamental question posed by the memory wall is this: Can we build models that learn continuously, remember selectively, and adapt on the fly, without breaking the bank or the power grid? The question demands a design answer, not incremental optimization. And recent research from Google suggests that answer may already exist.
If the cost of staying at the frontier doubles every five months while the performance advantage shrinks to fractions of a percent, what exactly is the scaling investment buying?
Each of the three scaling walls described above, cost, energy, and diminishing performance differentiation, would be a significant strategic constraint in isolation. Any one of them might be solvable with incremental improvements: larger capital allocations for compute, nuclear energy agreements for power, and more aggressive benchmark optimization for performance. But the critical insight is that these walls are not arriving sequentially. They are converging simultaneously, and their simultaneous convergence creates a forcing function that no amount of capital investment can overcome through scaling alone.
Consider the arithmetic. An organization that doubles its training budget can purchase proportionally more compute, but the energy cost of that additional compute is itself doubling annually. Securing additional energy through nuclear agreements takes years to bring online and billions in capital expenditure. And even if both constraints could be resolved instantly, the performance return on that investment is now measured in fractions of a percent. The three walls form a system of constraints that tightens from every direction simultaneously. Solving any one wall without addressing the other two produces marginal returns at exponentially increasing cost. The convergence marks a permanent inflection point in the development trajectory of machine intelligence.
The response to this convergence is not bigger models. It is fundamentally different designs. And recent research from Google provides the first empirical evidence that such designs are not only theoretically necessary but practically achievable.
The Titans model design, published by Google Research, introduces a capability that directly addresses the memory wall: test-time memorization. This is the ability of a model to incorporate new information into its long-term memory while the model is running, without dedicated offline retraining. The significance of this capability cannot be overstated. Under the current paradigm, updating a model's knowledge requires stopping the model, assembling a new training dataset that includes the new information, re-running the entire training process at a cost of millions of dollars and thousands of tons of carbon, and then redeploying the updated model. Test-time memorization eliminates this cycle entirely. The model learns from new inputs in real time, selectively storing information that is novel and contextually important while allowing routine or redundant information to pass through without permanent storage.
The mechanism by which Titans achieves this selective storage is both elegant and biologically inspired. Like the human brain, the design distinguishes between short-term and long-term memory systems. Information enters the system through a short-term processing module analogous to working memory. The system then evaluates each piece of information using what the researchers describe as a "surprise metric," the mathematical equivalent of the brain's recognition that something is unexpected and important. Low-surprise information, data that is consistent with what the model already knows, is processed and discarded. High-surprise information, data that contradicts or significantly extends the model's existing knowledge, triggers an update to long-term memory storage. The model uses its internal error signal, its gradient, as the mathematical mechanism for computing surprise. This allows the model to selectively update its long-term memory with only the most novel and context-breaking information, rather than attempting to store everything indiscriminately.
The convergence insight: Each wall alone might be solvable with incremental improvements. Their simultaneous convergence creates a forcing function that no amount of capital investment can overcome. The response is fundamentally different designs — ones that learn continuously, remember selectively, and perform more efficiently than scaling alone could ever deliver.
Two additional mechanisms make the Titans memory system robust enough for production deployment. The first is momentum: rather than evaluating surprise based solely on the current input token, the system considers both "momentary surprise" (the novelty of the current input) and "past surprise" (the accumulated context of recent inputs). This dual-signal approach allows the model to capture information that is contextually significant even when individual tokens are not surprising in isolation. A gradually building narrative shift, for example, might not produce high surprise at any single token, but the accumulated momentum of contextual change would trigger a long-term memory update.
The second mechanism is adaptive forgetting. Any system with finite memory capacity must manage that capacity actively. Titans implements a "forgetting gate" that continuously evaluates stored information and discards data that is no longer contextually relevant. The forgetting gate operates as a learned function that the model optimizes during training, allowing it to maintain a constantly refreshed memory that prioritizes the most useful information for the current task. The forgetting mechanism directly addresses one of the oldest challenges in machine learning: catastrophic forgetting, the tendency of neural networks to lose previously learned information when trained on new data. By implementing selective forgetting as a controlled design feature rather than an uncontrolled side effect, Titans transforms a historical weakness into a design advantage.
If Titans represents a specific design solution, MIRAS (Memory Is Really All you need for Sequences) represents something more profound: a generative framework for designing new memory-efficient designs. Rather than proposing a single model, MIRAS provides a theoretical blueprint informed by optimization theory and statistical principles that can generate families of designs optimized for different objectives. The framework explores design spaces beyond the Euclidean geometries and standard regularization techniques that have dominated deep learning, opening access to design configurations that would not have been discovered through conventional experimentation.
From this framework, three distinct attention-free designs have been generated, each optimized for a different performance characteristic: YAAD for robustness against noisy or inconsistent data, MONETA for stability under demanding mathematical conditions, and MEMORA for controlled and balanced memory updates. The fact that a single framework can produce multiple specialized designs suggests that the design space for memory-efficient AI is far larger and richer than the current concentration on transformer variants has explored. The framework reveals an entire landscape of alternatives to the transformer, each with different trade-offs optimized for different deployment conditions.
The combined significance of Titans and MIRAS extends beyond their individual technical contributions. Together, they demonstrate that the three scaling walls are not dead ends. They are forcing functions that are driving AI development toward a fundamentally different paradigm: one in which the defining characteristic of intelligence is not the quantity of parameters or the volume of training data, but the efficiency with which a system learns, remembers, and adapts. For enterprise leaders, this shift changes the evaluation criteria for every AI investment decision they will make over the next several years.
Five systems that illustrate the shift from brute-force scaling to design efficiency.
Titans and MIRAS variants consistently demonstrated higher accuracy and lower perplexity across language modeling and zero-shot reasoning benchmarks, while operating without the quadratic complexity of standard transformer attention.
| Model | C4 (Perplexity ↓) | HellaSwag (Accuracy ↑) | Design Type |
|---|---|---|---|
| Titans | 2.10 | 81.5% | Memory-augmented hybrid |
| MONETA | 2.12 | 80.9% | MIRAS / Attention-free |
| MEMORA | 2.14 | 80.5% | MIRAS / Attention-free |
| YAAD | 2.15 | 80.2% | MIRAS / Attention-free |
| Transformer++ | 2.30 | 77.5% | Standard transformer |
| Mamba-2 | 2.25 | — | State-space model |
| Gated DeltaNet | 2.40 | — | Linear attention |
Most vendor evaluations focus on model performance benchmarks. The Intelligence Organization™ approach evaluates design fit: does the vendor's technology match your operational reality, absorption capacity, and total cost trajectory? This scorecard applies eight criteria, each weighted by strategic importance.
| # | Criterion | What to Evaluate | Weight |
|---|---|---|---|
| 1 | Parameter Efficiency | Performance per billion parameters; smaller models that match larger ones on domain tasks | 15% |
| 2 | Memory Design | Context window, retrieval augmentation, caching strategy; determines real-world task complexity ceiling | 15% |
| 3 | Retraining Frequency | How often the model requires fine-tuning or retraining; affects operational burden and cost | 10% |
| 4 | Energy per Inference | Cost and carbon footprint per query at production scale | 10% |
| 5 | Domain Adaptation Speed | Time to fine-tune for your industry vertical, measured in days not months | 15% |
| 6 | Deployment Timeline | From contract to production deployment; includes integration, testing, change management | 10% |
| 7 | TCO Trajectory | 3-year total cost trend: compute, licensing, talent, energy, compliance | 15% |
| 8 | Integration Complexity | API maturity, existing ecosystem compatibility, middleware requirements | 10% |
Score each vendor 1-5 per criterion, multiply by weight, and sum for a weighted total. The principle from Band 01 (Right-Fit Technology): right-fit means matching vendor design to your operational reality, not purchasing the largest or most capable model. An 8B parameter model that deploys in weeks and integrates with existing systems often outperforms a 400B model that requires 6 months of infrastructure preparation.
Enterprise AI cost models typically account for compute and licensing. They systematically undercount six other cost categories that compound over a 5-year horizon. The Intelligence Organization's 10-20-70 rule applies: 10% of total investment should go to technology, 20% to process redesign, and 70% to people and adoption design. Most organizations invert this ratio.
| Cost Category | Variables to Model | Common Undercount Factor |
|---|---|---|
| Compute (Training) | GPU/TPU hours, dataset preparation, hyperparameter optimization, failed runs | 2-3x initial estimate |
| Compute (Inference) | Per-query cost at production volume, peak vs. average load, caching effectiveness | Grows linearly with adoption; budget for 10x query volume by year 3 |
| Compute (Retraining) | Model refresh frequency, data pipeline maintenance, A/B testing infrastructure | Often omitted entirely; budget 20-30% of initial training cost annually |
| Energy | Electricity, cooling, water, carbon offset obligations | Doubling every 18 months for leading AI workloads |
| Talent | AI architects, prompt engineers, change managers, data engineers, compliance specialists | Ratio shifts from engineers to change managers by year 2 |
| Integration | API development, middleware, legacy system adaptation, data pipeline construction | 40-60% of first-year technology budget |
| Compliance and Audit | Model documentation, bias testing, regulatory filing, third-party audits | Growing as EU AI Act and sector regulations take effect |
| Adoption Design | Training, workflow redesign, resistance management, measurement, ongoing support | The 70% in the 10-20-70 rule; almost always underfunded |
This is not a calculator. It is a diagnostic checklist. If your 5-year model does not include a line item for each of these categories, the model is incomplete. The most common failure mode: organizations budget for technology and discover they needed to budget for people.
The decisions your organization makes about AI design over the next 24 months will determine your cost structure, competitive position, and regulatory exposure for the next decade. RBD. helps leadership teams evaluate these decisions with clarity.
Schedule a Conversation