AI’s billion dollar bottleneck (It isn’t GPUs)
Recently I sat down with an interview for The Array to explore global AI policy and the infrastructure arms race.
Although the reporter ‘s angle was originally to explore which parts of the stack are most important for advantage, it became about something different — in fact, the exact opposite.
Because the bottleneck isn’t in the stack at all.
—
Technology isn’t where the value is; value is in the orchestration layer.
—
Buy first, integrate later
Every major tech wave follows this basic ideology.
During the cloud era, organizations rushed to adopt infrastructure they weren't ready to operationalize. They moved to the cloud before they had the capabilities, people, or systems to turn it into value.
At one point, close to 40% of cloud spend was wasted…just sitting on idle, misconfigured, or inefficiently used resources. Now it’s the —
—
Same pattern, higher stakes
What's happening now with AI is almost identical, except more expensive and more complex to integrate than any tech wave before it.
Companies are rushing to stockpile GPUs (again), buy compute credits, or whatever version of future-proofing they believe in,without defining real use cases, building governance, or laying down the organizational foundation needed to actually integrate AI as a core, value-generating capability.
These statistics confirm:
AI inference workloads often run at just 20-40% capacity
On-premise GPU clusters are even lower, around 15%
Globally, nearly half of GPUs are just sitting there, unused
This has given rise to an entire new market: "GPU as a service." People are trying to rent out unused capacity to at least monetize what would otherwise be sunk cost.
—
Where AI value is unlocked
AI value lies in the less visible aspects of AI; the adoption, culture, cross-functional collaboration, workflow orchestration.
Do models have the data they need?
Are people using the tools we build?
Everything BUT the technology itself is why the technology fails.
Global patterns emerging
These aspects of the integration layer are consistently overlooked by both public and private sectors.
We know this, because see the same dynamics playing out geopolitically:
The US chases speed, scale, and deregulation.
Move fast, build big, break stuff.
The EU emphasizes governance, control, consensus.
Lead on human rights and policy development but with more rules to play by.
The UK tries to straddle both speed and fairness, but after Brexit, that middle ground fractured with political division.
CIOs, sound familiar? It's exactly what we saw during the cloud boom:
US scaled infrastructure fastest while enterprise adoption lagged
EU led on privacy (GDPR) but was slower on innovation
UK tried to do both but ended up stuck in between
Bottom line — Different rhetoric, same outcome.
Nearly half of global GPUs are sitting unused because we're chasing infrastructure when the advantage is integration.
We're solving the wrong problem.
It's not about just building capability, but about making capability usable.
What good is building highways if no one has the right car or knows how to drive?
My book, the Enterprise AI Method addresses this through systematic integration that converts infrastructure investments into measurable business outcomes.
Through a “fortunate” string of hazards, it’s going through a rework, and coming through a different publisher. Thanks for reading.
-Megan Starkey
Company Updates & Events
Join me with Juliet Fox, CEO of AvidEdge Executive Search Services for this STUMP THE EXPERTS event September 25th, 3:00p. in Minneapolis, where CIOs will come with questions on AI followed by discussion. No RSVP needed.
https://www.linkedin.com/events/stumptheexpertswithavidedge-rbd7344028630487879680/
Next: Send this to someone who might value it.
Read the Array article here:
https://thearray.hitachivantara.com/en-us/analysis/ais-billion-dollar-bottleneck
