The boring problem under the AI boom: the plumbing can't keep up

Most of the AI conversation happens at the top of the stack. New models, new capabilities, what the chatbot can do this month. The layer underneath, the one nobody puts in a keynote, gets ignored: the infrastructure required to actually run all of this at scale, and the ways it's quietly straining.

This is the right thing to worry about, because infrastructure problems don't announce themselves. They show up as outages, sudden price changes, and capacity you can't get when you need it. For a small business, that means AI infrastructure costs you don't control can reset your budget overnight. The flashy layer moves fast and gets all the attention. The unglamorous layer (power, chips, data centers, networking, the cost of serving a model to millions of people at once) moves slow and breaks in ways that take years to fix.

Why the constraint is physical now

The thing that makes this different from past software booms is that the bottleneck isn't code. You can't fix a power shortage with a clever algorithm. Running large models at scale consumes electricity, requires specialized hardware that takes time to manufacture, and needs physical buildings full of cooling and networking. These are real-world things with real-world lead times. A data center isn't a weekend project. Power capacity isn't something a utility conjures because demand spiked.

When demand for compute grows faster than the physical world can build supply, you get a squeeze. Prices that were falling can stop falling or reverse. Access that felt unlimited starts coming with quotas. The capacity you assumed would be there gets allocated to whoever can pay the most or signed the earliest contract. None of this is visible while you're enjoying cheap, abundant API calls, which is precisely why it's a problem nobody is talking about. The good times hide the constraint.

What this means if you're building on top of it

You don't have to operate a data center for this to affect you. If your business depends on AI you don't host yourself, you're sitting on top of someone else's infrastructure bet, and their constraints become yours.

Don't assume today's prices are permanent. A lot of current AI pricing reflects a land-grab phase where providers are buying market share. If the underlying economics tighten, prices can move against you. Build your numbers so the business still works if the cost of inference is meaningfully higher than it is now. If your margins only survive at today's prices, you have a fragility you can't see.

Avoid single points of failure. If one provider is your only path to a capability your operation depends on, their capacity crunch is your outage. The defense is the boring one: keep your prompts and logic portable enough that you could switch providers without rebuilding everything. You may never need to. The option is worth having.

Match the model to the job. A surprising amount of AI work runs on the biggest, most expensive model out of habit, not need. When infrastructure is tight and pricey, the discipline of using a smaller, cheaper model for tasks that don't require the flagship stops being an optimization and starts being protection. Most tasks don't need the top of the line.

The hype-cycle read

There's a broader point under all this. Every technology boom front-loads the exciting part and back-loads the bill. We're in the phase where capabilities are dazzling and the constraints feel theoretical. They aren't theoretical. They're being built, or not built, right now, in concrete and copper and silicon, on timelines measured in years.

For a small business, the practical posture isn't fear. It's not assuming the current abundance is the permanent state of the world. Build so you'd survive a world where AI compute is more expensive and harder to get than it is today. Keep your dependencies flexible. Use the cheapest tool that does the job. The companies that get hurt when the plumbing strains will be the ones who built as if the easy phase would last forever. The ones who quietly planned for the constraint will mostly be fine, and they'll have leverage when everyone else is scrambling for capacity.

Related reading

- [Never build a critical workflow on a model you don't control](01-dont-depend-on-one-model.md)

- [Your flat software subscription is quietly becoming a usage bill](24-saas-usage-billing.md)

- [When the AI boom slows, the question is which kind of wall it hits](15-ai-boom-limits.md)