What Is a Neocloud? The AI-Native GPU Cloud, Explained

The one-sentence definition

A neocloud is a cloud provider whose entire business is renting accelerated compute — GPUs, the network between them, and the storage feeding them — to people training and running AI models. That is the whole product. The name is industry shorthand that stuck around 2023 as a way to distinguish these companies from both the hyperscalers above them and the bare-metal hosters below them.

The useful way to understand a neocloud is not by what it adds, but by what it removes. AWS sells you a GPU instance sitting on top of an enormous platform: identity and governance spanning an entire organization, thirty-plus regions, compliance attestations for regulated industries, and hundreds of managed services your GPU job will never touch. You pay for all of it in the hourly rate whether you use it or not. A neocloud deletes those layers and passes most of the savings on.

Notice what is identical across all three columns in Fig. 1: the GPU layer. An H100 in a CoreWeave rack and an H100 in an AWS rack are the same part, from the same vendor, with the same peak FLOPs. Nobody has a cheaper source of silicon. The price difference is entirely a function of what is stacked on top and what margin is taken.

Where neoclouds came from

Neoclouds exist because of a supply-and-shape mismatch that opened in 2023 and has not fully closed. Demand for large-scale GPU capacity exploded faster than the hyperscalers could rack it, and the capacity they did rack was shaped for their existing enterprise customers — heavily virtualized, spread thin across many regions, sold in single-instance units. AI labs wanted the opposite: hundreds or thousands of GPUs in one contiguous, low-latency island, on bare metal, for months at a time.

Several of the companies that filled that gap did so from an unusual starting position — they already had power and datacenter shells, which is the genuinely scarce input:

NVIDIA had its own reason to help this happen. A market where three hyperscalers are the only meaningful buyers of AI silicon is a market with three counterparties who can squeeze you — and all three are simultaneously building their own accelerators. Cultivating a competitive tier of independent GPU clouds gave NVIDIA both pricing power and demand diversity, which is why allocation flowed to neoclouds during the tightest periods of the shortage rather than exclusively to the largest buyers.

The economics, decomposed

“Neoclouds are cheaper” is only interesting if you know which line item is cheaper. Break an hourly rate into its parts and the answer is clear — and it is not the GPU.

Working through the components:

• GPU amortization. An 8×H100 node costs on the order of $250–300k. Spread over a 4–6 year depreciation schedule at high utilization, the silicon works out to roughly $1–2 per GPU-hour. Everyone buys from the same vendor, so this line is broadly similar — neoclouds edge it down mainly through higher guaranteed utilization (long contracts mean fewer idle hours to absorb) and denser racks.
• Power and cooling. An H100 draws ~700 W, and a full 8-GPU node with CPU, networking and cooling overhead lands near 10 kW. At $0.05/kWh in West Texas versus $0.12/kWh in Northern Virginia, the same rack costs less than half as much to run. Neoclouds site for cheap power; hyperscalers site for customer latency. That is a real and permanent structural gap.
• Platform overhead. This is the big one. Every hyperscaler GPU-hour carries a share of the cost of running hundreds of services, thirty regions, a global support organization, and continuous compliance work across dozens of frameworks. A neocloud with one product and a few sites simply does not have that cost base to recover.
• Margin. Hyperscaler infrastructure businesses are managed to protect high operating margins and would rather hold price than win low-margin AI volume. Neoclouds are fighting for position in a young market and price accordingly. This gap compresses as the market matures — it is the least durable part of the discount.

Where neoclouds are genuinely better

Price is the headline, but for distributed training the technical argument is often the stronger one. Training a large model across many nodes is a communication-bound problem: every step ends with an all-reduce across every GPU, and the slowest link sets the pace for the entire cluster.

The practical consequence is scaling efficiency. A job that holds 90% scaling efficiency at 256 GPUs on a well-built InfiniBand fabric but drops to 70% on a congested Ethernet network is not 20% more expensive — it is 20% more expensive on top of a rate that was already several times higher. The two effects compound, and on long training runs that difference is measured in weeks.

This gap has narrowed. Hyperscalers now ship AI-specific instance families with dedicated high-bandwidth fabrics and cluster placement guarantees of their own. The generalization that holds is narrower than it was: neoclouds tend to lead on price-performance for large contiguous training jobs, and that is the workload to send them.

What you give up

A neocloud is a smaller product on purpose. That is the source of the savings, and also the source of every drawback.

Two of these deserve more than a table row.

Data gravity. If your training data sits in S3 and your serving stack runs in the same account, moving compute elsewhere means paying egress on every dataset, running a second observability and security posture, and maintaining a network path between two providers. For a 50-node training job the GPU savings swamp that friction easily. For a modest fine-tuning workload it may not.

Counterparty risk. Many neoclouds financed their fleets with debt secured against contracted revenue and, in some cases, the GPUs themselves. That model works while utilization and rental rates stay high. It is exposed if either falls — and both are subject to the same forces, since a new NVIDIA generation lands roughly annually and drags the rental value of the prior one down with it. The question that matters is not whether a provider is cheap today, but whether its depreciation assumptions and financing survive a soft year.

The three tiers of the market

“Neocloud” covers companies with very different risk and service profiles. It helps to separate them:

A quote of “$1.80 per H100-hour” means something completely different depending on which tier it came from. Compare within a tier, and make the tier explicit in any procurement conversation.

So which should you use?

The decision usually collapses to a single binding constraint. Identify it honestly and the answer follows.

In practice most teams past a certain scale run a hybrid: training and batch inference on a neocloud where the cost gap is largest and the coupling is loosest, latency-sensitive serving next to the production data, and a hyperscaler contract kept alive for burst capacity and the services nobody wants to rebuild. The workloads that move are the ones that are big, self-contained, and tolerant of a region going away.

Diligence checklist

If you are about to sign a meaningful commitment with a neocloud, these are the questions that separate a good deal from an expensive lesson:

1. Do they own or lease the datacenter, and how long is the power contract? Secured multi-year power at a known price is the single strongest signal of durability.
2. How was the fleet financed, and against what? Debt secured on GPUs whose resale value falls with each NVIDIA generation is a materially more fragile structure than debt against long-term contracted revenue.
3. What is the actual fabric? Ask for the topology, not the adjective. Non-blocking or oversubscribed, and at what ratio? InfiniBand NDR or RoCE? Verify with a NCCL all-reduce benchmark at the node count you intend to run, before you sign.
4. What happens to your data if they enter distress? Get the exit terms in writing: notice period, data-egress assistance, and whether your storage is on hardware they own or a third party's.
5. Is the storage tier fast enough to keep the GPUs fed? A cluster starved by slow storage is an expensive idle cluster. Ask for sustained throughput per node under concurrent load, not peak marketing numbers.
6. What is the real availability commitment? Many neocloud SLAs credit you service time rather than compensating for a failed training run. Understand what a multi-day outage actually costs you and who absorbs it.

Frequently asked questions