The Inference Chips Are Down: Private AI Chipmakers

Public markets were mixed this week as a renewed sell-off in AI stocks offset broader strength elsewhere. Strong 3Q earnings supported stocks early in the week, but sentiment reversed sharply Thursday after Meta’s plans to boost capex reignited worries about the cost of the AI buildout. (October 21, 2026, Investing.com, “Nasdaq posts worst day since June as AI financing worries batter risk sentiment”) Oil prices climbed to a five-month high (~$113 on Brent) on Iran war concerns. (October 22, 2026, Yahoo Finance, “Stock market today: S&P 500, Nasdaq tanks as tech stocks sink amid AI trade worries, Meta shares plummet”; October 19, 2026, Bloomberg via Rigzone, “Oil Prices Hover Near Five-Month High”) The 10-year yield fell ~17bp on Thursday to 4.97%, back below 5% for the first time since 9/23. (October 22, 2026, Associated Press, “Asian shares slip as sharp drops for AI stocks overwhelm markets worldwide and oil prices jump”) Through Thursday, the S&P 500/Nasdaq Composite were +0.1%/-0.5%, respectively, for the week. (October 22, 2026, Associated Press, “How major US stock indexes fared Thursday 10/22/2026”)

On Thursday, a ~7% drop in chipmakers sent the Nasdaq Composite to its worst day since June, and the S&P 500 posted its worst day since July. (October 23, 2026, Tech Times, “Chip Stocks Lose $200 Billion as Equipment Maker’s Warning Counters NVIDIA’s AI Optimism”; October 22, 2026, Associated Press, “Asian shares slip as sharp drops for AI stocks overwhelm markets worldwide and oil prices jump”) Meta fell ~7% following its 3Q results, and CoreWeave dropped ~10% (down 20% on the week). (October 22, 2026, Associated Press, “Asian shares slip as sharp drops for AI stocks overwhelm markets worldwide and oil prices jump”) Despite recent volatility, the S&P 500 remains up ~12% YTD. (October 22, 2026, Associated Press, “How major US stock indexes fared Thursday 10/22/2026”) Our NPM Private Market Tracker, an average of the performance of the top 50 names by valuation, is +103% YTD (October 22, 2026, NPM, “NPM Private Market Tracker”).

The Next Leg of the AI Trade

Inference is quietly becoming the economic center of AI infrastructure. Gartner expects 2026 spending on “infrastructure-as-a-service” for inference to reach $23bn, roughly 55% of the total, compared with $19bn for training. (August 12, 2026, DIGIT, “AI IaaS Spending to Surge 96% in 2026, Gartner Says”) McKinsey projects that inference will account for roughly 60% of AI compute demand by 2030. (February 24, 2026, McKinsey & Company, “The future of AI workloads”; October 5, 2026, McKinsey & Company, “Scaling AI inference”) None of this implies that training is shrinking; rather, inference is growing faster.

If McKinsey’s estimated 93 GW of inference IT demand has an average system cost of $15–35mm/MW, and systems are refreshed every 5 years, we calculate average annualized replacement demand for inference systems could be ~$465bn (mid) annually. (February 24, 2026, McKinsey & Company, “The future of AI workloads”; October 23, 2026, NPM, “The Inference Chips Are Down”)

Workloads are Shifting Towards Inference

Training and inference are two distinct phases in the life of an LLM. Training builds the model: it runs passes over enormous datasets to refine the model’s weights. It is a highly computational workload that requires dense accelerator clusters joined by fast interconnect. “FLOPs” are a key constraint.

In contrast, inference puts the finished model to work: it executes the trained LLM for every user query. A model is trained once (and periodically refreshed) but utilized potentially billions of times. As a result, the cumulative tokens processed in inference can dwarf the tokens consumed in the original training run. (October 5, 2026, McKinsey & Company, “Scaling AI inference”)

McKinsey forecasts that the electrical capacity demanded by inference will rise from ~20.9 GW in 2025 to ~93.3 GW in 2030, a CAGR of ~35%, vs. ~62.2 GW for training in 2030. (February 24, 2026, McKinsey & Company, “The future of AI workloads”)

Why Training and Inference Need Different Compute

The computational profile of training is fundamentally different from that of inference. Training is dominated by matrix multiplication. Every step requires forward and backward passes to compute and optimize parameters, and arithmetic horsepower is the bottleneck. From an economic standpoint, the key variables are the time and cost to finish the run.

Inference is very different. From a high level, inference can be split into two phases: prefill and decode

• Prefill is the “prompt ingestion” phase. The model processes all of the input tokens in parallel to build the key value (“KV”) cache. The KV cache is an AI model’s short-term memory that stores information from earlier parts of a conversation so it can generate new responses faster without repeatedly processing everything that came before.
• Decode is the stage of AI inference where the model generates its response, one token at a time, using what it has already written to determine what comes next. The decode phase is where inference compute demands depart most sharply from training. The model produces output tokens individually, and each new token requires reading the full set of model weights and the growing KV cache from memory. As a result, this process is often limited by how quickly the chip can retrieve information from memory, rather than how fast it can perform calculations.

The differences in compute vs. inference have direct consequences for chip architecture. Training an LLM is a one-time or periodic capital project, where speed is often more important than cost per token. As a result, a training chip is optimized to maximize sustained compute throughput. Flexibility is also valuable, because model architectures can change. These requirements favor powerful, programmable devices, which is a major reason NVIDIA’s GPUs make flexible, baseline “all purpose” AI chips.

An inference chip faces a different optimization problem. Because decode is memory-bound, the most important design parameter is often memory bandwidth, and even how physically close the KV cache sits to the arithmetic units. Also, inference is a recurring operating cost (rather than a one-time capital project), making energy consumption and $/token matter more than raw speed. To face these challenges, the new generation of dedicated inference chip designs experiment with innovations like large on-chip SRAM, near-memory/in-memory compute and wafer-scale integration, all of which aim to reduce the cost of moving data.

Who are the Competitors?

Given the size of the potential market, a number of new entrants have sprouted in recent years. The issue for investors is that seemingly all of them have the “best” chip, yet the architectures and approaches are very different.

One group of new entrants, including SambaNova and Tenstorrent, move computation and data explicitly through an array of processing elements to reduce memory traffic.

A second group focuses on attacking the memory bottleneck directly. Cerebras and Groq focus on SRAM-based solutions. SRAM is very high bandwidth but is lower in storage capacity, requiring more real estate on the chip. Positron uses LPDDR memory, which is lower bandwidth but high in storage capacity. Both SRAM and LPDDR solutions avoid the HBM bottleneck that has hamstrung many chip makers, including NVDA. The chart below compares the major types of memory used by GPU and inference chip producers.

Source: Pitchbook, NPM

Other adjacent private companies valued >$1bn include Lightmatter ($4.4bn, optical interconnect), Ayar Labs (~$3.8bn, optical I/O) and SiFive (~$3.7bn, RISC-V IP). (October 16, 2024, Bloomberg, “Photonic Computing Startup Lightmatter Reaches $4.4 Billion Valuation”; March 3, 2026, Pulse 2.0, “Ayar Labs: $500 Million Series E Raised At $3.75 Billion Valuation For Optics Solutions For AI Scale-Up”; April 10, 2026, The Next Web, “SiFive raises $400m Series G at $3.65bn valuation in final round before IPO”) Note that these are enabling technologies rather than full inference chip producers.

Another – perhaps more agnostic – way investors may seek exposure to inference demand is through the dedicated inference cloud providers, such as Fireworks.AI ($17.5bn), Modal ($15.8bn), Baseten ($13bn) and Together.AI ($8.3bn) (July 16, 2026, Dealroom, “Fireworks AI raises $1.5B Series D at $17.5B valuation”; October 14, 2026, The Next Web, “Modal Raises $750M at $15.75bn Valuation as Run Rate Tops $1bn”; September 24, 2026, AI Weekly, “Modal in Talks at $15B, Baseten at $26B as Inference Booms”; July 1, 2026, TechCrunch, “Neocloud Together AI raises $800M, leaps to $8.3B valuation”).

Will the House Win?

In our view, the main risk to the new crop of inference chip producers is competing silicon from hyperscalers and NVDA. Google’s TPU, Amazon’s Trainium, Microsoft’s Maia and Meta’s MTIA all have potentially massive captive demand. In addition, NVIDIA’s pivot toward inference, is already underway. For instance, NVIDIA’s Groq technology license supports Groq’s LPUs integrated into its stack. (September 6, 2026, KuCoin News via CryptoBriefing, “Nvidia’s $20B Groq AI Inference System Enters Production”)

YTD Private Market Performance by Subsector