Groq's LPU Revolution: How SRAM is Breaking Nvidia's Inference Monopoly

I typed a prompt into Groq’s web interface, and before my finger physically lifted off the enter key, 500 words of Llama 3 output painted the screen. It was running at over 800 tokens per second. It felt like a parlor trick. But it is not magic; it is a fundamental rewrite of how we process AI inference.

Before you read: We previously explored how local models are ditching the cloud. This article looks at the hardware side of that equation, specifically how specialized chips are making traditional inference speeds look like a 56k dial-up connection.

Table of contents

• The GPU Bottleneck
• Enter the LPU and the SRAM Gamble
• The Beauty of Deterministic Hardware
• A Grounded Reality Check
• Fun Facts You Can Steal
• The Split of the AI Hardware Market
• Sources

The GPU Bottleneck

For the last decade, Nvidia has convinced the world that artificial intelligence requires Graphics Processing Units. They are absolutely right when it comes to training models. Training requires shuffling petabytes of data through massive parallel calculations.

But inference—the act of actually using the model to generate text—is a completely different beast. Large Language Models generate one token at a time. To predict the next word, the chip has to load the entire model into memory, do the math, spit out a token, and then do it all over again for the next word.

Nvidia’s H100 GPUs rely on High Bandwidth Memory (HBM). Think of HBM like a massive, multi-story library where you can check out 10,000 books at once. It is incredible for throughput. But for inference, you don’t need 10,000 books at once; you need one specific book instantly, over and over. The time it takes the librarian to fetch that single book is called latency. And in the AI world, we call this the “memory wall.”

Enter the LPU and the SRAM Gamble

Groq looked at this memory wall and decided to bypass it entirely. They built the Language Processing Unit (LPU). Instead of relying on external HBM, Groq uses SRAM (Static Random Access Memory) placed directly on the chip itself.

SRAM is blisteringly fast. It is the same type of memory your CPU uses for its L1 cache. But there is a catch: SRAM is physically massive and astronomically expensive. While an Nvidia H100 can hold 80 gigabytes of HBM, a single Groq chip holds a measly 230 megabytes of SRAM.

To run a 70-billion parameter model like Llama 3, you cannot fit it on one Groq chip. You have to string hundreds of them together. This sounds like a networking nightmare, but it works because of Groq’s secret weapon: deterministic architecture.

The Beauty of Deterministic Hardware

Modern GPUs are chaotic. They have complex schedulers, branch predictors, and memory managers trying to guess what the software wants to do next. GPUs manage memory like a stressed restaurant manager trying to seat 500 people at once while the kitchen catches fire.

Groq’s hardware is completely deterministic. There is no dynamic scheduling. The compiler analyzes the AI model before it even runs and maps out the exact clock cycle every piece of data will move. It assigns every byte a permanent seat before the restaurant even opens.

Because the hardware knows exactly what will happen and when, there are no cache misses. No bottlenecks. The chips act as one massive, perfectly synchronized brain.

A Grounded Reality Check

I need to temper the hype here. Is Groq going to put Nvidia out of business? Absolutely not.

Groq is effectively useless for training AI models. Their chips are purpose-built for inference. Furthermore, the capital expenditure required to set up a Groq cluster is staggering. Because each chip holds so little memory, you need racks upon racks of LPUs and thousands of fiber optic cables just to host a single instance of a large model.

Nvidia gives you a versatile Swiss Army knife. Groq gives you a custom-forged scalpel. If you ever need to turn a screw, the scalpel is worthless. But if you need to cut, nothing else comes close.

Fun Facts You Can Steal

• Groq was founded by Jonathan Ross, who previously helped invent Google’s TPU (Tensor Processing Unit). He essentially looked at Google’s AI hardware, realized it was still too complex, and left to build a simpler, faster architecture.
• Because Groq’s hardware is entirely deterministic, they do not even run operating systems on their chips. The compiler is so precise it acts as the sole orchestrator of the silicon.

The Split of the AI Hardware Market

We are witnessing the great schism of AI hardware. The era of buying one chip to do everything is ending. Nvidia will continue to dominate the training market, selling massive H100 and Blackwell clusters to OpenAI and Meta.

But for the deployment phase—when applications actually need to serve millions of users with zero latency—SRAM-based architectures like Groq are proving that brute force is not the only way forward. Sometimes, you just need a better map.

Sources

• Groq ISCA 2020 Paper: “A Software-Defined Tensor Streaming Multiprocessor”
• SemiAnalysis: AI Inference Hardware Economics
• Jonathan Ross Interviews on Deterministic Architecture
• MLCommons Inference Benchmark Results (2024)