What Is Groq? Ultra-Fast LLM Inference

There's a viral demo from early 2024 that explains Groq better than any marketing deck: someone types a question into a chat box and the entire multi-paragraph answer streams back faster than they can read it. No "thinking" delay, no slow first token, no friendly typewriter effect — just text appearing as fast as a 4K monitor can scroll.

That's Groq. The company built custom silicon designed not to train AI models, but to run them, at speeds that make GPUs look slow. By May 2026, GroqCloud serves popular open models — Llama, Mistral, Qwen, DeepSeek — at 500 to 1,000+ tokens per second, roughly ten times faster than the same models on typical cloud GPU stacks.

This article explains what Groq is, why it's fast, what it costs, and how it compares to the other startups trying to dethrone NVIDIA's grip on inference.

The Origin Story

Groq was founded in 2016 by Jonathan Ross, who had previously led the design of Google's first TPU (Tensor Processing Unit). That detail matters: TPUs were the original "AI chip is not a GPU" bet, and Ross took the same conviction to a startup. The company spent its first few years quietly designing what they call the LPU — Language Processing Unit.

The name is a bit of a backronym; the chip wasn't designed exclusively for language models. But by the time Llama 2 and Mistral ignited the open-model boom in 2023, "language processing unit" became the perfect positioning. Groq's hardware turned out to be ideal for streaming token generation, and the company pivoted hard toward LLM inference as a service.

Why Groq Is Fast: SRAM, Single-Stream, Determinism

Most GPUs (NVIDIA H100, H200, B200, AMD MI300) treat LLM inference as a memory-bandwidth problem. The model weights live in HBM (high-bandwidth DRAM), which is fast — but every token still costs a round trip to off-chip memory. To stay efficient, GPU inference engines like vLLM batch many user requests together, amortizing memory cost across users at the price of latency for any single user.

Groq's LPU inverts this. The architecture stores model weights in on-chip SRAM — much smaller, but much faster and predictably so. Combined with a deterministic single-stream execution model (no dynamic batching, no speculative reordering, no memory contention), the result is incredibly low per-token latency.

Trade-offs:

• Pro: unmatched single-user throughput; latency you can measure in milliseconds.
• Con: SRAM capacity is tiny per chip, so a single Llama-70B-class model is sharded across many LPUs in a rack. The economics only work at hyperscale.

For more on how LLMs run at inference time, see our piece on AI inference vs training.

GroqCloud: The Product

You don't buy LPUs. You use GroqCloud, an API service that exposes open models behind an OpenAI-compatible endpoint. As of May 2026, GroqCloud serves models including:

• Llama 4 (Meta)
• Llama 3.3 70B
• Qwen 3 (Alibaba)
• Mistral / Mixtral
• DeepSeek-V3 and R1 distillations
• Gemma 2 (Google)
• Whisper (OpenAI, for speech-to-text)

The free tier is generous: typically 30 requests/minute and 14,400/day for popular models. Paid pricing runs roughly $0.05 to $0.59 per million tokens depending on model size — competitive with, and often cheaper than, Together AI, Fireworks, or Anyscale.

Notably missing: proprietary models. You cannot run GPT-4 or Claude on Groq. Those companies don't release weights, so Groq can't load them onto LPUs. This is the single biggest constraint on Groq's addressable market.

The Inference Provider Landscape (May 2026)

Groq is not alone. The "fast inference of open models" category exploded in 2024 and is now a real market:

Provider	Hardware	Top speed (Llama-3 70B)	Open models	Proprietary	Pricing
Groq	LPU (custom SRAM)	~600 tok/s	Many	No	$0.05 – $0.59/M
Cerebras Inference	WSE-3 (wafer-scale)	~2,100 tok/s	Many	No	$0.10 – $0.85/M
SambaNova	RDU	~580 tok/s	Many	No	Competitive
Together AI	NVIDIA H100/H200	~250 tok/s	Many	No	$0.20 – $0.90/M
Fireworks AI	NVIDIA	~200 tok/s	Many	No	$0.20 – $0.90/M
NVIDIA NIM / vLLM (self-host)	H100/H200/B200	~150 – 300 tok/s	Anything	Anything	Varies
OpenAI / Anthropic API	NVIDIA + custom	~80 – 150 tok/s	No	Yes	Premium

Two things stand out. First, Cerebras is even faster than Groq on small/medium models thanks to its absurd wafer-scale chip — a single chip the size of a dinner plate, packed with on-chip memory. Second, NVIDIA still wins on flexibility — if you want to run anything, including proprietary models, on the latest weights, you go GPU. For a comparison of GPU-based inference providers, see Together AI vs Fireworks AI.

For context on the models being served, see our explainers on DeepSeek R1 and LLMs in general.

When You Actually Want Groq

Groq's speed isn't a vanity metric. It enables UX patterns that are awkward or impossible on slower inference:

• Voice assistants — low time-to-first-token means natural conversational latency.
• Real-time agents — agentic loops with 5+ LLM calls per user action need cheap, fast inference or they feel laggy. See AI agents explained.
• Streaming code completion — full-block completions appear in <1 second instead of 5.
• Bulk evaluation / scoring — throughput at low cost.
• Search re-ranking — you can afford to run an LLM in the critical path of every query.

If your app waits seconds for a response anyway (analysis, research, long-form writing), Groq's advantage matters less. Use the right tool for the job.

When Groq Doesn't Work for You

• You need GPT-4, Claude, or Gemini — Groq doesn't host them and never will, because the weights are private.
• You need a specific fine-tune that hasn't been published — Groq serves released checkpoints, not your custom LoRA.
• You need on-prem deployment — Groq sells racks, but the entry price is enterprise-only.
• You need multimodal (vision + text) at frontier quality — Groq's catalog is improving but lags GPU providers.

If you want to self-host on commodity hardware, see Ollama and local LLMs.

Technical Deep Dive: LPU Architecture

The LPU's speed comes from architectural decisions that are fundamentally different from GPUs. GPUs are memory-hierarchy optimized: they assume model weights live in slow off-chip memory (HBM), and they minimize round trips through batching and prefetching. The LPU inverts this assumption.

On-Chip SRAM: LPU boards pack up to 312 MB of SRAM per chip—fast, low-latency, but expensive and power-hungry per bit compared to HBM. At this capacity, a Llama-70B model (66 GB unquantized) requires ~210 LPU chips. A single request streams through all 210 chips in parallel, with each chip doing a fraction of the computation. The result: extremely low latency, because no off-chip memory access is needed.

Dataflow Execution: Unlike GPUs' SIMT (Single Instruction Multiple Thread) model, the LPU uses a dataflow architecture. Computation flows through the chip like water through pipes—values propagate from producer operations to consumer operations with minimal queuing or buffering. This is more similar to specialized hardware (TPUs, FPGAs) than to GPUs.

Compiler-Driven Optimization: Groq's compiler must schedule all operations (matrix multiplies, activations, moves) across the chip to minimize memory stalls. This is a much harder problem than GPU compilation (NVIDIA uses hardware schedulers; Groq must pre-plan). However, pre-planned schedules enable very predictable latency and throughput.

Deterministic Execution: Because schedules are pre-planned, execution is deterministic—the exact latency for a given input is known before runtime. This is radically different from GPUs, where batching depth, memory contention, and other dynamic factors create latency variability. This determinism is enormously valuable for latency-critical applications.

Source: Groq technical documentation and IEEE Micro conference presentations (2023-2024) — Detailed architectural descriptions of dataflow execution and on-chip memory management.

Real-World Performance: Benchmarks and Context

GroqCloud's 500–1,000 tokens/sec claims need context. Token generation throughput on inference depends heavily on time-to-first-token (TTFT) and inter-token latency (ITL), two different metrics often conflated.

Time-to-first-token (TTFT): How long before the first output token appears after a user's query. For a 100-token prompt, TTFT is the time to process the entire input and generate the first output token. Groq's ~50–100ms TTFT is exceptional—many GPU providers take 500ms–2 seconds.

Inter-token latency (ITL): How long between successive output tokens. This determines perceived responsiveness during streaming. Groq's 1–2ms ITL means output feels instant—no pause between words. GPU providers' 50–100ms ITL creates the familiar "typewriter" effect.

Throughput (tokens/sec): When users see "500 tokens/sec," this usually means sustained ITL at scale — when many users are simultaneously requesting tokens, Groq can serve 500 tokens across all users per second. This is not the same as a single user getting 500 tokens in their response in one second (which would be 500 ITL of 2ms — 1 second total, not possible given input processing).

A concrete example:

• Query: 100-token prompt + request for 500-token response.
• TTFT: 100ms (process prompt).
• ITL: 2ms per token × 500 tokens = 1,000ms.
• Total latency: ~1.1 seconds for complete response.

Compare to a GPU provider at 80ms TTFT + 50ms ITL:

• Total latency: 80ms + (50ms × 500) = 25 seconds.

Groq's advantage is particularly dramatic for long outputs or many sequential requests (agentic loops).

For throughput, Groq's infrastructure can sustain ~500 tokens/sec across many simultaneous users. A GPU provider with the same H100 allocation might sustain ~150–250 tokens/sec. The difference comes from:

• Lower ITL (2ms vs 50ms) = less time blocking between tokens.
• Deterministic scheduling = fewer tail latencies from contention.
• Aggressive batching of unrelated user requests = higher utilization.

Practical Use Cases and Performance Gains

Voice Assistants: A conversational voice loop (user speaks → transcribe → LLM reply → text-to-speech → user hears response) needs latency <500ms to feel natural. At 50ms ITL, a GPU provider's first-token latency alone (80ms) leaves little room for transcription and TTS. Groq's 100ms TTFT + 2ms ITL means the voice loop feels genuinely real-time.

Real-Time Chat with Agents: AI agents that iterate over 5+ LLM calls per user action (think: search → retrieve context → generate → check → refine) accumulate latency. At 1 call every 2 seconds (GPU), a 5-call loop takes 10 seconds. At 1 call every 200ms (Groq), the same loop finishes in 1 second. This transforms UX from "wait for the robot to think" to "instant collaboration."

Search Re-Ranking: Large-scale search systems rank thousands of results using expensive ML models. Running an LLM on the top-10 candidates (summarize, score relevance) is intractable at 50ms per result (500ms total). At 2ms per result (20ms total), it becomes viable—you can afford to use LLMs in the critical path of every search.

Bulk Evaluation: When you need to evaluate 1 million documents (classify, extract entities, score), latency is less critical than throughput. GPU providers excel here because batching amortizes cost. However, Groq's ~$0.10–0.20 per million tokens vs GPU at $0.40–0.80 often makes Groq cheaper even at lower per-token throughput.

Streaming to UI: Modern web applications stream LLM responses token-by-token. Users expect word-level streaming (characters appear every ~50ms, so each token visible immediately). Groq's 2ms ITL enables this natively. GPU providers need to buffer tokens to achieve the same user experience, adding 50–100ms of effective latency.

Cost Analysis: When to Use Groq

GroqCloud pricing (as of May 2026) runs:

• Llama-8B: $0.05/M tokens
• Llama-70B: $0.25/M tokens
• Mixtral 8×22B: $0.24/M tokens
• Qwen 3: $0.27/M tokens (larger model, higher cost)

For comparison, GPU providers (Together AI, Fireworks) charge $0.20–$0.90/M depending on model and load. Open-source self-hosting on H100 costs ~$2 per hour per chip; for 1M tokens at 150 tok/sec, you need ~7,000 seconds (~2 hours) → $4/M tokens at 100% utilization (unrealistic; add 3–5x overhead).

When Groq is cheaper:

• Your app is latency-sensitive but compute-efficient. E.g., voice assistant that calls the model 3–5 times per user interaction. Groq's lower latency means simpler app logic (no timeouts, retries, fallbacks), and lower cost per user due to higher success rates.
• You need consistent latency. Financial systems, safety-critical applications, and SLAs benefit from Groq's determinism.

When GPU providers win:

• Your app is throughput-optimized. Bulk processing (data labeling, document extraction) where you can batch 100+ requests together. GPUs' batching efficiency amortizes setup costs.
• You need proprietary models. If your fine-tuned GPT-4 is your moat, you have no choice.
• You're building in-house. Self-hosting avoids per-token pricing and vendor lock-in, but demands engineering resources.

Enterprise Deployment and Considerations

Groq offers on-premises deployment — you can buy LPU racks and run GroqCloud yourself. Pricing is enterprise-only (not public), but typically involves millions in hardware + software licensing. On-prem makes sense if:

• You process petabytes of sensitive data that can't cross your firewall.
• You need guaranteed latency and throughput for mission-critical systems (financial trading, autonomous vehicles, medical devices).
• Your inference volume is enormous (>100M requests/day) that API pricing would exceed on-prem TCO.

For most startups and mid-market companies, GroqCloud API is the right choice. The free tier (14,400 requests/day on popular models) is generous for prototyping. Paid tier scales from $10–100/month for hobbyist projects to thousands/month for production systems.

Vendor lock-in: GroqCloud's OpenAI-compatible API minimizes lock-in. Code written for GroqCloud can switch to Together AI, Fireworks, or self-hosted vLLM with minimal changes (swap the base_url in your client). However, Groq's speed advantage disappears if you switch to slower providers, so the real lock-in is performance, not API syntax.

Source: GroqCloud pricing and enterprise documentation (May 2026) — Public API pricing; enterprise terms available by request.

Limitations and the Proprietary Model Problem

Groq's biggest constraint is no proprietary models. If you need GPT-4, Claude, or Gemini, you must use OpenAI/Anthropic/Google's APIs. Groq's model catalog is strong for open-source (Llama, Mixtral, Qwen, DeepSeek), but many enterprises have invested in GPT-4 fine-tuning or rely on GPT-4's reasoning capabilities for complex tasks.

Multimodal is evolving. As of May 2026, Groq supports vision models (Llava variants) but not at frontier quality. For state-of-the-art vision understanding, GPT-4V remains superior. Groq's vision inference is fast but sometimes less accurate than GPU providers.

Custom quantization: If you want to quantize a private model (your company's 70B internal LLM) to 4-bit and run it on Groq, you'll face challenges. Groq's compiler toolchain is optimized for published models; custom weight loading and quantization are not self-serve.

Model updates: Groq's model catalog lags cutting-edge. When DeepSeek-V3 or Llama 5 launches, there's a delay before it's available on GroqCloud. GPU providers like Together AI often have new models within weeks.

Understanding these trade-offs helps match Groq to the right problems: fast inference on open models. For everything else, the inference landscape remains diverse and competitive.

The Broader Inference Ecosystem and Groq's Positioning

Groq is not alone in betting on specialized hardware for LLM inference. Understanding the competitive landscape reveals Groq's strategic positioning:

Cerebras WSE-3: Cerebras' wafer-scale engine is a dinner-plate-sized chip with 900 GB of on-chip memory and 40 million AI cores. It achieves 2,000–3,000 tokens/sec on smaller models, outpacing Groq on throughput. The tradeoff: costs millions with limited availability. Cerebras targets large enterprises and research labs, not the broader API market that Groq serves.

SambaNova RDU: SambaNova's Reconfigurable Data Unit uses a dataflow architecture similar to Groq's. Performance is competitive (~500–600 tok/sec); pricing similar to Groq. SambaNova focused on enterprise sales (banks, healthcare) rather than open API access, limiting visibility compared to GroqCloud's free tier.

NVIDIA GPU Optimization: NVIDIA's vLLM and upcoming inference products continue closing the latency gap through software optimization (paged attention, speculative decoding, continuous batching). GPUs won't match Groq on single-stream latency, but flexibility (supporting proprietary models, fine-tuned weights, multimodal) keeps them competitive for many workloads. For more on GPU architecture, see how GPUs accelerate computation.

Intel Habana Gaudi: Intel's Gaudi chips are optimized for training but also handle inference. Cheaper than H100s with lower power consumption, they appeal to cost-conscious deployers. However, software ecosystem lags NVIDIA and Groq, limiting adoption.

Groq's advantage is simplicity + speed + accessibility: GroqCloud's free tier and straightforward pricing make it accessible to startups. The LPU's deterministic latency is valuable for SLAs. But Groq's dependence on open models is a fundamental constraint that GPUs don't have. This market will likely segment: Groq for latency-critical open-model workloads; GPUs for flexibility and proprietary models; specialized hardware for massive-scale batch processing.

Source: Jouppi, N. P., et al. (2017) "In-Datacenter Performance Analysis of a Tensor Processing Unit." arXiv:1704.04760 — Foundational paper on hardware-software co-design for AI workloads, influencing all subsequent specialized chip designs including Groq's LPU.

The Future of Inference Hardware

Looking forward to 2027–2028, several trends are likely:

• Continued specialization: More custom chips optimized for specific operations (matrix multiplies, attention, activation functions). General-purpose GPUs remain, but specialized hardware will claim growing market share.
• Power efficiency focus: As energy costs and carbon footprint concerns mount, power-efficient inference becomes competitive advantage. Groq's low power per token appeals to environmentally conscious enterprises.
• Model quantization: 8-bit and 4-bit quantization of LLMs reduce memory and compute requirements, making inference feasible on smaller chips and edge devices. This level-playing-field effect may reduce Groq's advantage but expands the total market.
• Multimodal inference: Vision+language models are increasingly demanded. Hardware optimized for multimodal will emerge, though most inference providers (including Groq) are catching up on audio and video.

Frequently Asked Questions

Q: Is Groq the same as Grok (Elon Musk's xAI)?
A: No. Groq (with Q) is the hardware company founded by Jonathan Ross in 2016. Grok (with K) is xAI's chatbot launched in 2023. The name collision is famously annoying, and Groq sent xAI a public open letter about the trademark dispute.

Q: What is an LPU?
A: Language Processing Unit — Groq's custom chip optimized for low-latency LLM inference. Key architectural choices: on-chip SRAM (instead of off-chip HBM), deterministic single-stream execution (no batching), and compiler-driven dataflow design.

Q: Is Groq faster than NVIDIA?
A: For single-stream LLM inference on open models, yes — typically 5–10× faster on tokens-per-second than NVIDIA H100 with vLLM at the same model size. For training, multimodal, or proprietary models, NVIDIA still wins.

Q: Can I run my own model on Groq?
A: Not directly. Groq serves a curated list of open models. Enterprise customers can request custom deployments, but there's no self-serve "upload weights" option as of May 2026.

Q: How does Groq compare to Cerebras?
A: Both bet on on-chip memory instead of HBM. Cerebras's wafer-scale chip is even faster (2,000+ tok/s on smaller models) with more on-chip memory per unit. Groq is more widely available with a broader model catalog. They're competing for similar customers.

Q: Is GroqCloud free?
A: Yes, with generous free tier (thousands of requests daily on popular models). Paid usage runs $0.05–$0.59 per million tokens — competitive with other providers and often cheaper for the speed delivered.

Sources

• Groq company page and GroqCloud documentation (groq.com)
• Artificial Analysis benchmarks, accessed May 2026
• Jonathan Ross interviews on Google TPU origin (various, 2017 – 2024)
• Cerebras Inference launch materials (Aug 2024)
• vLLM project documentation (vllm.ai) for baseline GPU inference numbers