What Is Ollama? Run LLMs Locally Explained

Ollama is an open-source tool that lets you download and run open-weight large language models (LLMs) locally on macOS, Linux, and Windows with a single command. It was first released in mid-2023 by Michael Chiang and Jeffrey Morgan, who wrapped the high-performance C/C++ inference engine llama.cpp in a small Go binary and exposed it through a friendly command-line interface (ollama run, ollama pull, ollama list) and an HTTP API on port 11434. By mid-2026, with the project on release v0.32.0 and a Series B round closed in July, Ollama has become the default on-ramp for developers who want to experiment with local LLMs without writing a line of inference code.

What problem Ollama actually solves

Before Ollama, running an open-weight model on your own machine meant compiling llama.cpp from source, tracking down the right GGUF weight file from a scattered set of Hugging Face repositories, guessing at the correct quantization for your hardware, and writing your own thin server if you wanted an API. Every step was a place to get stuck, and every model required repeating the process from scratch.

Ollama collapses all of that into one command line and a background service. Type ollama run llama3.2 and, behind the scenes, three things happen in sequence.

First, Ollama checks its model registry (or a Hugging Face GGUF mirror as of the 0.30 release) and pulls a pre-quantized weight file if you don't already have it cached. Second, it spins up a llama.cpp-based inference process that loads those weights into RAM or VRAM and keeps them warm between requests. Third, it exposes a REST API — including an OpenAI-compatible endpoint at http://localhost:11434/v1/chat/completions — so any tool built for GPT-style APIs works against your local model with a one-line change to the base URL.

Think of it the way you'd think of Docker for containers. Before Docker, running an isolated Linux process meant hand-configuring cgroups and namespaces; Docker turned that into docker run. Ollama does the same job for model weights: it turns "compile an inference engine and manage multi-gigabyte files by hand" into "run one command."

Ollama's GitHub repository has accumulated more than 150,000 stars since its 2023 launch, and by mid-2026 the project has closed a Series B funding round backed by investors including 8VC, Garage Capital, and GTMfund — evidence that "run an LLM locally" moved from hobbyist trick to venture-backed infrastructure in under three years. ([Ollama GitHub](https://github.com/ollama/ollama))

How Ollama works under the hood

The model format: GGUF and quantization

Every model Ollama runs is stored as a GGUF file — a single-file binary format that packages model weights, tokenizer data, and metadata together, designed specifically for llama.cpp-family inference engines. GGUF replaced the older GGML format in 2023 to support better versioning and a wider range of model architectures without breaking backward compatibility. (Hugging Face GGUF documentation)

GGUF files come in different quantization levels, which determine how many bits are used to store each weight. A model at full precision (fp16) uses 16 bits per parameter; a q4_K_M quantized version uses roughly 4 bits per parameter on average, cutting file size by close to 4x with a modest quality trade-off. Ollama defaults new pulls to q4_K_M because it hits the sweet spot most people want: a 7-billion-parameter model that would need about 14 GB at fp16 shrinks to roughly 4–5 GB, letting it fit on a consumer laptop's GPU or unified memory. If you want to understand exactly how INT4 and INT8 quantization reduce model size and where the accuracy loss comes from, see our deeper explainer on LLM quantization.

The inference engine: llama.cpp and MLX

For most of its life, Ollama has run entirely on llama.cpp, a C/C++ project originally written by Georgi Gerganov to run Meta's LLaMA models on a MacBook without a GPU. llama.cpp implements the transformer forward pass — the same attention and feed-forward math that powers GPT-style models — but optimized for CPU SIMD instructions and consumer GPU memory layouts rather than the data-center-scale batched matrix multiplication that frameworks like PyTorch assume.

As of the Ollama 0.19 release in March 2026, Apple Silicon machines get a second option: Apple's own MLX framework, a machine learning library built specifically for the unified-memory architecture of M-series chips. On an M-series Mac, MLX-backed inference in Ollama measurably outperforms the llama.cpp/Metal path for certain model families because MLX was designed from scratch around Apple's memory bandwidth and Neural Engine, rather than being retrofitted onto it. On Linux and Windows with NVIDIA hardware, Ollama uses CUDA; on supported AMD cards it uses ROCm; and the 0.30 release added Vulkan support as a fallback path that works across a wider range of GPUs without vendor-specific drivers.

What happens when you type ollama run

1. Ollama checks whether the requested model tag is already cached locally in ~/.ollama/models.
2. If not, it pulls the GGUF manifest and layers from the registry, resuming interrupted downloads automatically.
3. It loads the weights into RAM (CPU inference) or VRAM/unified memory (GPU-accelerated inference), offloading as many transformer layers to the GPU as will fit.
4. It starts a local HTTP server that exposes chat, generate, and embeddings endpoints.
5. The CLI opens an interactive prompt session against that server; closing the CLI does not necessarily stop the server, which by default keeps the model loaded in memory for a few minutes to serve follow-up requests fast.

What models are actually good for what

The single most surprising fact for newcomers: a locally-run 7B or 8B model is not a smaller GPT-5. It is a genuinely different tier of capability, trained on far less compute, and it shows most clearly on multi-step reasoning and long-context tasks — not on casual conversation, where small models can feel deceptively competent.

Supported models and the 2026 library

Ollama's library now hosts several hundred curated models, and as of the 0.30 release it can also pull arbitrary GGUF weights directly from Hugging Face when a fine-tune or merge isn't in the official catalog. (Ollama model library)

Model family	Typical sizes	Best for
Meta Llama 3.1 / 3.2 / 3.3	1B–405B	General-purpose chat, broad ecosystem support
Mistral / Mixtral	7B, 8x7B, 8x22B (MoE)	Fast, efficient reasoning at low VRAM cost
Alibaba Qwen 2.5 / Qwen3	0.5B–72B, Coder variants	Coding, multilingual tasks, strong at small sizes
Google Gemma 2 / 3	2B, 9B, 27B	Lightweight, efficient on-device inference
Microsoft Phi 3 / 3.5	3.8B, 14B	Reasoning-tuned small models
DeepSeek V3 / R1 distills	1.5B–671B	Chain-of-thought reasoning, coding
OpenAI gpt-oss	20B, 120B	Open-weight release from OpenAI, Apache-licensed
Embedding models	nomic-embed-text, mxbai-embed-large, bge-m3	Retrieval-augmented generation pipelines

Table: Ollama's most-pulled model families as of mid-2026, sourced from the Ollama library.

For models too large to fit on any consumer machine — gpt-oss:120b, qwen3-coder:480b, deepseek-v3.1:671b — Ollama added a hosted option called Ollama Cloud in 2026, which runs those weights on remote infrastructure but keeps the same local CLI and API surface, so a script written against localhost:11434 can point at a cloud-hosted large model with a config change rather than a rewrite. (Releasebot Ollama changelog)

Hardware requirements: what you actually need

The rule of thumb that's held steady since Ollama's early days: budget roughly 1 GB of memory per 1 billion parameters at 4-bit quantization, plus a few extra gigabytes of overhead for the context window, the KV cache, and the operating system.

Model size	4-bit RAM/VRAM needed	Practical hardware	Typical throughput
3B (Llama 3.2)	~3 GB	Any modern laptop, no discrete GPU required	30–60 tokens/sec on CPU
7B / 8B	~6–8 GB	16 GB Mac, RTX 3060	20–40 tokens/sec
13B	~10 GB	16 GB Mac, RTX 3080	15–25 tokens/sec
30B / 34B	~22 GB	32 GB Mac, RTX 3090 / 4090	10–20 tokens/sec
70B	~42 GB	64 GB Mac, 2x RTX 4090, single A100	5–12 tokens/sec
120B–405B	~80–240 GB	Multi-GPU server or Ollama Cloud	Varies by cluster

Table: Approximate memory needs and observed throughput for 4-bit quantized models, compiled from Ollama documentation and community benchmarks. Actual tokens-per-second varies significantly with GPU model, context length, and whether layers are fully offloaded to the GPU.

Why RAM isn't the only constraint

Memory capacity determines whether a model loads at all, but memory bandwidth determines how fast it generates text. This is the detail that trips up a lot of newcomers who buy a machine with "enough RAM" and are still disappointed by speed. Generating each new token requires reading the entire set of model weights from memory once, so a model that just barely fits in VRAM on a card with slow memory bandwidth will feel sluggish even though it technically "runs." This is exactly why Apple's M-series chips punch above their weight for local inference: their unified memory architecture gives the GPU direct access to system RAM at bandwidth figures that rival mid-range discrete GPUs, without needing to copy data across a PCIe bus first.

H3: How much RAM do I need to run a 7B model locally?

A 7-billion-parameter model quantized to 4-bit needs roughly 6–8 GB of RAM or VRAM to load comfortably, including headroom for the context window. On Apple Silicon, 16 GB of unified memory is the practical minimum for smooth day-to-day use; on a discrete NVIDIA GPU, a 12 GB card like an RTX 3060 handles it well.

H3: Does Ollama use my GPU automatically?

Yes. Ollama detects available GPU acceleration (Metal on Apple Silicon, CUDA on NVIDIA, ROCm on supported AMD cards, or Vulkan as a fallback) and automatically offloads as many transformer layers as fit in VRAM, falling back to CPU for the remainder. You don't need to configure this manually in most cases.

CLI basics and the Modelfile

# install (macOS)
brew install ollama

# pull and chat
ollama run llama3.2

# list installed models
ollama list

# remove a model
ollama rm llama3.2

# check what's currently loaded in memory
ollama ps

# create a custom variant via Modelfile
ollama create my-coder -f ./Modelfile

A Modelfile is a small declarative configuration file — think of it as a Dockerfile for a model instead of a container image. It specifies a base model, generation parameters, and a system prompt:

FROM qwen2.5-coder:7b
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
SYSTEM "You are a senior Rust engineer. Reply with code only."

Running ollama create my-coder -f ./Modelfile bakes those settings into a new named model you can ollama run my-coder from then on, without having to pass the system prompt and parameters every time.

H3: What is a Modelfile used for in Ollama?

A Modelfile lets you create a customized, reusable version of a base model with a fixed system prompt, temperature, and context length baked in, without fine-tuning any weights. It's the fastest way to turn a general-purpose model into a task-specific one, such as a code reviewer or a customer-support assistant with a locked persona.

Using Ollama from Python and other frameworks

Because Ollama exposes an OpenAI-compatible endpoint, you can point the official OpenAI Python SDK at it with a two-line change:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain HNSW in one sentence."}],
)
print(resp.choices[0].message.content)

The same trick works for LangChain, LlamaIndex, Continue, Cline, and Aider, since all of them can be configured to hit a custom base_url instead of OpenAI's servers. If you're deciding which of the two major RAG (retrieval-augmented generation) frameworks to build on top of a local Ollama model, see our LangChain vs LlamaIndex comparison — both integrate with Ollama out of the box.

Ollama also ships a native ollama-python and ollama-js client library with its own request format if you'd rather not go through the OpenAI-compatibility shim, plus support for tool calling (also called function calling), which lets a model request that your code execute a specific function and feed the result back in — the same mechanism that underlies most AI agent frameworks. For a deeper look at how that pattern generalizes into fully autonomous, multi-step systems, see our AI agents explainer.

Comparing Ollama to other local LLM runners

Ollama is not the only way to run open-weight models locally, and it isn't the fastest option for every situation. The honest comparison depends heavily on whether you're prototyping alone or serving a team.

Tool	Interface	Engine	Strengths	Weakness
Ollama	CLI + REST API	llama.cpp / MLX	Easiest install, huge library, OpenAI-compatible API, tool calling	Not built for multi-user throughput
LM Studio	GUI	llama.cpp / MLX	Friendly desktop app, built-in model browser, good on low-spec hardware	Closed source
Jan	GUI	llama.cpp	Open source, privacy-first, no telemetry	Smaller ecosystem and model library
llama.cpp	CLI / server	native	Lowest-level control, smallest footprint	Manual model management, steeper learning curve
vLLM	Python server	custom CUDA	~2.3x higher throughput than Ollama at 8+ concurrent users, continuous batching	Linux + NVIDIA only, heavier setup
MLX (Apple)	Python	Apple's framework	Best raw performance on M-series chips	Apple Silicon only
KoboldCpp	Web UI	llama.cpp fork	Strong roleplay and long-form story features	Niche use case, smaller community
TGI (Hugging Face)	Server	Rust	Production-grade serving, wide model support	Heavier operational overhead

Table: Local and self-hosted LLM tooling landscape as of mid-2026, based on published benchmark comparisons. (codersera vLLM vs Ollama vs LM Studio benchmark, 2026)

The clearest way to think about the trade-off: at a single concurrent user, Ollama and vLLM perform in a near dead heat, because there's no queue of competing requests for vLLM's batching engine to optimize. Once you're serving eight or more simultaneous users, vLLM's continuous batching and PagedAttention memory management pull meaningfully ahead — roughly 2.3x the throughput in published 2026 benchmarks — because it's solving a fundamentally different problem: keeping a GPU saturated across many overlapping requests rather than serving one person's conversation as fast as possible. (codersera production benchmark)

H3: Is Ollama better than LM Studio?

Neither is strictly better; they solve different preferences. Ollama is CLI-first and integrates more easily into scripts, IDEs, and automated pipelines. LM Studio is GUI-first, ships a built-in model browser, and tends to be friendlier for people who don't want to touch a terminal, including reasonable performance on machines with only integrated graphics.

H3: Can Ollama replace vLLM in production?

For a single user or a small team with light, non-simultaneous usage, yes. For serving many concurrent users with predictable latency, no — published 2026 benchmarks show vLLM delivering roughly double the throughput or more under concurrent load, because it was purpose-built for batched, multi-tenant serving rather than single-session convenience.

The ecosystem that's grown up around Ollama

By 2026, a whole layer of tooling assumes Ollama is running in the background as a local inference backend:

• Open WebUI — the most widely deployed self-hosted, ChatGPT-style chat interface. One Docker command gives you a multi-user web app pointed at your Ollama instance, complete with chat history, document upload, and user accounts.
• Continue.dev and Cline — VS Code extensions that use a local Ollama model for in-editor code completion and chat, so proprietary code never leaves the machine.
• Aider — a terminal-based AI pair programmer that edits files directly in your repository and supports Ollama through its OpenAI-compatible URL.
• LangChain and LlamaIndex — both ship first-party Ollama integrations, letting you swap a cloud model for a local one in a retrieval pipeline with a one-line change.
• ollama launch — introduced in January 2026, this command auto-configures popular coding tools to point at a local model, removing the manual environment-variable setup that used to be the biggest friction point for new users. (webscraft.org Ollama 0.30 changelog)

If your goal with local models is building a document-search or question-answering system rather than a chat assistant, you'll also need a place to store the numerical representations of your text. See our explainer on what a vector database is for how embeddings, similarity search, and retrieval-augmented generation fit together with a local model like the ones Ollama serves.

Where local inference genuinely wins

• Privacy-sensitive workflows. Legal documents, medical records, and internal corporate data that legal or compliance teams won't let touch a third-party API can be processed entirely on hardware you control.
• Zero marginal cost per token. Once you own the hardware, generating another million tokens costs electricity, not a metered API bill — useful for development loops, batch document processing, and high-volume agent workflows that would get expensive fast on a pay-per-token API.
• Offline and air-gapped environments. A laptop on a flight, a secure government network, or a factory floor without reliable internet can still run a capable assistant.
• Fine-tuning and adapter experimentation. Developers often prototype LoRA (low-rank adaptation) fine-tunes locally before committing to paid cloud GPU time for a full training run.
• Education without an API key. Students learning prompt engineering, retrieval-augmented generation, or agent design can experiment freely without signing up for a paid account or worrying about usage limits.

Where local inference genuinely loses

The quality gap between the best local models and frontier hosted models is real and worth naming plainly: it is not a marketing exaggeration, and it shows up most on tasks that require multiple steps of reasoning, not on casual chat.

• Quality on hard reasoning tasks. Even a 70-billion-parameter open model trails the current generation of frontier hosted models on multi-step math, complex code generation, and long-document reasoning. Smaller 7B–13B models hallucinate more frequently and are noticeably weaker at following long, multi-constraint instructions.
• Raw speed for a single query. A consumer GPU rarely beats a well-optimized cloud API for single-request latency, because API providers run continuously-batched, purpose-built inference clusters that amortize cost across thousands of simultaneous users.
• Context window ceilings. Most locally-runnable quantized models cap out well below what top hosted models offer, and pushing context length up drives memory use for the KV cache sharply higher — doubling context length roughly doubles KV cache memory for a fixed model size.
• No built-in multi-user batching. Ollama serves one conversation efficiently but wasn't designed to keep a GPU saturated across many concurrent users the way vLLM or TGI were. Trying to put Ollama directly behind a busy production API is where teams hit a wall.
• Lighter safety tuning. Open-weight models generally ship with less extensive safety fine-tuning and red-teaming than closed frontier APIs, which matters for anything user-facing without your own moderation layer.

To go further into how those local models get compressed small enough to run on a laptop in the first place — and where accuracy is actually lost in that process — read our explainer on LLM quantization, covering INT8, INT4, GPTQ, and AWQ.

Common confusions and myths about Ollama

"Ollama is a model." It isn't. Ollama is the runtime and packaging layer; the models — Llama, Qwen, Mistral, Gemma, and dozens of others — are separately trained and licensed. Ollama's own code is MIT-licensed, but each model file you pull carries its own license from its original creator, and those licenses vary in what they permit for commercial use.

"A local model is just a smaller, private ChatGPT." The underlying architecture (a transformer trained to predict the next token) is the same family of technology, but the training compute, dataset scale, and reinforcement learning from human feedback (RLHF) that goes into a frontier hosted model dwarfs what goes into most open-weight releases. Local models are a genuinely different capability tier, not a scaled-down clone with identical behavior.

"Running Ollama makes your machine part of a distributed network." No — unless you explicitly opt into Ollama Cloud for a specific large model, everything runs entirely on your own hardware, with no data leaving the machine and no participation in any shared compute pool.

"More parameters always means a better answer." Not for every task. A well-tuned 7B coding-specific model like Qwen2.5-Coder will often out-perform a generic 70B chat model on a narrow coding task, because specialization and instruction-tuning quality matter as much as raw parameter count for a given use case.

H3: Is Ollama free to use?

Yes. Ollama itself is MIT-licensed and free for both personal and commercial use. The model weights you download through it carry their own separate licenses — the Llama 3 community license, Apache 2.0 for Mistral and Qwen, and so on — so check the specific model's license if you plan to redistribute outputs commercially.

H3: How is Ollama different from llama.cpp?

Ollama is a higher-level wrapper built on top of llama.cpp (and, on Apple Silicon since 2026, MLX). It adds a model registry, automatic downloads, a REST API, Modelfiles for customization, and background model lifecycle management. llama.cpp itself gives you lower-level control and a smaller footprint but requires manually compiling the engine and sourcing model files yourself.

H3: Can I run Ollama in production?

For single-user or low-concurrency internal tools, yes — plenty of teams run Ollama behind an internal API for a handful of engineers. For customer-facing production serving with many simultaneous users and predictable latency under load, purpose-built serving systems like vLLM or Hugging Face's TGI generally outperform it by a wide margin, per 2026 throughput benchmarks.

H3: Does Ollama send any of my data to the cloud?

By default, no. Standard local model usage runs entirely on your machine and Ollama's servers are only contacted to check for and download model weights. Data leaves your machine only if you explicitly opt into a network-facing feature like Ollama Cloud for a model too large to run locally.

The takeaway

Ollama didn't invent local LLM inference — llama.cpp and the broader open-weight model movement did that groundwork years earlier. What Ollama did was remove the friction between "a capable open-weight model exists" and "I'm actually running it," collapsing a multi-step compile-and-configure process into a single command and a REST API that speaks the same language as OpenAI's SDK. That's why it became the default choice for privacy-sensitive, offline, and cost-conscious LLM work rather than a niche developer tool. The honest trade-off to keep in mind is capability, not convenience: local models are a genuinely different, currently weaker tier than frontier hosted APIs on hard reasoning tasks, and Ollama itself isn't built for high-concurrency production serving the way vLLM is. Pick it for what it's actually good at — fast local prototyping, private data, offline use, and zero per-token cost — and reach for a different tool once your bottleneck becomes throughput at scale.

---

Read next:

• LangChain vs LlamaIndex — LLM Framework Compared
• AI Agents Explained — How Autonomous AI Systems Actually Work
• LLM Quantization — Run Models Locally

Sources

• Ollama official site — https://ollama.com
• Ollama GitHub — https://github.com/ollama/ollama
• Ollama model library — https://ollama.com/library
• llama.cpp — https://github.com/ggerganov/llama.cpp
• Hugging Face GGUF documentation — https://huggingface.co/docs/hub/gguf
• Meta Llama 3 release — https://ai.meta.com/blog/meta-llama-3/
• Ollama 0.30 release notes (GGUF, Vulkan, NVIDIA acceleration) — https://webscraft.org/blog/ollama-030-scho-novogo-gguf-vulkan-llamacpp-i-tool-calling?lang=en
• Ollama release notes tracker — https://releasebot.io/updates/ollama
• vLLM vs Ollama vs LM Studio production benchmark (2026) — https://codersera.com/blog/vllm-vs-ollama-vs-lm-studio-production-2026/
• Ollama company funding profile — https://getlatka.com/companies/ollama.com