Edge AI: How AI Runs on Your Phone

When you unlock a phone with your face, dictate a message, or ask a device to summarize a notification, a neural network is usually running on the device in your hand, not on a server in another country. That is edge AI: running trained models directly on phones, laptops, cars, and sensors. It sounds simple, but a model built on racks of data-center GPUs has to be squeezed into a chip with a small battery, a tight thermal budget, and a few gigabytes of shared memory. This article explains how that squeezing works and where its limits are.

Why run AI at the edge at all?

Cloud AI has obvious strengths: nearly unlimited compute, easy updates, and the largest models. So why bother pushing work onto the device? There are five practical reasons, and they trade off differently for each application.

Latency. A request to a server has to travel over a network and back, and the delay depends on signal strength, congestion, and distance. For face unlock, camera effects, or a car's lane-keeping system, the answer has to arrive within a frame or two. A model running locally removes the network from the path entirely, although it is still limited by how fast the local chip can run the math.

Privacy. Voice recordings, face images, and health data are sensitive. Processing them on the device means the raw data never has to leave it. Google's developer documentation says this directly about its on-device model: running inference locally "enhances privacy by keeping sensitive data on the device."

Reliability. A cloud feature stops working in a tunnel or on an airplane. An on-device model keeps working because it needs no connection, which the same documentation lists as offline functionality.

Cost. Serving a large model costs the provider money for every request. When the user's own chip does the work, the marginal cost per inference to the provider falls toward zero, though the user pays in battery and device cost.

Bandwidth. Streaming continuous video or audio from millions of devices to servers would take enormous capacity. Local processing sends only results, or nothing at all.

The trade-off is raw capability. A phone chip has a small fraction of the compute and memory of a data-center accelerator. Edge AI is therefore a story of compromise: making models smaller, cheaper to run, and more efficient per watt without making them much worse.

What actually limits a phone: memory, not just math

It is tempting to think the limit is computing speed, the "trillions of operations per second" in a spec sheet. For large language models, the harder limit is usually memory.

Think of a chef (the processor) and a pantry (memory). A fast chef is useless if every ingredient takes a long walk to fetch. Generating one word of text from a language model requires reading essentially all of the model's weights once. If the model is 4 GB and the memory system can deliver, say, 40 GB per second, the ceiling is roughly 10 words per second no matter how fast the chip can multiply. That is a simplified rule of thumb, not an exact law, but it explains why shrinking a model's size usually makes it faster too.

There are two separate memory constraints on a phone:

• Capacity. The model has to fit in RAM alongside the operating system and apps. A 7-billion-parameter model stored at 32-bit precision needs 28 GB for weights alone, more than any phone has.
• Bandwidth. Every generated token pulls the weights through the memory bus, which uses energy and time. Shuttling data through memory also costs energy, which is why compression pays off twice, in space and in power.

Apple researchers addressed a related problem in a 2023 paper, "LLM in a flash." It describes keeping model parameters in flash storage and loading them into DRAM on demand, using techniques called windowing and row-column bundling, and reports running models up to twice the size of available DRAM with inference about 4 to 5 times faster on a CPU and 20 to 25 times faster on a GPU than naive loading (arXiv:2312.11514). The existence of that research shows how central the memory problem is.

Quantization: trading precision for size

Neural network weights are typically trained as 32-bit or 16-bit floating-point numbers. Quantization stores them with fewer bits, often 8 or 4, and sometimes runs the arithmetic in integers too. Because each weight takes less space, the model shrinks and the memory traffic drops.

The size arithmetic for a hypothetical 7-billion-parameter model, counting weights only, is straightforward.

Precision	Bits per weight	Approximate weight size	Typical use
FP32	32	28 GB	Training in some setups
FP16 or BF16	16	14 GB	Cloud inference
INT8	8	7 GB	Server and high-end device
INT4	4	3.5 GB	Phones and laptops
Ternary (about 1.58 bits)	about 1.58	about 1.4 GB	Research

The real figure is a bit higher once you add the scaling factors that quantization schemes store alongside the weights, plus memory for the conversation context (the "KV cache"), which grows with the length of the input.

Why quantization does not wreck accuracy

The central reason is that trained networks are over-provisioned. The exact decimal value of a weight matters less than its rough size and sign, and a network has so many weights that small rounding errors tend to average out. The art is in the exceptions. Some weights and activations are far more important than others, and treating them all the same loses accuracy.

Three research results illustrate how the field handles this.

LLM.int8(). Researchers found that large transformers develop "emergent outlier features," a few dimensions with unusually large values that break simple 8-bit schemes. Their method quantizes most values with vector-wise 8-bit math and keeps the outlier dimensions in 16-bit, which roughly halves inference memory while keeping full-precision accuracy for models up to 175 billion parameters (arXiv:2208.07339).

GPTQ. This one-shot method uses approximate second-order information (how sensitive the model's error is to each weight) to quantize weights to 3 or 4 bits with little accuracy loss. Its authors report quantizing a 175-billion-parameter model in about four GPU hours and running it for generation on a single GPU (arXiv:2210.17323).

AWQ (Activation-aware Weight Quantization). AWQ rests on an observation that "protecting only 1% salient weights can greatly reduce quantization error," where "salient" is judged by activation statistics rather than by the weights themselves. It scales those channels before quantizing, avoids the hardware-unfriendly trick of mixed precision, and ships with an on-device inference framework called TinyChat (arXiv:2306.00978).

For more on how these methods are used in practice, see our explainer on LLM quantization from INT8 to INT4.

Post-training versus quantization-aware training

There are two broad ways to quantize. Post-training quantization takes a finished model and converts it, which is fast but can lose accuracy at low bit-widths. Quantization-aware training (QAT) simulates the rounding during training so the network learns to compensate. A 2017 Google paper by Jacob and colleagues proposed a scheme for integer-only inference together with a co-designed training procedure that preserves end-to-end accuracy, and reported significant gains even on MobileNets, a model family already known for efficiency (arXiv:1712.05877).

How low can you go?

In 2024, a research team described BitNet b1.58, in which every weight is restricted to one of three values: minus 1, 0, or 1. Their abstract claims it matches a full-precision transformer of the same size and training data on perplexity and end tasks, with large gains in latency, memory, throughput, and energy use (arXiv:2402.17764). The paper is labeled work in progress, so treat it as a promising research direction rather than a shipping technique. It also requires training the model that way from the start, rather than compressing an existing one.

Pruning: removing what the network does not need

A trained network contains many weights close to zero that contribute little. Pruning removes them. Imagine editing a long draft: cut the sentences that do no work and the argument still stands, and it gets faster to read.

Unstructured pruning zeroes out individual weights. It can reach high sparsity, but the surviving weights are scattered irregularly, which standard hardware does not exploit well, so file size may shrink without much speedup. Structured pruning removes whole neurons, attention heads, or layers, which maps more directly onto real speed gains because the remaining matrices are simply smaller.

The lottery ticket hypothesis

In a widely cited 2019 ICLR paper, Jonathan Frankle and Michael Carbin proposed that dense, randomly initialized networks contain subnetworks, "winning tickets," that when trained in isolation can match the full network's test accuracy in a similar number of iterations. On MNIST and CIFAR-10 the winning tickets they found were less than 10 to 20 percent of the size of the original networks (arXiv:1803.03635). The takeaway is that much of a network's capacity is scaffolding.

Deep Compression: stacking techniques

The classic demonstration of combining methods is the 2015 "Deep Compression" paper by Song Han, Huizi Mao, and William Dally, which chained three stages: pruning, trained quantization, and Huffman coding. They report shrinking AlexNet from 240 MB to 6.9 MB (35 times) and VGG-16 from 552 MB to 11.3 MB (49 times) without loss of accuracy, with 3 to 4 times layerwise speedup and 3 to 7 times better energy efficiency on mobile and desktop hardware (arXiv:1510.00149). The principle carries over to modern language models, though the specifics differ.

Knowledge distillation: teaching a small model from a large one

Knowledge distillation trains a small "student" model to imitate a large "teacher." The 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean showed that compressing the knowledge of an ensemble of models into a single model works well, including for a heavily used commercial acoustic model (arXiv:1503.02531).

The intuition is the difference between learning from an answer key and learning from a tutor who explains. If the teacher says an image is 92 percent cat, 7 percent bobcat, and 1 percent other, the student learns not only "cat" but also that cats resemble bobcats more than they resemble toasters. Those soft probabilities carry more information per example than a hard label.

DistilBERT is the standard small example. Its authors report reducing the size of a BERT model by 40 percent, making it 60 percent faster, while retaining 97 percent of its language understanding capabilities (arXiv:1910.01108). For a deeper treatment, see our article on AI model distillation.

Distillation is now routine at the frontier. Meta's Llama 3.2 1B and 3B models, released September 25, 2024, were built by applying structured pruning in a single shot to the Llama 3.1 8B model, then using the output logits of the Llama 3.1 8B and 70B models as token-level targets during training to recover quality, according to Meta's announcement. The same post says they support a 128K-token context window and were enabled on day one for Qualcomm and MediaTek hardware and optimized for Arm processors. That single recipe uses pruning and distillation together.

Efficient architectures: designing small from the start

Compression starts with a trained model. Another approach is to design a network that is cheap by construction.

MobileNets (Google, 2017) are built on depthwise separable convolutions, which split a standard convolution into a per-channel spatial step and a cheap channel-mixing step. The authors also introduced two global hyperparameters that let builders trade latency against accuracy to fit a given device (arXiv:1704.04861). The architecture became a common starting point for mobile vision.

SqueezeNet reached AlexNet-level accuracy on ImageNet with 50 times fewer parameters, and under 0.5 MB with compression, which its authors describe as 510 times smaller than AlexNet (arXiv:1602.07360).

For language models, the main architecture-level levers are in the attention mechanism. Standard attention stores a key and value for every earlier token, and that cache grows with context length. Techniques such as multi-query and grouped-query attention share keys and values across attention heads, shrinking the cache; sliding-window attention limits how far back each token looks. Each of these cuts memory use at some cost in flexibility.

Hardware: neural engines and NPUs

Software tricks only go so far. Phone makers also add dedicated silicon that does the core operation of neural networks, large matrix multiplications at low precision, using far less energy than a general-purpose CPU. These blocks go by different names: Apple's Neural Engine, Qualcomm's Hexagon NPU, and the TPU-style blocks in Google's Tensor chips.

Apple's history shows how quickly the category matured. The A11 Bionic in the iPhone X (2017) carried Apple's first Neural Engine, described by Apple as a dual-core design performing up to 600 billion operations per second and used for Face ID and Animoji (Apple Newsroom, 2017). Apple's A18 chip in the iPhone 16 has a 16-core Neural Engine that Apple says runs machine-learning models up to 2 times faster than the A16 Bionic (Apple Newsroom, 2024). The M4 chip's Neural Engine is rated by Apple at up to 38 trillion operations per second (Apple Newsroom, 2024).

On Android, Qualcomm positioned the Snapdragon 8 Gen 3 (2023) around on-device generative AI. Coverage of the launch reported a Hexagon NPU that was 98 percent faster and 40 percent more efficient than the previous generation, and the ability to run a 7-billion-parameter Llama 2 model locally at up to 15 tokens per second (9to5Google). These are manufacturer claims reported at launch, measured under conditions the company chose, and real-world speed depends on the phone, the quantization, and thermal limits.

Note the units. "Operations per second" ratings are peak numbers for specific low-precision operations. They tell you how fast the chip could go on an ideal workload, not how fast a particular language model will generate text. For language models, the memory-bandwidth ceiling described earlier is often the tighter constraint.

Why your phone gets warm

Running a model for a long time draws power and produces heat. Phones have no fans, so when temperature rises the system lowers clock speeds to protect components and the user, which slows the model down. Our article on why thermal throttling slows devices explains the mechanism, and the same energy argument is why the efficiency of an NPU matters more than its peak speed. Battery chemistry sets the other side of the budget; see how lithium-ion batteries work.

On-device language models

Three years ago, running a useful language model on a phone sounded unrealistic. It is now a mainstream product feature, with several designs.

Apple's approach: a small local model plus a private cloud. Apple's 2024 research post describes a roughly 3-billion-parameter on-device language model, compressed with a mixed 2-bit and 4-bit scheme averaging 3.7 bits per weight. On an iPhone 15 Pro, Apple reports a time to first token of about 0.6 milliseconds per prompt token and a generation rate of 30 tokens per second, before further speedups. Small adapters, typically tens of megabytes for a rank-16 adapter, can be loaded and swapped on top of the base model for specific tasks (Apple Machine Learning Research). For requests that need more capacity, a larger model runs on Private Cloud Compute.

Google's approach: a system service. Gemini Nano runs inside Android's AICore system service, which manages distribution and updates so each app does not ship its own large model. Google's documentation lists on-device features such as summarization, proofreading, rewriting, image description, and speech recognition, and notes constraints including token limits, hardware limits, and model capability boundaries, with hybrid inference as the fallback when the local model cannot handle a request (Android Developers).

Open small models. Microsoft's Phi-3-mini has 3.8 billion parameters, was trained on 3.3 trillion tokens, scores 69 percent on the MMLU benchmark (a broad multiple-choice test of knowledge), and is, in its authors' words, "small enough to be deployed on a phone" (arXiv:2404.14219). The Llama 3.2 models covered above fill the same niche. For a survey of what such models are good for, see what small language models are good for and LLMs vs SLMs.

What "private cloud" means

Hybrid designs raise a fair question: if the cloud is involved, what happens to privacy? Apple's description of Private Cloud Compute lists design goals including stateless computation (user data is not retained after the response is returned), no privileged runtime access such as remote shells, resistance to targeting a specific user's requests, and verifiable transparency, with plans to publish every production software image for independent inspection (Apple Security Research). These are Apple's stated design goals, and the post itself does not report independent verification of them.

What edge models can and cannot do

Edge models are good at tasks that are narrow, frequent, and latency-sensitive: classifying images, transcribing speech, detecting a wake word, summarizing a short text, suggesting a reply. They are weaker at long, open-ended reasoning, tasks needing broad up-to-date knowledge, and very long inputs.

Frontier models in the cloud are much larger than anything that fits on a phone today, and several providers do not disclose their parameter counts. The honest comparison is not "phone model versus cloud model on everything" but "which tasks reach acceptable quality at small size." On many everyday tasks, small models are now adequate; on hard ones, they fall behind, which is why hybrid designs exist. If you want to try this yourself on a laptop, our piece on the hidden costs of running a local LLM covers speed, memory, and quality trade-offs, and what MLX is covers Apple's framework for Apple silicon.

Common misconceptions

"On-device means private." Not automatically. It means the model runs locally, but an app can still send data elsewhere. Privacy depends on the whole system, including what is logged and what is uploaded.

"More TOPS means a better AI phone." TOPS is a peak throughput number for a particular precision. For language models, memory bandwidth, software support, and thermal limits often decide the experience.

"A compressed model is just the big model, smaller." Compression changes behavior. Rare facts, unusual languages, and long reasoning chains tend to degrade first. Good benchmarks on the compressed model, not the original, are what count.

"Edge AI will replace the cloud." The trend is hybrid: local models for what they can do quickly and privately, cloud models for what needs scale.

Key Takeaways

• Edge AI runs neural networks on the device for lower latency, offline use, better privacy options, and lower serving cost
• For language models, memory capacity and bandwidth often limit speed more than raw compute does
• Quantization stores weights in fewer bits, shrinking a 7-billion-parameter model from about 28 GB at 32-bit to about 3.5 GB at 4-bit, weights only
• Pruning removes unneeded weights or structures, and structured pruning maps more directly to speedups
• Distillation trains a small model to imitate a large one, and recent small models such as Llama 3.2 1B and 3B combine pruning and distillation
• Neural engines and NPUs run low-precision matrix math efficiently, but peak TOPS figures do not predict real language-model speed
• Hybrid designs, local first and cloud when needed, are the current norm

FAQ

Q: Can you run ChatGPT on a phone? A: Not the full cloud models. They are far larger than a phone can hold, and their exact sizes are not disclosed by the providers. Small models such as Llama 3.2 1B and 3B, Phi-3-mini, and Gemini Nano do run on phones and handle tasks like summarizing, rewriting, and simple questions, with lower quality on hard reasoning.

Q: What is ONNX and why does it matter for edge AI? A: ONNX (Open Neural Network Exchange) is a standard file format for neural networks. It lets a model trained in one framework, such as PyTorch, be converted for another runtime, such as Core ML or TensorFlow Lite, without rewriting it. Because devices differ in chips and software, a portable format shortens the path from training to deployment.

Q: Why does quantization not destroy accuracy? A: Trained networks carry more precision than they need, so small rounding errors tend to average out. The risk lies in outliers and sensitive weights. Methods such as LLM.int8(), GPTQ, and AWQ exist to protect those, which is why 8-bit and 4-bit models are often close to full precision. Accuracy falls off faster at very low bit widths.

Q: How does Private Cloud Compute differ from regular cloud AI? A: Apple's design goals for Private Cloud Compute include stateless processing, no user data retained after the response, no remote administrative shells, and published software images that outside researchers can inspect. A conventional cloud service typically has fewer public commitments of that kind. The post describing it states design goals and does not itself report independent verification.

Q: What is the difference between an NPU, a GPU, and a CPU for AI? A: A CPU is a general-purpose processor that handles sequential logic well but parallel matrix math less efficiently. A GPU is built for massive parallel arithmetic, which suits both training and inference. An NPU is specialized for the low-precision matrix operations in neural networks and is designed for energy efficiency, which matters on battery-powered devices. Training large models still happens mostly on GPUs and similar accelerators in data centers.

Q: Will edge AI replace cloud AI? A: Unlikely in the near term. Edge handles latency-sensitive, privacy-sensitive, and frequent tasks. The cloud handles the largest models, training, and tasks needing fresh information. Products increasingly use both, deciding case by case where each request runs. The boundary moves as phone hardware and compression methods improve.

The takeaway

Edge AI is less a single technology than a set of compromises made well. Quantization trades bits for size, pruning trades redundancy for speed, distillation trades a teacher's knowledge for a student's compactness, and neural engines trade generality for efficiency. The constraint that ties them together is memory, because every generated word has to move the model's weights through a narrow bus. Expect the on-device ceiling to keep rising, but expect the sensible architecture to remain hybrid for some time, with the small model close to you and the large one far away.

Sources

• Sanh et al. (2019), DistilBERT; Hinton, Vinyals, and Dean (2015), Distilling the Knowledge in a Neural Network.
• Frankle and Carbin (2019), The Lottery Ticket Hypothesis; Han, Mao, and Dally (2015), Deep Compression.
• Howard et al. (2017), MobileNets; Iandola et al., SqueezeNet.
• Jacob et al. (2017), Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference; Dettmers et al., LLM.int8(); Frantar et al., GPTQ; Lin et al., AWQ; BitNet b1.58.
• Alizadeh et al., LLM in a flash; Phi-3 Technical Report.
• Apple, Introducing Apple's On-Device and Server Foundation Models and Private Cloud Compute.
• Apple Newsroom: iPhone X, iPhone 16, M4.
• Meta AI, Llama 3.2 announcement; Google, Gemini Nano on Android; 9to5Google, Snapdragon 8 Gen 3.

---

Read next:

• Generative Adversarial Networks: How GANs Work
• How Diffusion Models Work: Stable Diffusion, DALL-E, and the Science of AI Art
• How Do Neural Networks Work? The Architecture Behind Modern AI
• AI Training vs. Inference
• How GPUs Work: Why AI Needs Them