AI Compiler Optimization and Code Generation, Explained

A compiler turns source code into machine instructions, and along the way it makes thousands of small decisions: whether to copy a function's body into its caller, which values deserve a spot in a CPU register, in what order to run its own passes. For decades those decisions came from hand-tuned rules of thumb. AI compiler optimization swaps some of those rules for models trained on real programs. The measured gains are modest, a few percent, but they are real, and they now ship in LLVM, the compiler infrastructure behind Clang, Rust, and Swift.

What a compiler decides, and where the rules of thumb run out

A modern optimizing compiler such as LLVM or GCC does not translate code in a single sweep. It converts the program into an intermediate representation (IR), a simplified, machine-independent form, and then runs dozens of passes over it. Each pass rewrites the IR to make the program smaller or faster while keeping its behavior identical. A few of the decisions that matter most:

• Function inlining. Replace a call with a copy of the function's body. This removes call overhead and exposes new optimizations, but it makes the program bigger, and a bigger program can crowd the CPU's instruction cache.
• Loop transformations. Unroll a loop to cut bookkeeping, vectorize it so one instruction handles several data elements, or fuse two loops so data stays in cache.
• Register allocation. Decide which values live in the handful of fast CPU registers and which get pushed out to slower memory. When registers run out, a value is "evicted," and the program pays for extra loads and stores.
• Instruction scheduling. Reorder instructions so the processor is never idle waiting on memory.

Think of a chef working at a tiny counter. The counter is the register file, and the pantry down the hall is main memory. Every ingredient left on the counter saves a trip, but the counter holds only a few items, and which ones to keep close depends on the recipe about to be cooked. Register allocation is that decision, made millions of times per build.

Compiler engineers have encoded their experience as heuristics: if a function is smaller than some threshold and called from a hot loop, inline it. Heuristics are fast and predictable, and they are tuned on benchmarks that may not look like your code. They also interact. Inlining changes what the loop optimizer sees, and the loop optimizer changes what the register allocator faces. No one can reason through every combination by hand, which is the opening that machine learning uses.

Google's MLGO: the best-documented production case

The clearest public example is MLGO (Machine Learning Guided Optimization), a framework from Google that integrates learned models into LLVM. Google Research described it in a July 2022 blog post as the "first industrial-grade general framework" for the job. It covers two optimizations: inlining for code size and register allocation for performance (Google Research, July 6, 2022). Both live in the main LLVM repository, and the training tools are open source (google/ml-compiler-opt on GitHub).

The numbers below are the ones Google published. They are worth reading carefully, because popular summaries of this work often attach the wrong figure to the wrong optimization.

Optimization	Goal	Reported result	Source
Inlining	Smaller code	Size reduced by 3% to 7% on software other than the training set	Google Research blog, 2022
Inlining	Smaller code	Up to 7% size reduction compared with LLVM's -Oz level	Trofin et al., arXiv:2101.04808
Inlining in Fuchsia	Smaller code	6.3% size reduction for C++ translation units	Google Research blog, 2022
Register allocation	Speed	0.3% to 1.5% more queries per second on internal datacenter applications	Google Research blog, 2022

Two points follow. First, the inlining policy was trained to shrink binaries, not to speed them up. Second, the register-allocation policy was the one tuned for speed, and its gain is small in percentage terms. A figure like 1% sounds trivial until you multiply it by a large fleet. As an illustration only, a service that needs 100,000 cores would free up about 1,000 cores at a 1% improvement, and Google runs far more than that.

How the agent learns to decide

MLGO uses reinforcement learning, a method in which a model learns from trial and error against a reward signal. The loop is simple to describe. The compiler runs with the current model making the decisions, logs each decision with the features that prompted it, and a trainer updates the model. Google's post says the training uses policy gradient and evolution strategies (Google Research). The repository's README says policy gradient is what the public tooling supports, with evolution strategies planned.

At each call site, the model sees a numeric description of the situation. The blog post gives examples of the features, such as the number of basic blocks, the number of users of a function, and the call site's height in the call graph. It answers one question: inline or not. For register allocation, the model decides how to handle the eviction of live ranges, the stretches of code in which a value must stay in a register, and the reward is tied to the extra loads and stores that eviction and splitting cause.

An analogy helps with the training process. A new hire learns to pack suitcases by watching an expert, then packs a few hundred and gets feedback on how well each one closed. Over time, the new hire learns patterns the expert never wrote down, such as "bulky items first when the bag is small." The reward signal is the feedback, and the policy is the packing habit.

Why the model does not produce wrong code

The most common worry about AI in compilers is that a model might break a program. In the MLGO design, it cannot, for a structural reason. LLVM's documentation says MLGO integrates machine learning "to replace heuristics within LLVM with machine learned models." A heuristic only chooses among transformations that the compiler has already proven legal. The model takes the heuristic's place, so it can pick a worse option, such as inlining something that bloats the binary, but it cannot invent an illegal one (LLVM MLGO documentation).

That boundary is why the approach is deployable. The failure mode of a bad model is a slower or larger program, which a benchmark can detect, rather than a miscompiled one, which can corrupt data silently.

How the model ships inside the compiler

A compiler cannot ship with a Python runtime. MLGO's answer is to compile the trained TensorFlow model ahead of time. Google's post says the model is embedded using XLA's ahead-of-time compilation, which converts it to executable code, avoids the TensorFlow runtime dependency, and reduces the inference overhead at compile time. LLVM's documentation describes a "development mode" that loads models dynamically for training, and a "release mode" that embeds pre-trained models at build time. A third path, in which models are lowered through MLIR to C, is described as planned to replace the TensorFlow route.

Durability over time

Software changes, and so do compilers. A policy trained on last year's code might degrade. Google's post addresses this directly: it reports that the inlining policy still performed well three months after training, with only slight degradation, and that the register-allocation gains persisted for months after deployment. The post does not describe a fixed retraining schedule, and it lists better algorithms and more features among future work. For anyone planning to adopt the technique, drift is the operational question to ask first.

Tensor compilers: searching a space too large to measure

The second major arena is machine learning's own workload. A neural network is a graph of tensor operations: matrix multiplications, convolutions, element-wise functions. The same mathematical operation can be executed in thousands of ways, depending on how loops are tiled, which dimension is parallelized, and how data moves between memory levels. On a GPU or a phone chip, the fastest schedule can be several times faster than a naive one.

Two obstacles make the search hard. The number of candidate schedules is enormous, and measuring each candidate on real hardware takes time. The standard remedy is a learned cost model: a neural network or similar predictor that estimates a schedule's speed without running it. A food critic who can predict a dish's rating from the recipe saves the restaurant from cooking every variation. The critic is imperfect, so the best few candidates are still cooked and timed.

System	Year and venue	Approach	Reported result
AutoTVM (Learning to Optimize Tensor Programs)	NeurIPS 2018	Statistical cost models guide the search for operator implementations, with transfer across workloads	Performance "competitive with state-of-the-art hand-tuned libraries" on low-power CPUs, mobile GPUs, and server GPUs
Halide learned autoscheduler	SIGGRAPH 2019	Beam search over a large schedule space, ranked by a cost model trained on hundreds of thousands of random programs	Nearly 2x faster than the previous Halide autoscheduler, over 2x with autotuning, and the first to significantly outperform human experts on average
Ansor	OSDI 2020	Hierarchical sampling of programs, evolutionary search guided by a learned cost model, and a task scheduler across subgraphs	Up to 3.8x on Intel CPUs, 2.6x on ARM CPUs, and 1.7x on NVIDIA GPUs versus the state of the art

Sources: arXiv:1805.08166, Halide autoscheduler page, arXiv:2006.06762.

The comparisons in the right-hand column are against the baselines each paper chose, which differ by system and hardware, so the numbers should not be compared across rows. The pattern across all three is the same: machine learning supplies the prediction, and a conventional search procedure supplies the structure.

Triton and torch.compile: a language for writing kernels

A kernel is a small program that runs on a GPU. Writing a fast one in CUDA, NVIDIA's programming platform, demands detailed knowledge of the hardware's memory hierarchy. Triton is a Python-based language and compiler that lets a developer describe a kernel in terms of blocks of data, and leaves tiling and memory-access details to the compiler. It originated in a 2019 academic paper by Philippe Tillet, H. T. Kung, and David Cox at Harvard, and was later developed at OpenAI. It is not an NVIDIA product, which older summaries sometimes state.

Triton's biggest deployment is inside PyTorch. PyTorch 2.0 made torch.compile its main compilation API. Its compiler backend, TorchInductor, uses OpenAI Triton as a building block on NVIDIA and AMD GPUs, and generates C++ code with multithreading and vector instructions for Intel CPUs. The PyTorch team reported that Triton-generated kernels reach performance on par with hand-written kernels and specialized CUDA libraries such as cuBLAS. They also reported that torch.compile worked on 93% of 165 open-source models and ran 20% faster on average at float32 precision and 36% faster on average with automatic mixed precision (PyTorch 2.0 release notes).

Be clear about what is and is not machine learning here. TorchInductor and Triton are compilers that generate code, and the PyTorch notes do not describe a neural network making their decisions. They matter to this story because they are the infrastructure that learned optimization plugs into, and the reason hand-tuned kernels are no longer the only path to speed. If you want the framework-level comparison, our guide to PyTorch, TensorFlow, and JAX covers the trade-offs, and the article on why AI needs so much computing power explains why this kernel work pays off.

Learning to write the assembly itself: AlphaDev

A more radical idea skips the heuristic and asks a model to produce the low-level code. In June 2023, DeepMind described AlphaDev, a reinforcement learning system built on AlphaZero that works at the level of assembly instructions. It treats sorting as a single-player game: the system adds one instruction at a time and is rewarded for correctness and speed. It found sorting routines for sequences of three to five elements that were up to 70% faster for shorter sequences, and about 1.7% faster for sequences of more than 250,000 elements. The routines were added to LLVM's libc++ sorting library, the first change to that part of the library in over a decade (DeepMind, June 7, 2023).

The result is narrow but telling. AlphaDev found "swap and copy" moves that save one instruction each time they run, and the post compares them to the unconventional moves of AlphaGo. The same project's hashing algorithm for 9 to 16 byte inputs was reported as 30% faster and released in the Abseil library. If you are curious about the game-playing lineage, see our explainer on AlphaGo and the broader piece on reinforcement learning.

The caution is scope. These gains are for small, heavily used library routines, where a saved instruction is multiplied across billions of calls. They do not mean a model can now rewrite arbitrary programs better than a compiler.

Language models inside the compiler loop

A newer line of work trains language models directly on compiler artifacts. In June 2024, Meta released the Meta Large Language Model Compiler, built on Code Llama in 7-billion and 13-billion parameter sizes and trained on about 546 billion tokens of LLVM IR and assembly code. Fine-tuned versions for code-size optimization reached 77% of the optimizing potential of an autotuning search, a brute-force method that tries many pass sequences. Fine-tuned disassembly models translated x86-64 and ARM assembly back to LLVM IR with a 45% round-trip success rate, 14% of them exact matches (Meta AI research page).

Those figures cut both ways. Reaching three-quarters of a costly search's benefit with a single model pass is a useful shortcut. A 45% round-trip rate for disassembly shows how far such models are from replacing deterministic tools, where an error rate in that range would be unacceptable without a verifier. For background on the underlying technology, see our explainer on how large language models work.

Phase ordering and autotuning

Beyond single decisions there is a meta-problem: in what order should the passes run, and how many times? The standard optimization levels, such as -O2 and -O3, use fixed pipelines. Researchers have long known that a different sequence can do better on a particular program, and that finding it is a search problem with an astronomical number of possible sequences.

CompilerGym, released by researchers in September 2021, packages this and related tasks as environments using the OpenAI Gym interface, so that reinforcement learning researchers can work on real compiler problems without building the plumbing (arXiv:2109.08267). The aim is a shared testbed, so that results from different groups can be compared on the same tasks.

A common mix-up is worth correcting. Google's AutoFDO is sometimes described as an ML system. It is not. It samples hardware performance counters on production machines and feeds the resulting profiles back to the compiler. The CGO 2016 paper reported a geometric-mean speedup of about 10.5% on the authors' benchmarks, reaching 85% of the gains of traditional feedback-directed optimization, and the paper's abstract says over half of the CPU cycles used are now spent in some flavor of FDO-optimized binaries (Google Research). It is a data-driven technique, and it can be combined with learned policies, but its gains are from profiles rather than a trained model.

Code generation: what language models write

The other half of the title is a different technology. Instead of transforming code that exists, language models write new code from a description. The two jobs meet at the compiler's front door: a model writes the source, and the compiler optimizes it.

AlphaCode and AlphaCode 2

DeepMind's AlphaCode, described in 2022, was trained on competitive programming problems. In simulated Codeforces contests with more than 5,000 participants, it ranked in the top 54.3% on average. The authors credited three components: a large, clean dataset, efficient transformer architectures, and large-scale sampling filtered by program behavior down to a few submissions (arXiv:2203.07814).

AlphaCode 2, announced in December 2023, was built on Gemini Pro. On 77 problems across 12 Codeforces contests, it solved 43% of the problems, compared with 25% for the original system on a different problem set, and Google estimated it performed at about the 85th percentile of competitors. In two of the twelve contests it beat 99.5% of participants (The Register, December 7, 2023).

The details around that headline matter. AlphaCode 2 could submit up to ten solutions per problem, while human entrants get one attempt. It generated roughly a million candidate programs per problem, filtered out about 95%, clustered the remaining programs, and used a separate model to score the best clusters. Google itself said the system remained too costly to operate at scale. It is an impressive demonstration of search plus generation, and not a tool available to developers.

Assistants in daily use

Everyday coding assistants work differently. They complete code in an editor, with one suggestion at a time. Two studies give a useful, if limited, picture.

In a controlled experiment, developers asked to implement an HTTP server in JavaScript as quickly as possible finished 55.8% faster with GitHub Copilot than a control group without it (Peng et al., arXiv:2302.06590). The task was a single, well-specified one, which is the setting where assistants are strongest, so the figure should not be read as a general productivity multiplier.

Meta's CodeCompose, an assistant trained on Meta's internal code, had been used by 16,000 developers at the time of its 2023 report. It supported nine programming languages, and the developers' code was 8% directly from CodeCompose (arXiv:2305.12050).

A user study from Stanford found a different effect on security. Participants with access to an AI assistant built on OpenAI's codex-davinci-002 wrote significantly less secure code than those without, and were more likely to believe their code was secure. Participants who trusted the assistant less, and who engaged more with their prompts, produced fewer vulnerabilities (Perry et al., arXiv:2211.03622). The lesson is about review habits, not about one product. For the practical side of working with these tools, our overviews of vibe coding and Claude Code are good next stops, and the piece on why AI hallucinates facts explains why generated code can look right and be wrong.

Why generated code still needs a compiler

A language model produces text that resembles code. A compiler checks that the text is code, and then makes it fast. The two share a boundary: the compiler can reject what the model writes, and the model cannot override the compiler's legality checks. Researchers are working on tools that let models propose transformations which a verifier then accepts or rejects, but that remains a research direction, not a settled workflow.

What can go wrong

Risk	What it means in practice
Compile-time cost	Running a model during compilation adds time. MLGO mitigates this by compiling the model ahead of time into native code
Distribution shift	A policy trained on one company's code may do worse on code that looks different. Google reports that its inlining policy transferred to other software and stayed good for three months, which is encouraging but not a guarantee
Drift	Compilers and codebases change, so policies need monitoring and, eventually, retraining
Hard-to-debug behavior	A learned decision has no one-line rule behind it, which makes regressions harder to explain
Reproducibility	Builds should produce the same output each time. Training pipelines and model versions become part of the build
Over-claiming	A gain measured against one baseline on one benchmark set is not a universal speedup

None of these risks is unique to compilers. They are the ordinary costs of putting learned components into infrastructure, and the reason adoption has focused on narrow, well-instrumented passes rather than whole-pipeline rewrites.

Common confusions

"AI compilers generate faster code from nothing." The gains reported for learned inlining and register allocation are single-digit percentages, and the inlining policy was trained for size. Larger multipliers appear in tensor compilers, but those compare against specific baselines on specific hardware.

"Compiler optimization and LLM code generation are the same thing." They are complementary. One transforms correct code into better machine code and preserves meaning by construction. The other writes new code that may or may not be correct.

"Triton is NVIDIA's compiler." Triton began in academia at Harvard and is developed by OpenAI and a wider community. It supports NVIDIA and AMD GPUs through PyTorch's backend.

"Benchmarks like Codeforces measure everyday programming." They measure puzzle-solving under contest rules, and AlphaCode 2 played with a different submission allowance than humans. That is useful evidence about reasoning, and weak evidence about maintaining a large codebase. Our guide to AI benchmarks explains the pitfalls of such comparisons.

Frequently asked questions

Q: Why doesn't every compiler use machine-learned optimization? A: Adoption has costs: training infrastructure, data collection, extra compile time, and the need to monitor for drift. MLGO's design shows a pragmatic path, with narrow policies in single passes and the model compiled into native code. LLVM includes the inlining and register-allocation advisors, but the training orchestration is outside LLVM, so each team has to supply its own.

Q: Can a learned optimizer change what my program does? A: In the MLGO design, no. The model replaces a heuristic that chooses among transformations the compiler already treats as legal, so a bad choice costs size or speed, not correctness. Systems that generate code directly, such as language models, do not have this guarantee and need testing or verification.

Q: What is a learned cost model? A: It is a predictor, often a neural network, that estimates how fast a given tensor schedule will run. AutoTVM, Ansor, and the Halide autoscheduler use such models to rank candidates so that only the most promising ones are measured on real hardware. The model is trained on measured timings from earlier runs.

Q: How good is AlphaCode 2 compared with human programmers? A: Google estimated it performed at about the 85th percentile on 12 Codeforces contests, and above 99.5% of participants in two of them. It could submit up to ten solutions per problem, whereas humans get one, and it relied on generating about a million candidates per problem. Google said it was too costly to operate at scale.

Q: Does GitHub Copilot make developers faster? A: In one controlled experiment, developers building an HTTP server in JavaScript finished 55.8% faster with Copilot. Another study found that assistant users wrote less secure code on some tasks and felt more confident about it. Both are limited to their tasks, so neither number is a general rule.

Q: What is the relationship between Triton and PyTorch? A: PyTorch 2.0's torch.compile uses a backend called TorchInductor, which relies on OpenAI Triton to generate GPU code for NVIDIA and AMD hardware. The PyTorch team reported that Triton-generated kernels achieve performance on par with hand-written kernels and libraries such as cuBLAS.

Q: Is AutoFDO machine learning? A: No. It collects sampled hardware performance profiles from production machines and uses them to guide optimization. Google reported about a 10.5% geometric-mean speedup on its benchmarks. It is often mentioned alongside learned optimization because both feed real-world data to the compiler.

The takeaway

The practical story of AI in compilers is less dramatic than the headlines and more durable. Machine-learned policies handle decisions that were once governed by guesses, within guardrails that keep the output correct, and they deliver percentages rather than multiples. Tensor compilers show larger gains, because they search a vast space where a cost model saves most of the measuring. Language models write source code, and the compiler still decides whether it is valid and how fast it runs.

The pattern that holds across MLGO, AlphaDev, and the tensor compilers is the same: the biggest wins come where the problem is narrow, the reward is measurable, and a conventional system stays in charge of correctness. Anything that gives up one of those three conditions needs far more scrutiny than a press release invites.

Sources

• Google Research, MLGO: a machine learning framework for compiler optimization (2022)
• Trofin et al., MLGO: a Machine Learning Guided Compiler Optimizations Framework, arXiv:2101.04808
• LLVM documentation, MLGO
• PyTorch 2.0 release notes
• DeepMind, AlphaDev discovers faster sorting algorithms (2023)
• Meta AI, Meta Large Language Model Compiler
• Google Research, AutoFDO (CGO 2016)

---

Read next:

• Quantum Computing and Machine Learning: Real Applications Emerging Now
• Neural Networks in Network Routing: How AI Optimizes 5G and Beyond
• How AI Designs Chips: Hardware-Aware Neural Architecture Search