How AI Designs Chips: Hardware-Aware Neural Architecture Search

AI chip design is a loop with a strange shape: software that learned on one generation of processors now helps lay out and tune the next. Google trained a reinforcement learning agent to place the large blocks on its TPU accelerators. NVIDIA used another to shrink arithmetic circuits inside its GPUs. Researchers at Google, MIT, and elsewhere built search algorithms that design neural networks around a specific phone's speed limits. This article explains how each method works, what the evidence shows, and where the claims are still contested.

Key takeaways

• Neural architecture search (NAS) automates the choice of a neural network's structure, and hardware-aware versions put measured latency or memory use into the objective alongside accuracy.
• Google's reinforcement learning floorplanner, described in Nature in 2021, reports layouts "superior or comparable" to human ones in under six hours. Independent critics dispute the benchmarking, so treat it as a real production tool with a contested evidence base.
• NVIDIA reported that a reinforcement learning agent produced a 64-bit adder with 25% less area than an electronic design automation (EDA) tool's version at the same delay, and that its Hopper GPU architecture contains nearly 13,000 AI-designed circuits.
• The next frontier is co-design, where the chip and the network are optimized together instead of one after the other.

Why designing a chip is a search problem

A modern system-on-chip (SoC, a single chip that holds processors, memory controllers, and other blocks) forces thousands of linked decisions. Where does each functional block sit on the die? How do wires route between them? Which operations get dedicated fixed-function hardware, and which run on programmable cores? Each answer changes power, clock speed, heat, and manufacturing yield, and the answers constrain each other.

Think of city planning rather than building design. Putting a hospital on the east side changes traffic on every road, which changes where the fire station should go, which changes where homes are worth building. A planner cannot optimize one neighborhood at a time, and a chip architect cannot optimize one block at a time either. The space of possible layouts is far too large to try exhaustively, so the job is search under constraints.

Human experts handle this with intuition built over years, and that expertise is scarce. A July 2023 report from the Semiconductor Industry Association (SIA) and Oxford Economics projected that the U.S. semiconductor workforce would grow from about 345,000 to about 460,000 jobs by 2030, with roughly 67,000 of those positions at risk of going unfilled across technicians, computer scientists, and engineers (SIA press release). That figure covers the United States only, and it is a projection, not a count of chip designers.

Automation targets three layers of the problem. Design space exploration asks what the chip's organization should be. Physical design asks how components are arranged on silicon and wired together. Verification asks whether the finished design does what its specification says. Machine learning has entered all three, with very different levels of maturity.

The hardware being designed also matters for the software on top of it. How GPUs work and why AI needs them covers the parallel arithmetic that accelerators are built to deliver, and why AI needs so much computing power explains the demand that pushes teams toward faster design cycles.

How a reinforcement learning agent floorplans a chip

Floorplanning is the step where large blocks, called macros (memory arrays such as SRAM are the standard example), get positions on a rectangular canvas. The smaller standard cells that implement logic are placed afterward, usually by conventional algorithms. A good floorplan keeps connected blocks close so wires stay short, avoids congested routing channels, and meets timing, meaning signals arrive within each clock cycle.

In the paper "A graph placement methodology for fast chip design," published in Nature in June 2021, Azalia Mirhoseini, Anna Goldie, and colleagues at Google framed macro placement as a game (Nature 594, 207-212). The agent sees the chip's netlist, a graph that records which components connect to which. It places macros one at a time on a grid. At the end of an episode it receives a reward based on estimated wire length, congestion, and related quality measures. An edge-based graph convolutional network, a neural network that reads graphs, encodes the netlist so the agent can compare very different chips.

The paper's headline claim is that, in under six hours, the method automatically generates floorplans superior or comparable to those of human experts across the key metrics. The comparison point is a human team working with standard tools, a process that typically takes weeks or months.

What "pre-training" adds

A new game board is expensive to learn from scratch. Google's approach pre-trains the policy on many previously designed blocks, then fine-tunes on the target block. Google's September 26, 2024 DeepMind post, which gave the method the name AlphaChip, describes training first on diverse blocks from earlier generations such as network blocks, memory controllers, and data transport buffers (DeepMind, "How AlphaChip transformed computer chip design"). The same post says AlphaChip has been used in every TPU generation since 2020, with its charts covering the v5e, v5p, and Trillium (sixth generation) chips, and that MediaTek extended it to speed up development of its own chips.

An analogy helps here. A chess engine that has studied millions of positions does not start from zero in a new game. Pre-training is meant to give the floorplanner a similar head start, so that fine-tuning on a new chip block starts from learned layout habits and not from random placement.

Why the reward function is the hard part

Everything depends on what the reward measures. Wire length is only a proxy for the real goals, which are speed, power, and area after full physical design. If the proxy and the goals drift apart, an agent that maximizes the proxy can produce layouts that look excellent to the reward and mediocre to the final timing report. Much of the engineering in this field is about building proxies that stay faithful, and about checking results with the full downstream flow.

The dispute over Google's floorplanning results

The 2021 paper drew sharp criticism, and a fair account has to include it. Igor L. Markov published "The False Dawn: Reevaluating Google's Reinforcement Learning for Chip Macro Placement," which has gone through ten versions on arXiv, the latest dated September 28, 2024. He argues that the Nature paper omitted inputs and methodological steps needed to reproduce its results, and that independent evaluations found the Google method trailing human designers, simulated annealing (a classic randomized placement search), and commercial software. He also reports that in a 2023 open research contest, reinforcement learning methods did not place in the top five (arXiv:2306.09633).

Google's 2024 materials respond in part by clarifying method details. The DeepMind post notes that a Nature addendum accompanied it, along with a pre-trained checkpoint, and that the method was open sourced. The post is a company publication with supportive outside quotes, and it presents no limitations.

What can be said with confidence? Google says it used the method on production TPU designs, which is a claim about deployment and not about benchmark superiority. Markov's claims concern comparisons and reproducibility. Both can hold: a tool can be useful in one company's pipeline and still fail to beat strong open baselines on public benchmarks. Readers should hold the strongest versions of either side loosely until independent groups publish head-to-head comparisons on shared designs.

The practical lesson is not that AI floorplanning works or fails. It is that "better than humans" depends on which humans, which tools, which benchmark, and who ran the comparison.

Neural architecture search: letting software design the network

Floorplanning decides where blocks sit on silicon. Neural architecture search answers a different question: what should the neural network itself look like? The classical workflow is hand design. A researcher proposes a structure such as AlexNet or ResNet, trains it, measures accuracy, and iterates. NAS turns that loop into an optimization problem.

In the original formulation, Barret Zoph and Quoc Le (Google Brain) used a recurrent network as a controller that wrote out architecture descriptions, then trained it with reinforcement learning to maximize the validation accuracy of the networks it generated. Their November 2016 paper reported a CIFAR-10 image model with a 3.65 percent test error and a language-modeling cell with a Penn Treebank test perplexity of 62.4, 3.6 better than the previous state of the art (arXiv:1611.01578).

The catch was cost. Each candidate had to be trained to measure its quality. A follow-up search that produced NASNet-A required about 2,000 GPU-days, and AmoebaNet-A, found with evolutionary search, about 3,150 GPU-days, according to the comparison table in the DARTS paper (a GPU-day is one GPU running for 24 hours).

How DARTS made search cheaper

Differentiable Architecture Search (DARTS), by Hanxiao Liu, Karen Simonyan, and Yiming Yang, relaxed the problem. Instead of choosing one operation per connection from a discrete list, it assigns a continuous weight to every candidate operation and learns those weights by gradient descent, alongside the network weights. The paper was published at ICLR 2019 and reports search costs of 1.5 GPU-days (first-order) and 4 GPU-days (second-order) on CIFAR-10, with test errors of 3.00 and 2.76 percent (arXiv:1806.09055). The authors note those figures exclude about 1 GPU-day of architecture selection and 1.5 GPU-days of final evaluation.

An analogy: older NAS was like hiring a new chef for each candidate recipe and tasting every dish, while DARTS is like a chef adjusting the proportions of every ingredient at once and tasting a single blended dish as the proportions shift. The blend is an approximation of the real search, but a speedup of roughly 500 times against NASNet-A's 2,000 GPU-days (second-order DARTS at 4 GPU-days) changed what a lab could afford.

Method	Search type	Search cost (GPU-days)	CIFAR-10 test error	Source
NASNet-A + cutout	Reinforcement learning	2,000	2.65%	DARTS paper, Table 1
AmoebaNet-A	Evolution	3,150	3.34%	DARTS paper, Table 1
DARTS, first order + cutout	Gradient-based	1.5	3.00%	DARTS paper, Table 1
DARTS, second order + cutout	Gradient-based	4	2.76%	DARTS paper, Table 1

Costs are as reported by each paper's authors (via the DARTS table) and exclude final retraining. The lower search cost did not come with a uniformly better error, as the first row shows.

Hardware-aware NAS: optimizing for the device, not just accuracy

Accuracy alone is the wrong target for a phone, a camera, or a car. A network must also run within a latency budget and a memory limit. A common shortcut is to count floating-point operations (FLOPs), but FLOPs correlate imperfectly with real speed, because memory access patterns, parallelism, and the specific chip's strengths all change timing.

The MnasNet paper (Tan et al., Google, CVPR 2019) put measured latency directly into the search objective, running candidates on real Pixel phones. Its best model reached 75.2 percent ImageNet top-1 accuracy at 78 ms latency on a Pixel phone, 1.8 times faster than MobileNetV2 with 0.5 percent higher accuracy, and 2.3 times faster than NASNet with 1.2 percent higher accuracy (arXiv:1807.11626). The authors also introduced a factorized hierarchical search space so that different layers could use different operations, since the best choice early in a network differs from the best choice late in it.

MobileNetV3 (Howard et al., 2019) combined hardware-aware NAS with the NetAdapt algorithm, which trims individual layers to hit a latency target. Compared with MobileNetV2, the Large version was 3.2 percent more accurate on ImageNet with 15 percent lower latency, and the Small version was 4.6 percent more accurate with 5 percent lower latency, with tuning for mobile phone CPUs (arXiv:1905.02244).

EfficientNet (Tan and Le, 2019) used NAS to design a small baseline network, then scaled depth, width, and resolution together with a fixed ratio. Its largest model, EfficientNet-B7, reached 84.3 percent ImageNet top-1 accuracy while being 8.4 times smaller and 6.1 times faster at inference than the best existing convolutional network the authors compared against (arXiv:1905.11946).

How the search "knows" about the hardware

Three approaches show up in practice. Each trades accuracy of the estimate against speed of the search.

Approach	How latency is obtained	Strength	Weakness
On-device measurement	Run each candidate on the real phone or chip	Most accurate	Slow, needs hardware in the loop
Lookup table	Pre-measure each layer type and sum the results	Fast, hardware-specific	Misses interactions between layers
Learned predictor	Train a small model to predict latency	Fast, generalizes across candidates	Needs training data from real measurements

MnasNet used on-device measurement. Many later systems use tables or predictors because search runs thousands of candidates.

Once-for-All: train a family, specialize cheaply

A different strategy avoids searching again for every device. Once-for-All (OFA), from Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han (ICLR 2020), trains one large network designed so that smaller sub-networks can be extracted without retraining. The abstract counts more than 10^19 possible sub-networks. It reports up to 4.0 percent higher ImageNet top-1 accuracy than MobileNetV3, or the same accuracy at 1.5 times the speed, and 80.0 percent top-1 accuracy under 600 million multiply-accumulate operations (arXiv:1908.09791). The paper also motivates itself with the carbon cost of repeating searches for each device, comparing it to the lifetime emissions of five cars.

That cost argument connects to a broader trend toward small, efficient models. Techniques such as LLM quantization shrink a trained model's numbers to fit modest hardware, and edge AI on your phone shows where hardware-aware networks end up running.

AI inside the chip design toolchain

Floorplanning and NAS get the headlines, but AI also shows up in less glamorous parts of the electronic design automation (EDA) flow. Four examples have public documentation.

Tool or paper	What it does	Reported result	Source and date
Google AlphaChip	Places macros on a chip floorplan	Layouts in hours instead of weeks or months; used on TPU v5e, v5p, Trillium	DeepMind, September 26, 2024
NVIDIA PrefixRL	Designs arithmetic circuits such as adders	25% lower area for a 64-bit adder at the same delay; nearly 13,000 instances in Hopper	NVIDIA blog, July 8, 2022
Synopsys DSO.ai	Searches EDA tool settings to improve power, speed, and area	Company said 100 commercial tape-outs had used it	Synopsys release, February 7, 2023
NVIDIA ChipNeMo	Domain-adapted language models for engineers	Assistant chatbot, EDA script generation, bug summarization	arXiv, October 31, 2023

PrefixRL: reinforcement learning on arithmetic

Adders (circuits that add binary numbers) sit all over a GPU. Their design is a prefix graph, and a good graph balances delay against area. NVIDIA's PrefixRL agent adds or removes nodes in a prefix graph and receives the change in circuit area and delay as its reward. NVIDIA reported that the best PrefixRL 64-bit adder had 25 percent lower area than the adder from a state-of-the-art EDA tool at the same delay, and that the Hopper architecture contains nearly 13,000 AI-designed circuit instances (NVIDIA Technical Blog, July 8, 2022). The same post is candid about cost: training the 64-bit case took over 32,000 GPU hours, with physical simulation needing 256 CPUs per GPU, which is why NVIDIA built a distributed platform called Raptor to run it.

This example is useful because it shows both sides of the technology. A 25 percent area reduction in a widely replicated circuit is a meaningful gain. The compute bill to find it was large, and the design was tested at the adder level and not as a full chip.

DSO.ai: tuning the tuners

EDA tools expose thousands of settings that interact in hard-to-predict ways, and engineers spend weeks sweeping them. Synopsys DSO.ai wraps that process in a reinforcement learning loop. Synopsys said on February 7, 2023 that it had monitored the first 100 commercial tape-outs using DSO.ai (Synopsys announcement via EIN). A tape-out is the point where a design is finalized and sent to the factory. These are vendor claims and are not independently audited, so the best reading is that the tool is widely adopted, not that a specific percentage gain is proven.

ChipNeMo: language models as design assistants

Not every AI application in chip design writes layouts. NVIDIA's ChipNeMo work adapted large language models to chip-design text and tasks through a custom tokenizer, continued pre-training on in-domain data, domain-specific instruction tuning, and adapted retrieval. The authors evaluated three applications: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Their largest model beat GPT-4 on the first two and was competitive on the third (arXiv:2311.00176). The authors say there is still room for improvement.

What Apple's design loop shows, and what it hides

Apple is secretive about its internal chip design process, so any description of its methods beyond public statements is speculation. What is documented is the product. Apple introduced a dual-core Neural Engine in the A11 Bionic in September 2017, with a stated peak of 600 billion operations per second, and used it first for Face ID (CNBC, September 12, 2017). The 600 billion figure is Apple's own peak number, not an independent measurement.

The general principle behind such designs is workload-driven hardware. Silicon designers study the neural networks that software teams plan to run, then choose the number of multiply-accumulate units, the data types, and the memory bandwidth to match. Software teams in turn choose or search for network structures that map well onto the result. That loop is what the research literature calls co-design. How much of it uses NAS inside any given company is not public, and this article does not claim otherwise.

For a view of how the surrounding hardware is manufactured, see how semiconductors are made and EUV lithography, which explain the physical limits that designers are optimizing against.

Co-design and the hardware lottery

Co-design means searching over the chip and the network at the same time. The motivation comes from an uncomfortable observation by Sara Hooker, then at Google Brain, in her 2020 essay "The Hardware Lottery." She defines it as the case where "a research idea wins because it is suited to the available software and hardware" and not because it is superior to alternatives. The essay argues that specialized hardware makes straying from mainstream ideas more costly, so gains from computing will be uneven across research directions (arXiv:2009.06489).

Apply that to today. Transformers and convolutional networks run extremely well on current accelerators, so they receive the research attention, the tuning effort, and the software support. An alternative architecture could be better on hardware built for it but never gets the chance to show that. Joint search is one proposal for escaping the trap, because it treats the hardware as a variable and not as a fixed stage.

A useful comparison is chiplet design, where the building blocks of a processor are separate pieces assembled into a package. Why chiplets are changing processor design shows how a modular design space creates more choices, and more choices create exactly the kind of search problem these tools target.

Why co-design is hard

Three obstacles stand out. The first is search-space size. Chip parameters and network parameters multiply, and an exhaustive search is out of the question. The second is evaluation cost. Judging a hardware candidate means simulation or synthesis, which is slow compared with running a network on an existing device. The third is that hardware cannot be iterated like software. A mistake found after tape-out costs months and a large sum of money, so teams are reluctant to hand a decision to a model they cannot fully inspect.

Where the limits are

Three limits recur across the sources above.

Compute cost. PrefixRL needed more than 32,000 GPU hours for a single 64-bit adder case. Early NAS needed thousands of GPU-days. Efficiency tricks such as weight sharing and pre-training reduce the bill, but they also add assumptions that can bias the result.

Evaluation fidelity. Every automated search optimizes a proxy. If the proxy misses a failure mode, such as a thermal hotspot or a rare timing path, the output looks good until late in the flow. That is why verification remains a human-heavy stage, and why AI-powered software testing has a hardware parallel in test generation for chips.

Reproducibility. The floorplanning dispute illustrates the issue. When important inputs, training data, and settings are not public, outsiders cannot separate what the method does from what a specific company's data and tooling add. Google's open-sourcing of the method and its release of a pre-trained checkpoint in 2024 help here, and independent comparisons on shared benchmarks would help more.

There is also the matter of scope. Everything above is about a narrow slice of chip design, and it sits alongside decades of conventional automation. The tools are adopted because they sometimes beat a human-tuned baseline for a particular sub-problem, not because they have replaced the architect's high-level judgment.

Common misconceptions

"AI designs entire chips on its own." No public source supports this. The documented uses are macro placement, arithmetic circuit generation, setting-space search, and assistant tools. Architects still define the specification, constraints, and overall organization.

"NAS finds the best possible network." It finds the best network within a search space someone defined, under an objective someone chose. Change the space or the objective, and the winner changes. Hand-designed architectures also influence which operations are in the search space in the first place.

"The 2021 Google paper was refuted." It was challenged, and the challenges are specific: reproducibility, benchmark choice, and comparisons against strong baselines. Google continues to describe production use. The honest position is that the benchmark question remains open to independent testing.

"Hardware-aware means FLOPs-aware." FLOP counts are a rough guide. MnasNet's central point was that measured latency on a real device is a better objective than counting operations.

"AI chip design will compress schedules by a fixed percentage." Claims of precise schedule or efficiency gains for the whole industry circulate often, but they are not backed by the sources used here. The reported results are per tool and per sub-task, such as the 25 percent area reduction in one adder.

FAQ

Q: Does AI-driven floorplanning beat human engineers? A: Google reports that its method produces floorplans superior or comparable to human experts' in under six hours, and says it has used the method on TPU designs since 2020. Independent critics, notably Igor Markov, argue that comparisons against strong open baselines show weaker results. The answer depends on the benchmark, and public head-to-head tests on shared designs are still limited.

Q: What is the difference between NAS and hyperparameter tuning? A: Hyperparameter tuning adjusts training settings such as learning rate, batch size, and regularization for a fixed model structure. NAS changes the structure itself: the number and type of layers, how they connect, and how wide they are. NAS searches a much larger and more irregular space, which is why its early versions cost thousands of GPU-days.

Q: How does hardware-aware NAS know about the target hardware? A: It brings hardware cost into the search objective. MnasNet ran candidate models on real Pixel phones and used the measured latency. Other systems sum pre-measured layer times from a lookup table or train a predictor on measured samples. Direct measurement is the most accurate and the slowest.

Q: Why is NAS so much cheaper now than in 2017? A: Two ideas did most of the work. Weight sharing trains one large network containing many candidates, so candidates are not trained independently. Differentiable search, as in DARTS, turns discrete choices into continuous weights that gradient descent can tune. DARTS reported 1.5 to 4 GPU-days on CIFAR-10 against 2,000 for NASNet-A.

Q: Is Apple using NAS to design its Neural Engine? A: That is not publicly documented. Apple has described the Neural Engine's capabilities, such as the A11 Bionic's stated 600 billion operations per second, but not the internal methods used to choose its architecture. Any detailed account of Apple's process beyond what Apple has published is conjecture.

Q: Can I try neural architecture search myself? A: Yes, at small scale. Open-source projects such as the Once-for-All code and pre-trained models released by its authors, and libraries built on PyTorch or JAX, make it possible to experiment on modest hardware, particularly with weight-sharing methods. Full-scale searches on large datasets still require significant compute.

Q: What would count as real evidence that AI chip design works at scale? A: Independent, reproducible comparisons on shared, public designs, with clear baselines and full-flow results for power, performance, and area. Vendor tape-out counts show adoption. They do not show how large the gain is.

The takeaway

AI is already a working part of chip design, but its role is narrower and more argued over than the headlines suggest. The best-documented wins are specific: a floorplanner that Google uses and critics question, an arithmetic-circuit agent that cut one adder's area by a quarter at a high compute cost, and search tools that let a phone-sized network be built around a measured latency. The deeper shift is conceptual. Once the hardware is treated as a variable in the search and not a fixed constraint, the question "which network fits this chip" becomes "which chip and network fit each other," and that is the step researchers are now working toward.

Sources

• Mirhoseini et al., "A graph placement methodology for fast chip design," Nature 594, 207-212 (2021): https://www.nature.com/articles/s41586-021-03544-w
• DeepMind, "How AlphaChip transformed computer chip design," September 26, 2024: https://deepmind.google/discover/blog/how-alphachip-transformed-computer-chip-design/
• Markov, "The False Dawn: Reevaluating Google's Reinforcement Learning for Chip Macro Placement," arXiv:2306.09633: https://arxiv.org/abs/2306.09633
• Zoph and Le, "Neural Architecture Search with Reinforcement Learning," arXiv:1611.01578: https://arxiv.org/abs/1611.01578
• Liu, Simonyan, and Yang, "DARTS: Differentiable Architecture Search," ICLR 2019: https://arxiv.org/abs/1806.09055
• Tan et al., "MnasNet: Platform-Aware Neural Architecture Search for Mobile": https://arxiv.org/abs/1807.11626
• Howard et al., "Searching for MobileNetV3": https://arxiv.org/abs/1905.02244
• Tan and Le, "EfficientNet": https://arxiv.org/abs/1905.11946
• Cai et al., "Once-for-All": https://arxiv.org/abs/1908.09791
• NVIDIA, "Designing Arithmetic Circuits with Deep Reinforcement Learning," July 8, 2022: https://developer.nvidia.com/blog/designing-arithmetic-circuits-with-deep-reinforcement-learning/
• Liu et al., "ChipNeMo: Domain-Adapted LLMs for Chip Design": https://arxiv.org/abs/2311.00176
• Hooker, "The Hardware Lottery": https://arxiv.org/abs/2009.06489
• Semiconductor Industry Association, workforce report press release, July 2023: https://www.semiconductors.org/america-faces-significant-shortage-of-tech-workers-in-semiconductor-industry-and-throughout-u-s-economy/

---

Read next:

• Neural Networks in Network Routing: How AI Optimizes 5G and Beyond
• AI Compilers and Code Generation: How ML Is Reinventing Software Optimization
• AI-Powered Software Testing: How Machine Learning Finds Bugs Before You Do
• How the Transistor Changed Everything
• How Do Neural Networks Work? The Architecture Behind Modern AI