Transfer Learning: Why AI Doesn't Start From Scratch

Transfer learning is the reason a model trained once on a huge pile of text or images can be adapted to a new task with a few thousand examples instead of millions. Before 2018, most language systems reused only pretrained word vectors and trained everything else from scratch for each task. After BERT and its contemporaries, reusing an entire pretrained network became the default. This article explains how that works, what the main adaptation methods trade off, and where reuse quietly fails.

What transfer learning actually is

Transfer learning means taking a model trained on one task and reusing some or all of it on a different task. The first task is usually broad and data-rich, and the second is usually narrow and data-poor. The model carries over what it learned, so the second task starts from a trained network, not from random numbers.

Human training offers a close analogy. A doctor spends years in general medical education before specializing, and a residency in dermatology builds on that base rather than starting from biology textbooks. The general training is expensive and happens once, and the specialty training is shorter because the foundation is already in place. In machine learning the general phase is called pre-training and the specialty phase is called fine-tuning.

A hypothetical example makes the stakes concrete. Suppose you want to classify chest X-rays as showing pneumonia or not, and you have only 5,000 labeled images. Training a large network from random initialization on that little data risks memorizing the examples (see overfitting and underfitting). Starting from a network already trained on millions of ordinary photographs gives you a head start on detecting edges, textures, and shapes. As we will see later, how large that head start is for medical images is less clear than the textbook story suggests.

The underlying assumption is that features useful in one domain are useful in another. That assumption holds often enough to have reshaped the field, and it fails often enough that knowing when it fails is half the skill.

Why deep networks transfer well

A neural network learns in layers (for a primer, see how neural networks work). In image models, early layers respond to simple patterns such as edges and color gradients. Middle layers combine those into textures and parts of objects. The last layers are tied to the specific labels the network was trained to predict.

Think of learning to read. The alphabet and the sounds letters make are reusable across every book you will ever open. The plot of one particular novel is not. Early layers in a network are the alphabet, and late layers are the plot.

The most cited test of this idea came from Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson in 2014. They measured how well features from each layer of an image network carried over to a different task. Early layers learned broadly useful, edge-like and color filters that transferred well, and later layers became tied to the original task. They also found a second, unexpected problem, co-adaptation, where neurons in adjacent layers come to depend on each other in ways that break when you split the network apart. Even features from a distant task still beat random initialization, and the benefit of transferred features persisted after fine-tuning (Yosinski et al., NeurIPS 2014).

Layer depth	What it tends to represent	How well it transfers
Early	Edges, color blobs, simple textures	Very well, across almost any image task
Middle	Shapes, object parts, patterns of texture	Well, if the new task is similar
Late	Whole objects tied to the original labels	Poorly; usually replaced for the new task

The dataset that made this practical was ImageNet. Its image-classification challenge uses about 1.28 million training images across 1,000 classes, and in 2012 a neural network called AlexNet won with a top-5 error of 15.3 percent, more than 10.8 percentage points better than the runner-up (Wikipedia, ImageNet and AlexNet). After that, pretrained ImageNet networks became the standard starting point for object detection, segmentation, and many niche vision problems. Sebastian Ruder's 2018 essay on the topic notes that fine-tuning ImageNet-trained weights worked with as little as one positive example per category (The Gradient).

The 2018 turning point in language

Language models had a version of transfer learning for years, but a shallow one. Pretrained word embeddings such as word2vec and GloVe supplied the first layer of a network, and everything above it was trained from scratch for each task. Ruder compared that to a vision model that only knows edges (The Gradient). It helped, but it did not carry grammar, world knowledge, or reasoning patterns from task to task.

In 2018 several groups pretrained entire networks and reused them. Jeremy Howard and Sebastian Ruder's ULMFiT, posted in January, reduced error by 18 to 24 percent on the majority of six text-classification datasets, and with only 100 labeled examples it matched the performance of training from scratch on 100 times more data (arXiv:1801.06146). ELMo, from Matthew Peters and colleagues in February, produced word representations that change with the surrounding sentence and improved six language tasks when added to existing models (arXiv:1802.05365). OpenAI released a transformer-based model the same year.

BERT, from Jacob Devlin and colleagues at Google in October 2018, made the approach hard to ignore. It pushed the GLUE benchmark score to 80.5 percent, a 7.7-point absolute improvement, and reached an F1 score of 93.2 on the SQuAD v1.1 question-answering test (arXiv:1810.04805). BERT is built from transformer encoders (our explainer on transformers covers the architecture), comes in two sizes of 110 million and 340 million parameters, and was pre-trained on BooksCorpus (800 million words) and English Wikipedia (2,500 million words). Its authors note that fine-tuning is comparatively cheap: all of the paper's results can be reproduced in at most one hour on a single Cloud TPU, or a few hours on a GPU, starting from the same pretrained model.

Ruder titled his essay "NLP's ImageNet moment has arrived," and the label stuck. The point was that language had finally found what vision had in 2012: a pretraining task, language modeling, that needs no human labels, so the training data is effectively unlimited.

Pre-training: learning without labels

Pre-training works because the data supplies its own answers. This is called self-supervised learning, and the common versions look like this:

• Next-token prediction. Read a sequence of words and predict the next one. Every sentence on the internet is a free training example. GPT-style models use this objective.
• Masked language modeling. Hide some words and predict them from the words on both sides. BERT masks 15 percent of the tokens in each sequence.
• Contrastive learning on image and text pairs. Train the model so that an image and its caption land near each other in a shared space, and mismatched pairs land far apart. CLIP does this.

CLIP, from Alec Radford and colleagues at OpenAI, was trained on 400 million image and text pairs collected from the internet. Without ever seeing the ImageNet training set's 1.28 million labeled examples, it matched the accuracy of the original ResNet-50 on ImageNet, and it was tested on more than 30 other vision datasets (arXiv:2103.00020). That is transfer without any fine-tuning at all.

Model	Pre-training objective	Pre-training data	Size
BERT (2018)	Masked language modeling plus next-sentence prediction	BooksCorpus and English Wikipedia, about 3.3 billion words	110 million and 340 million parameters
GPT-3 (2020)	Next-token prediction	300 billion tokens, 60% filtered Common Crawl	175 billion parameters
CLIP (2021)	Matching images to captions	400 million image and text pairs	Not covered here

Sources: Devlin et al. (2018); Brown et al. (2020), Tables 2.1 and 2.2; Radford et al. (2021). The word total for BERT is the sum of the two corpora.

GPT-3, from OpenAI's Tom Brown and colleagues, is the clearest case of scale. The paper trained eight model sizes ranging from 125 million to 175 billion parameters, all on a total of 300 billion tokens. The mix was 60 percent filtered Common Crawl web text, 22 percent WebText2, 8 percent each for two books corpora, and 3 percent Wikipedia (Brown et al., 2020). Training at that scale needs enormous computing power, a topic we cover in why AI needs so much computing power, and the full process is described in how AI models are trained. What matters here is the asymmetry: pre-training is done once by an organization with the budget, and everyone else inherits the result.

Fine-tuning: adapting to your task

Fine-tuning continues training the pretrained model on task-specific data, usually with a small learning rate so the existing knowledge gets nudged rather than overwritten. There is a spectrum of how much of the model you change.

Full fine-tuning updates every weight. It works well for large shifts between the pretraining and target tasks, and it needs enough memory to store gradients for the whole model. Linear probing freezes everything and trains only a new output layer on top. Partial fine-tuning unfreezes just the last few layers.

The more recent family is parameter-efficient fine-tuning, which adds a small number of trainable parameters and leaves the original weights frozen. An analogy: instead of reprinting a 1,000-page textbook for each course, you keep the book as it is and add sticky notes in the margins. Several students can share one book and each carry their own notes.

• Adapters insert small trainable modules inside the network. Houlsby and colleagues showed that adapters on BERT came within 0.4 percent of the performance of full fine-tuning on GLUE while adding only 3.6 percent more parameters per task (arXiv:1902.00751).
• LoRA (Low-Rank Adaptation) represents the change to each weight matrix as the product of two much smaller matrices and trains only those. The original paper reports that, compared with fine-tuning GPT-3 175B using the Adam optimizer, LoRA reduces the number of trainable parameters by 10,000 times and the GPU memory requirement by three times, with no additional inference latency (Hu et al., 2021).
• QLoRA combines LoRA with a compressed, 4-bit version of the frozen base model (the idea is explained in our piece on LLM quantization). Its authors report fine-tuning a 65-billion-parameter model on a single 48 GB GPU, and their Guanaco model family reached 99.3 percent of ChatGPT's performance on the Vicuna benchmark after 24 hours of fine-tuning on one GPU (Dettmers et al., 2023).

Method	What is trained	Reported result	Source
Full fine-tuning	All weights	Standard baseline	Common practice
Linear probing	New output layer only	Can beat full fine-tuning on out-of-distribution data	Kumar et al., 2022
Adapters	Small inserted modules	Within 0.4% of full fine-tuning on GLUE, 3.6% more parameters per task	Houlsby et al., 2019
LoRA	Low-rank update matrices	10,000 times fewer trainable parameters than full fine-tuning of GPT-3 175B	Hu et al., 2021
QLoRA	LoRA on a 4-bit frozen base	65B model fine-tuned on one 48 GB GPU	Dettmers et al., 2023
Prompting	Nothing (weights frozen)	Sometimes competitive with fine-tuned systems	Brown et al., 2020

The hidden cost of full fine-tuning shows up in a 2022 study by Ananya Kumar and colleagues. Across 10 distribution-shift datasets, full fine-tuning gained about 2 percent in accuracy on data like the training data but lost about 7 percent on out-of-distribution data compared with linear probing. Their explanation is that updating the whole network can distort the pretrained features. A two-step recipe, linear probing first and then full fine-tuning (LP-FT), beat plain fine-tuning by about 1 percent in-distribution and about 10 percent out-of-distribution (Kumar et al., ICLR 2022). The result is a reminder that pretrained features are valuable and can be damaged.

Few-shot, zero-shot, and instruction tuning

The most striking consequence of large-scale pre-training is that you sometimes do not need to update the weights at all. In few-shot learning, you put a handful of worked examples in the prompt and the model continues the pattern. Zero-shot learning goes further: you describe the task and give no examples.

The GPT-3 paper made this concrete. It applied the model to every task "without any gradient updates or fine-tuning," with tasks and demonstrations specified purely through text, and found that few-shot performance improved with model size and was sometimes competitive with earlier fine-tuned systems. The same paper is honest about limits, noting datasets where few-shot learning still struggles and methodological problems with training on large web corpora (Brown et al., 2020). That second point is not trivial: the authors report that a bug in their filtering let some benchmark overlap slip into the training data, and that retraining was not feasible. The problem of tests leaking into training data is explained in why data leakage breaks AI evaluations.

Few-shot prompting is, in effect, a way to program a model at inference time, and the craft of writing those prompts is covered in prompt engineering explained. A related approach is instruction tuning, in which a model is fine-tuned on a large collection of tasks phrased as instructions so that it handles unseen instructions better. Google's FLAN, a 137-billion-parameter model tuned on more than 60 language datasets, surpassed zero-shot 175B GPT-3 on 20 of the 25 tasks the authors evaluated (Wei et al., 2021).

Prompting and fine-tuning solve different problems. Fine-tuning changes the model's weights and is better for consistent, specialized behavior, but it needs data, compute, and maintenance. Prompting costs nothing to set up and is more flexible, but it is less reliable when you need exact behavior. For tasks that mainly need fresh or private facts, a third option, retrieval, often beats both, as described in what RAG is.

Foundation models: one base, many uses

In 2021 a group at Stanford's Center for Research on Foundation Models, led by Rishi Bommasani, named the pattern. A foundation model is one "trained on broad data at scale" and "adaptable to a wide range of downstream tasks" (arXiv:2108.07258). GPT-4, Claude, Gemini, and Llama are examples for language, and CLIP and Stable Diffusion for images.

This creates a particular economy. A small number of organizations can afford the data and computing power to train a foundation model from scratch. A much larger number adapt it, through fine-tuning, prompting, or retrieval, and the rest of the world uses the resulting applications without training anything. For a startup, the question shifted from "how do we train our own model?" to "how do we best use an existing one?"

Scale is part of why the arrangement works. The GPT-3 authors found that scaling up "greatly improves task-agnostic, few-shot performance," which is why they trained eight sizes to see the trend. The general relationship between model size, data, and accuracy is the subject of neural scaling laws. Related techniques make the base cheaper to run, such as model distillation, which trains a small model to imitate a large one. Our article on how large language models work covers what these systems are made of.

The report also named the risk. Because so many applications are adapted from the same few models, the authors warn that "the defects of the foundation model are inherited by all the adapted models downstream." They call this homogenization. A bias, a security flaw, or a systematic error in a popular base model can propagate to thousands of products at once, and the authors note that scale produces new "emergent" capabilities that make it harder to predict how these models fail.

Domain adaptation: when the target looks different

Transfer works best when the new task resembles the old one. When it does not, performance drops. This is the distribution-shift problem: the data the model sees in use differs from the data it learned from. A language model trained on general web text sees legal contracts, clinical notes, and source code as unfamiliar dialects with specialized vocabularies.

One fix is to continue pre-training on text from the target domain before fine-tuning. Suchin Gururangan and colleagues called this domain-adaptive pre-training and showed that a second phase of in-domain pre-training leads to gains on tasks in that domain (Gururangan et al., ACL 2020). BioBERT, a BERT model pre-trained further on biomedical literature, improved F1 by about 0.62 points on named-entity recognition and 2.80 points on relation extraction over earlier systems, and by 12.24 percent in mean reciprocal rank on biomedical question answering (Lee et al., 2019).

A different route is instruction prompt tuning. Google's Med-PaLM was not built by continuing to pre-train on medical text. Its authors started from Flan-PaLM, which reached 67.6 percent on the MedQA test of U.S. medical licensing exam questions, surpassing the previous best by over 17 percent, and adapted it with instruction prompt tuning, a parameter-efficient method that uses a small number of examples. They report that the resulting model "performs encouragingly, but remains inferior to clinicians" (Singhal et al., 2022). Those are benchmark scores, not clinical performance. For a broader look at the field, see AI in healthcare.

When transfer learning does not help

The textbook story is that pre-training always helps. The evidence is more mixed than that, and the best-known counterexamples come from computer vision.

In 2018, Kaiming He, Ross Girshick, and Piotr Dollár trained object-detection models on the COCO dataset from random initialization and found they matched the ImageNet-pretrained versions, provided the randomly initialized models were trained for more iterations. The result held even with only 10 percent of the training data. Their conclusion was that pre-training mainly speeds up early convergence and does not reliably improve final accuracy or add regularization, and they urged researchers to reconsider pre-training as the default (arXiv:1811.08883). The caveat is that this was a large detection dataset, and the benefit of pre-training is largest when labeled data is scarce.

Medical imaging, the textbook case for transfer, is itself contested. In a 2019 study of two large medical-imaging tasks, Maithra Raghu and colleagues found that "transfer offers little benefit to performance," and that "simple, lightweight models can perform comparably to ImageNet architectures." Some of the apparent benefits came from over-parameterized models, and the paper separates out the places where genuine feature reuse does occur (arXiv:1902.07208). It does not show that transfer never helps medical models, only that the size of the gain depends on the task and the model.

Situation	Transfer tends to help	Transfer tends to disappoint
Labeled data	Scarce, so the base model supplies most of the knowledge	Plentiful and training time is not a constraint
Source and target tasks	Closely related	Very different, so source features can mislead
Adaptation method	Parameter-efficient, with the base left mostly intact	Aggressive full fine-tuning on a narrow dataset
Test conditions	Similar to training data	Out-of-distribution, where fine-tuning can distort features
Evaluation	Clean, held-out data	Benchmarks that overlap with pre-training data

When the source and target tasks are too different, the result is sometimes called negative transfer: reusing the pretrained model leaves you worse off than starting fresh. Yosinski's experiments pointed in the same direction, since transferability declined as the tasks grew further apart. The practical lesson is to treat transfer as a hypothesis to test, with a from-scratch or simple baseline in the comparison.

Catastrophic forgetting

A second failure happens after adaptation, not before. When you fine-tune a network on a new task, the weight changes that help the new task can overwrite the patterns that supported the old one. The model gets better at B and forgets A. This is catastrophic forgetting, and it limits continual learning, the goal of a system that keeps picking up new skills without losing old ones.

The cause is that neural networks store knowledge as distributed patterns across many weights rather than in labeled slots. Nothing marks which weights matter for task A, so training on task B has no reason to leave them alone.

Researchers have tried several defenses. In 2017, James Kirkpatrick and colleagues proposed elastic weight consolidation, which "remembers old tasks by selectively slowing down learning on the weights important for those tasks," and demonstrated it on MNIST digit classification and on learning a series of Atari 2600 games in sequence (arXiv:1612.00796). Other approaches add new network columns for each task or replay stored examples from earlier tasks while training on new ones. None of these is a complete solution, which is one reason today's deployed models are mostly updated by periodic retraining rather than by learning continuously from experience.

A practical path: how teams choose an adaptation method

Once you treat transfer as a hypothesis to test, adapting a pretrained model becomes a sequence of decisions, and the evidence above suggests a sensible order. Practitioners generally work from the cheapest, most reversible option toward the most invasive one.

1. Try prompting first. Give the base model a few worked examples. If the output meets the bar, nothing needs training, and nothing can be damaged.
2. Add retrieval if the gap is knowledge. When the task needs fresh or private facts, not a new skill, supplying documents at query time leaves the weights alone.
3. Fine-tune with a parameter-efficient method if the gap is behavior. For a consistent format, tone, or specialized task, LoRA or QLoRA trains a small add-on while the base stays intact, so one base model can serve many tasks.
4. Guard against distribution shift. If real-world inputs will differ from your training data, consider linear probing or the two-step LP-FT recipe, given the out-of-distribution results from Kumar and colleagues.
5. Keep evaluation clean. Hold out data the base model could not have seen. The GPT-3 authors' own account of benchmark overlap shows how easily this fails.
6. Run a baseline. Compare against a smaller model trained from scratch or a simple classical method. The studies by He and by Raghu show why the comparison matters.
7. Monitor after launch. Real data changes over time, and model drift can quietly erode a fine-tuned model's accuracy.

The reason the order matters is cost and reversibility. A prompt can be rewritten in minutes. A LoRA adapter is a small file that can be swapped in or out without touching the base model, which is the property that makes serving many customized versions of one model practical. Full fine-tuning gives the most freedom and the most ways to go wrong, from distorted features to forgetting, so it is usually reserved for cases where the lighter methods have been tried and found short.

What this means

Transfer learning changed what building AI means. Before it, a new application meant collecting a big labeled dataset and training a model, and every project paid the full price. Now the expensive step happens once, upstream, and downstream teams pay for adaptation, which can be as light as a prompt or a few megabytes of LoRA weights.

The second shift is less visible: capability and risk are both concentrated. The same pretrained base can make hundreds of products good at language, and the same flaws in it can ride along into all of them. The research on fine-tuning distorting features, on from-scratch training matching pre-training, and on test data leaking into training sets argues for humility about what we actually know. Transfer is a powerful default, and it is still a claim that has to be checked against your own data.

Frequently Asked Questions

Q: Is transfer learning only for deep learning?
A: No. The idea of reusing knowledge from one task on another predates deep learning, and pretrained word embeddings such as word2vec and GloVe were an early form. Deep networks made it far more powerful because their layered representations, from edges to objects or from letters to meaning, are naturally reusable. Reusing whole pretrained networks became standard in language processing in 2018.

Q: What is the difference between fine-tuning and prompt engineering?
A: Fine-tuning changes the model's weights through additional training on task data. Prompt engineering leaves the weights untouched and steers the model with carefully written inputs, including examples. Fine-tuning gives more consistent specialized behavior but needs data and computing resources, while prompting needs neither and is less reliable for exact outputs.

Q: Why can't I just train my own model from scratch?
A: You can, and for small tasks with plentiful data it can work. But for tasks needing broad knowledge, a pretrained model has effectively learned from billions of examples, such as the 300 billion tokens GPT-3 saw, which no individual team can match. Studies also show from-scratch training can match pretrained models on some vision tasks if you train longer, so it depends on the task and your data.

Q: What is LoRA and why is it popular?
A: LoRA, short for Low-Rank Adaptation, freezes a pretrained model and trains small low-rank matrices that capture the change needed for a new task. The authors report 10,000 times fewer trainable parameters and three times less GPU memory than full fine-tuning of GPT-3 175B, with no extra inference delay. That makes fine-tuning large models feasible on modest hardware, especially in its QLoRA form.

Q: Are foundation models risky because so many systems depend on them?
A: Researchers at Stanford argue yes. Because applications are adapted from a small number of base models, the defects of a foundation model are inherited by every model adapted from it. They call this homogenization. The risk covers biases, errors, and security flaws, and it is one reason diversity of models and independent evaluation matter.

Q: Does transfer learning always beat training from scratch?
A: No. Studies on object detection found that from-scratch models can match ImageNet-pretrained ones given longer training, and a study of medical imaging found that transfer offered little benefit on two large tasks. It helps most when labeled data is scarce and the tasks are related. Testing against a from-scratch baseline is the way to find out.

Q: How does CLIP connect images and text?
A: CLIP trains an image encoder and a text encoder together on 400 million image and caption pairs, so a picture and its caption end up close in a shared space. This lets it classify images by comparing them to text descriptions of the categories, with no task-specific training, and it matched the original ResNet-50's ImageNet accuracy this way.

Sources

• Yosinski et al. (2014), How transferable are features in deep neural networks?, NeurIPS
• Devlin et al. (2018), BERT; Howard and Ruder (2018), ULMFiT; Peters et al. (2018), ELMo
• Brown et al. (2020), Language Models are Few-Shot Learners (GPT-3); Radford et al. (2021), CLIP
• Houlsby et al. (2019), Adapters; Hu et al. (2021), LoRA; Dettmers et al. (2023), QLoRA; Kumar et al. (2022), LP-FT
• Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models
• He, Girshick, and Dollár (2018), Rethinking ImageNet Pre-training; Raghu et al. (2019), Transfusion
• Gururangan et al. (2020), Don't Stop Pretraining; Lee et al. (2019), BioBERT; Singhal et al. (2022), Large Language Models Encode Clinical Knowledge; Wei et al. (2021), FLAN
• Kirkpatrick et al. (2017), Overcoming catastrophic forgetting; Ruder (2018), NLP's ImageNet moment has arrived

---

Read next:

• Reinforcement Learning Explained: How AI Learns to Play Games
• What Is Machine Learning? Explained Simply, with Real Examples
• Edge AI: How AI Runs on Your Phone