Fine-Tuning LLMs Explained — How and When to Do It

Fine-tuning LLMs means taking a language model that has already read a large slice of the internet and training it a little longer on a small, carefully chosen dataset, so that it answers in the format, tone, or skill you need. Done well, a few hundred examples can fix a stubborn output problem that no amount of prompt rewriting touched. Done badly, it burns a weekend of GPU time and leaves a model that is worse at everything outside your narrow test. This article explains what actually changes inside the model, how LoRA and QLoRA make it affordable, what the research says about data and forgetting, and how to decide between fine-tuning, retrieval, and plain prompting.

What fine-tuning changes, and what it does not

A pretrained model is like a medical school graduate. It has absorbed a vast amount of general knowledge, but it has never worked a shift in your hospital. Fine-tuning is the residency: a short, focused period in which the graduate learns your forms, your abbreviations, and the way your department wants a case presented. The graduate does not come out knowing a thousand new diseases. They come out knowing how to apply what they already knew in your setting.

That analogy matches the research. Gekhman and colleagues tested what happens when a model is fine-tuned on facts it did not already know. They found that such examples are learned much more slowly than examples consistent with existing knowledge, and that once the model does absorb them, its tendency to hallucinate rises in proportion. Their conclusion is that factual knowledge is acquired mostly during pretraining, and fine-tuning mainly teaches the model to use that knowledge more efficiently (Gekhman et al., 2024).

A separate comparison by Ovadia and colleagues pitted unsupervised fine-tuning against retrieval-augmented generation for injecting knowledge. Fine-tuning helped somewhat, but retrieval consistently outperformed it, both for facts the model had seen before and for entirely new ones (Ovadia et al., 2023). If your problem is "the model does not know our return policy," the evidence points to retrieval-augmented generation, not training.

In the LIMA study, a 65B-parameter LLaMA model fine-tuned on only 1,000 carefully curated prompt-and-response pairs, with no reinforcement learning, produced responses that human raters judged equivalent to or better than GPT-4's in 43% of comparisons (Zhou et al., 2023). The authors argue that almost all knowledge in a language model is learned during pretraining, so alignment needs only limited, high-quality data.

Where fine-tuning sits in the training pipeline

Building a chat assistant takes several stages, and fine-tuning is the name for most of the later ones. Pretraining on a huge text corpus produces a base model that predicts the next token. Supervised fine-tuning (SFT) on instruction-and-answer pairs turns it into an instruction-following model. Preference tuning, which includes RLHF and DPO, then nudges it toward the answers people rate higher. A fuller walk through the earlier stages is in how AI models are trained.

The payoff from the later stages can be large. In OpenAI's InstructGPT work, human evaluators preferred outputs from a 1.3B-parameter model trained with human feedback over outputs from the 175B-parameter GPT-3, a model roughly 100 times larger (Ouyang et al., 2022). Scale buys knowledge. Fine-tuning buys behavior.

When you fine-tune an open model yourself, you are almost always repeating the first of those later stages on a smaller scale: SFT, sometimes followed by a preference step. The rest of this article follows that path.

Full fine-tuning versus parameter-efficient methods

Full fine-tuning updates every weight in the network. It has the highest ceiling, but memory is the problem. Training needs the weights, the gradients, and the optimizer's running statistics for every parameter, which is why full fine-tuning of even a mid-sized model is often out of reach on a single consumer GPU.

Parameter-efficient fine-tuning (PEFT) freezes the original weights and trains a small number of new ones. The dominant PEFT method is LoRA, which stands for Low-Rank Adaptation. Hu and colleagues introduced it in 2021 and reported that, compared with fine-tuning GPT-3 175B with the Adam optimizer, LoRA could cut the number of trainable parameters by 10,000 times and the GPU memory requirement by three times, with quality on par with or better than full fine-tuning on the models they tested (Hu et al., 2021).

How LoRA works, with real numbers

A transformer layer contains large weight matrices. Imagine one that is 4,096 by 4,096, which holds 16,777,216 numbers. Full fine-tuning would adjust all of them. LoRA instead leaves that matrix untouched and learns a correction expressed as the product of two thin matrices: one that is 4,096 by 16 and another that is 16 by 4,096. The number 16 is the rank.

The thin pair holds 2 × 4,096 × 16 = 131,072 numbers, about 0.78% of the original matrix. During the forward pass, the model adds the small correction to the frozen layer's output. Think of it as a clip-on lens: the camera body stays the same, and you can swap lenses without rebuilding the camera.

Hugging Face's PEFT library exposes this through a handful of settings. The rank r controls how many parameters are trainable, target_modules chooses which layers get adapters, and lora_alpha sets a scaling factor; in the original formulation the adapter's output is multiplied by lora_alpha / r. Once training ends, merge_and_unload() folds the adapter into the base weights, so the finished model runs with no added inference latency (Hugging Face PEFT docs).

Does LoRA match full fine-tuning?

The honest answer is "sometimes, and it depends on how you set it up." Two studies point in different directions, and both are worth knowing.

Biderman and colleagues compared LoRA with full fine-tuning on programming and math tasks. In the standard low-rank settings they tested, LoRA substantially underperformed full fine-tuning. It did, however, better preserve the base model's performance on tasks outside the target domain, and it forgot less than common regularizers such as weight decay and dropout. They also found that full fine-tuning learns perturbations with a rank 10 to 100 times greater than typical LoRA configurations (Biderman et al., 2024).

In September 2025, John Schulman and colleagues at Thinking Machines Lab reported the other side. On small-to-medium supervised datasets, LoRA matched full fine-tuning's sample efficiency and final loss, provided two conditions held: the adapters were applied to all layers, especially the MLP layers rather than attention alone, and the adapter was not capacity-constrained. When a dataset was larger than the adapter could absorb, LoRA fell behind. They also found the best learning rate for LoRA was about 10 times the full fine-tuning rate, and that in reinforcement learning, even rank-1 LoRA matched full fine-tuning (Thinking Machines, 2025).

Put together, the guidance is practical. For a style change, a format, or a few thousand examples, LoRA on all layers is a sound default. For teaching a model a large new body of skills from a big dataset, the capacity limit becomes real, and a higher rank or full fine-tuning deserves a test.

QLoRA: fitting a large model on one GPU

LoRA shrinks what you train. It does not shrink what you must hold in memory, because the frozen base model still has to sit on the GPU. QLoRA, from Dettmers and colleagues in 2023, attacks that second problem. It quantizes the frozen base model to 4-bit precision and then trains LoRA adapters on top of it (Dettmers et al., 2023).

The paper describes three ingredients. A data type called 4-bit NormalFloat (NF4) is designed to be information-theoretically optimal for normally distributed weights. Double quantization quantizes the quantization constants themselves, trimming memory further. Paged optimizers absorb the memory spikes that otherwise crash a long run. With these, the authors fine-tuned a 65B-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning task performance. Their Guanaco models reached 99.3% of ChatGPT's performance level on the Vicuna benchmark, and they used QLoRA to fine-tune more than 1,000 models in total.

The memory arithmetic shows why this works. Counting only the weights, a 16-bit number takes two bytes, an 8-bit number one byte, and a 4-bit number half a byte.

Model size	16-bit weights	8-bit weights	4-bit weights
7B parameters	14 GB	7 GB	3.5 GB
13B parameters	26 GB	13 GB	6.5 GB
65B parameters	130 GB	65 GB	32.5 GB
70B parameters	140 GB	70 GB	35 GB

These figures are weights only, calculated from bytes per parameter. Real training also needs memory for activations, the adapter's own gradients and optimizer state, and the sequence length you train on, so the 65B model's 32.5 GB of 4-bit weights leaves the remaining part of the paper's 48 GB budget for those extras. For a deeper look at how the compression itself works, see LLM quantization explained.

SFT, DPO, and reinforcement: choosing the training objective

LoRA and QLoRA describe which weights you train. The objective describes what you train them to do. Three objectives cover most projects.

Supervised fine-tuning (SFT) is the simplest. Each example is an input and the output you want. The loss is the token-level cross-entropy: the model is penalized by how little probability it gave to each correct next token. Hugging Face's TRL library, which implements SFT, notes that this is the simplest and most commonly used way to adapt a model to a target dataset, and it can compute the loss only on the assistant's reply so the model is not rewarded for echoing the prompt (TRL SFT docs).

Direct Preference Optimization (DPO) works from pairs: one preferred answer and one rejected answer for the same prompt. Rafailov and colleagues showed that the usual RLHF objective can be reparameterized so that the optimal policy is found with a simple classification loss, with no separate reward model and no sampling from the model during training. They report DPO matched or improved response quality on summarization and single-turn dialogue while being stable and computationally lightweight (Rafailov et al., 2023). A side-by-side of the preference methods is in RLHF vs RLAIF vs DPO, and the original reinforcement-learning pipeline is covered in how AI models learn from feedback.

Reinforcement fine-tuning grades the model's own attempts and reinforces the reasoning behind the high-scoring ones. OpenAI's documentation describes it as applicable to reasoning models and as requiring expert graders who agree on the ideal output (OpenAI model optimization guide). It is a specialist tool, not a first step.

A typical sequence for an application team is SFT first. DPO comes later, and only if SFT leaves a consistent preference problem, such as the model being too wordy, that you can express as chosen-versus-rejected pairs.

Fine-tune, retrieve, or just prompt?

Most wasted fine-tuning projects began with the wrong question. The useful question is what kind of gap you are closing.

Problem	Best first tool	Why
The model lacks current or private facts	Retrieval (RAG)	Facts change; weights do not update by themselves
The answer format keeps drifting	Prompt with examples, then fine-tune	Format is behavior, which training teaches well
Brand voice or persona is inconsistent	Fine-tune	Style is hard to describe and easy to demonstrate
A narrow classification at high volume	Fine-tune a small model	A shorter prompt cuts cost and latency
A one-off or changing task	Prompt engineering	No training run to maintain
Very long reference documents	Long context or retrieval	The documents change

OpenAI's own guidance supports this ordering. It lists use cases for supervised fine-tuning such as classification, nuanced translation, content in a specific format, and correcting instruction-following failures, and for direct preference optimization such as getting the right focus in summaries or the right tone in chat messages. The recommended sequence is to write evals first, refine prompts, and fine-tune only if needed (OpenAI model optimization guide).

If you have not squeezed a prompt yet, start with prompt engineering. Many "the model cannot do this" problems turn out to be a vague instruction. And if your real constraint is the amount of text the model can see at once, read about the context window before reaching for training.

How much data you need, and what good data looks like

There is no universal number, but the published anchors are smaller than most people expect.

• A floor. OpenAI's supervised fine-tuning guide sets the minimum at 10 examples, recommends starting with about 50 well-crafted ones, and says improvements tend to appear at roughly 50 to 100 examples, depending on the use case. If 50 good examples have no effect, it suggests revisiting the task or the prompt before adding more data (OpenAI SFT guide).
• A thousand, carefully chosen. The LIMA result above used 1,000 curated pairs to get a strong instruction-following model out of a large base.
• More for new skills. Tasks that require new capabilities rather than new formatting generally need far more data and a base model that already has the underlying ability. Treat any specific number you see for this as a rough planning figure, not a guarantee.

What matters more than count is fit and cleanliness. Your examples should look like the real inputs the model will face, including the messy ones. Labels must be correct, because a model trained on contradictory answers learns the contradiction. If you want JSON out, every training target must be valid JSON. Hugging Face's TRL accepts plain text, chat-style messages, and prompt-and-completion pairs, and applies the model's chat template automatically for conversational data, which removes one common source of silent formatting bugs.

Synthetic data deserves care. Generating training examples with a stronger model is common and can work, but errors and stylistic tics in the generator get baked into your model. The trade-offs are covered in synthetic data explained. A related idea, training a small model to imitate a large one, is the subject of model distillation.

One more rule belongs here. Split off your evaluation set before you collect training data, and keep the two from overlapping. If test questions leak into training, your score measures memory, not skill.

A step-by-step process that avoids the usual traps

1. Write down the task. One page: what goes in, what comes out, and what counts as success.
2. Build the evaluation set first. A held-out set of 50 to 200 examples with reference answers is a reasonable starting range. Without it you cannot tell progress from noise. The mechanics of building one are in LLM evaluation explained.
3. Baseline the base model. Score it zero-shot and with a few examples in the prompt. If the gap to your target is small, you may be done.
4. Collect and clean training data. Remove duplicates, fix labels, and check that no evaluation example slipped in.
5. Format it. Convert to the chat template the model expects. Decide whether the loss covers only the assistant's tokens.
6. Pick the smallest model that might work. Smaller models are cheaper to train, serve, and iterate on.
7. Start with LoRA on all layers. Choose a moderate rank, apply adapters to every layer type unless you have a reason not to, and use the QLoRA recipe if memory is tight.
8. Set the learning rate for the method. TRL's defaults for full SFT use a learning rate of 2e-5 and three epochs, and its documentation suggests roughly 1e-4 when training adapters. The Thinking Machines work found LoRA's best rate to be about 10 times the full fine-tuning rate, which agrees with that range.
9. Evaluate on two things. Your task set, and a handful of general prompts unrelated to the task. The second check catches forgetting.
10. Fix data before hyperparameters. When results disappoint, inspect the failures and improve examples first. Tuning knobs on bad data rarely helps.

What fine-tuning costs in 2026

The pricing landscape has shifted, so avoid copying numbers from older tutorials.

OpenAI's documentation now says it is winding down its fine-tuning platform: new users can no longer access it, existing users can create training jobs for a limited time, and existing fine-tuned models remain available for inference until their base models are deprecated (OpenAI SFT guide). Any plan that depends on starting a new OpenAI fine-tune should be checked against that notice first. Anthropic made fine-tuning of Claude 3 Haiku generally available in Amazon Bedrock on November 1, 2024, initially in the US West (Oregon) region (AWS announcement); check the current Bedrock model and region list before planning around it.

For open models, the cost is mostly GPU time. The formula is simple: hours of training multiplied by the hourly price of the GPU, plus the time you spend on data. A QLoRA run that fits on one GPU is a short, cheap experiment; a full fine-tune across many GPUs is a project. Optimization libraries narrow the gap. Hugging Face's TRL documentation describes Unsloth as training models up to twice as fast with up to 70% less VRAM, while staying compatible with the Hugging Face workflow.

The cost that surprises teams is not compute. It is the human time to write, review, and correct a few hundred to a few thousand good examples, and to rebuild the evaluation set when the product changes.

Forgetting, drift, and safety

Fine-tuning can damage what a model already did well. Luo and colleagues studied continual instruction tuning on models from about 1B to 7B parameters and found catastrophic forgetting was generally observed across that range. In their tests the larger models forgot more, and general instruction tuning before task-specific tuning reduced later forgetting (Luo et al., 2023). Combined with Biderman's finding that LoRA forgets less than full fine-tuning, the practical defenses are clear:

• Use LoRA rather than updating every weight when you can.
• Keep learning rates and epoch counts modest.
• Mix in some general instruction data so the model keeps practicing broad behavior.
• Test outside the target task every time.

Safety is a separate concern. Qi and colleagues showed that fine-tuning GPT-3.5 Turbo on just 10 adversarially designed examples, at a cost under $0.20 through OpenAI's API, removed its safety guardrails. Fine-tuning on ordinary benign datasets also weakened safety alignment, though less severely (Qi et al., 2023). If you ship a fine-tuned model to users, rerun your safety and refusal tests after training. Alignment that came with the base model is not guaranteed to survive.

Fine-tuning also does not repair a model's tendency to state wrong things confidently. If hallucination is your core problem, read why AI hallucinates facts first; teaching new facts through training can make it worse, as the Gekhman study found.

Serving what you trained

A trained LoRA adapter is a small file, often a tiny fraction of the base model's size. That has a useful consequence: one base model can sit in memory while you swap different adapters for different tasks or customers. When you want a single standalone model instead, PEFT's merge function folds the adapter into the base weights, and the result runs like any other model.

For local experiments, merged models can be converted to formats that local runtimes understand. Our overview of Ollama and local LLMs covers the usual path for running a finished model on your own machine.

Common mistakes and misconceptions

"Fine-tuning teaches the model our documents." It mostly does not. It teaches style, format, and task patterns. For knowledge, retrieval has the stronger evidence.

"More epochs mean a better model." Past a point, the model memorizes the training set and gets worse on new inputs. Watch the validation loss and your held-out evaluation, not only the training loss.

"LoRA is always as good as full fine-tuning." It often matches on small datasets when applied to all layers, and it can fall short when the dataset exceeds the adapter's capacity.

"We need a huge dataset." The LIMA result and OpenAI's guidance both suggest that a small, clean set often beats a large, noisy one.

"Safety comes with the base model." The Qi study shows it can erode with only a few examples.

"If it fails, tune the hyperparameters." Look at the data first. Mislabeled or inconsistent examples are the most common culprit.

Frequently asked questions

Q: Should I fine-tune or use RAG? A: Use retrieval when the gap is missing or changing facts, and fine-tune when the gap is behavior such as format, tone, or a repeatable skill. Research comparing the two for knowledge injection found retrieval consistently outperformed unsupervised fine-tuning. Many production systems use both.

Q: How many examples do I need? A: OpenAI's guide sets a minimum of 10, suggests starting with about 50 well-crafted examples, and notes improvements at roughly 50 to 100. The LIMA study reached strong results with 1,000 curated examples. New capabilities generally need more, and quality outweighs quantity.

Q: What is the difference between LoRA and QLoRA? A: LoRA freezes the base model and trains small low-rank adapters. QLoRA does the same but first compresses the frozen base model to 4-bit precision, which is what lets a 65B-parameter model be fine-tuned on a single 48 GB GPU in the original paper.

Q: Can I fine-tune a frontier model like GPT or Claude? A: It depends on the provider and the date. OpenAI says it is winding down its fine-tuning platform to new users. Fine-tuning of Claude 3 Haiku became available in Amazon Bedrock in November 2024. Check each provider's current documentation, because availability changes quickly.

Q: Will fine-tuning make the model forget other things? A: It can. Studies of continual fine-tuning on 1B to 7B models observed catastrophic forgetting, and LoRA tends to forget less than full fine-tuning. Mix in general data, keep learning rates modest, and always test outside your task.

Q: Does fine-tuning make a model safer or less safe? A: Neither automatically. Research has shown that even a handful of examples can remove safety behavior, and that benign data can weaken it too. Re-run safety tests after every fine-tune.

The takeaway

Fine-tuning is a tool for changing behavior, and the evidence is consistent about that. A model learns its facts during pretraining, and a short round of training mostly teaches it how to present and apply them. That is why a few hundred clean examples can transform a model's output format, while a pile of company documents rarely transforms its knowledge. LoRA and QLoRA turned the technique from a lab-scale expense into something one engineer can try in an afternoon, but they did not remove the hard parts: a trustworthy evaluation set, clean data, and a check for what the model forgot or lost along the way. Start with the prompt, add retrieval for facts, and reach for fine-tuning when you can point to a specific behavior the first two cannot fix.

Sources

• Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
• Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
• Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.
• Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
• Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. arXiv:2305.11206.
• Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. arXiv:2405.09673.
• Schulman, J. et al. (2025). LoRA Without Regret. Thinking Machines Lab.
• Gekhman, Z. et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? arXiv:2405.05904.
• Ovadia, O. et al. (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. arXiv:2312.05934.
• Luo, Y. et al. (2023). An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv:2308.08747.
• Qi, X. et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To. arXiv:2310.03693.
• Hugging Face. LoRA in PEFT and TRL SFT Trainer documentation.
• OpenAI. Model optimization and Supervised fine-tuning guides.
• Amazon Web Services (2024). Fine-tuning for Anthropic's Claude 3 Haiku in Amazon Bedrock is now generally available.