Why 99% Accuracy Can Be Useless: Precision and Recall

A classifier that labels every transaction "legitimate" is 99% accurate on a dataset where 1% of transactions are fraud, and it catches zero fraud. Accuracy counts correct answers, and when one class makes up almost everything, guessing that class is correct almost every time. Precision and recall were built to see past this: precision asks how many of the model's alarms were real, recall asks how many real cases the model found. This article builds the confusion matrix, works one example through every metric with real arithmetic, and explains how thresholds, F1, ROC curves, and precision-recall curves behave when the thing you care about is rare.

The 99%-accurate model that finds nothing

Google's Machine Learning Crash Course makes the point with a rare class that appears about 1% of the time: a model that predicts negative 100% of the time would score 99% on accuracy, despite being useless. The same trap appears everywhere that important events are uncommon. Rare diseases in screening data, fraudulent payments, equipment about to fail, defective parts on a line, the one email in a thousand that is a phishing attempt.

The reason is arithmetic, not a bug. Accuracy is the proportion of all classifications that were correct, whether positive or negative. If 9,900 of 10,000 cases are negative, a model that is right about every negative and wrong about every positive scores 9,900 out of 10,000. The 100 cases you actually wanted to find are rounding error in the denominator.

An analogy: a smoke detector that never goes off is correct on every day your house is not burning, which is nearly every day. Nobody would call it 99.9% good. You judge a smoke detector on the rare event, and on how often it cries wolf.

That is the shift this topic asks for. For balanced problems, such as classifying photos that are half cats and half dogs, accuracy is a fine summary. For imbalanced problems, you need metrics that look at the minority class on its own terms.

The confusion matrix

Every classification metric in this article comes from four counts. For a binary problem with a "positive" class (the thing you are looking for) and a "negative" class, each prediction lands in one of four cells:

	Predicted positive	Predicted negative
Actually positive	True positive (TP): correctly flagged	False negative (FN): missed
Actually negative	False positive (FP): false alarm	True negative (TN): correctly ignored

The matrix is a ledger of the two ways to be right and the two ways to be wrong, and the two kinds of wrong have different costs. A false negative is a fraud that got through. A false positive is a legitimate customer whose card was blocked. Accuracy adds the two right cells and divides by everything, which is why it cannot tell these errors apart.

One dataset to use throughout

To keep the numbers honest, here is a constructed example (not real data). A payments system reviews 10,000 transactions, of which 100 are fraudulent. Three models are tried.

Model	TP	FP	FN	TN
A: always predicts "legitimate"	0	0	100	9,900
B: cautious threshold	80	70	20	9,830
C: aggressive threshold	95	305	5	9,595

Each row sums to 10,000, and each has 100 actual positives (TP plus FN) and 9,900 actual negatives (FP plus TN). We will compute every metric for all three.

Precision vs recall, with a worked example

Recall (also called sensitivity or true positive rate) is TP divided by TP plus FN: of all the real positives, what fraction did the model catch? Precision is TP divided by TP plus FP: of everything the model flagged as positive, what fraction was really positive? Scikit-learn's documentation gives the same formulas.

Running the numbers:

Model	Accuracy	Recall	Precision
A	99.0%	0%	undefined (no flags)
B	99.1%	80%	53.3% (80 of 150 flags)
C	96.9%	95%	23.75% (95 of 400 flags)

Model A: accuracy (0 + 9,900) / 10,000 = 99.0%, recall 0 of 100. Model B: accuracy (80 + 9,830) / 10,000 = 99.1%, recall 80 of 100, precision 80 of 150. Model C: accuracy (95 + 9,595) / 10,000 = 96.9%, recall 95 of 100, precision 95 of 400.

Look at what accuracy says. It ranks the useless Model A (99.0%) above the aggressive Model C (96.9%), even though C catches 95 frauds and A catches none. Model B edges A by a tenth of a point, a gap that gives no hint that one model is a fraud detector and the other is a rubber stamp. Accuracy has lost the plot, and recall found it immediately.

Precision tells the other half of the story. Model C catches nearly everything, but three out of four of its alarms are false. Whether that is acceptable depends on what a false alarm costs, which is why a single number can never settle the question.

Remembering which is which

A way to keep them straight: precision looks at the model's output ("when it says yes, how often is it right?"), and recall looks at reality ("when the answer really is yes, how often does the model say so?"). Precision is about trust in an alarm. Recall is about coverage of the problem.

The threshold dial: why you cannot maximize both

Most classifiers do not output a hard yes or no. They output a score, often a probability, and a threshold converts the score into a decision: flag everything above 0.5, say. Moving the threshold changes the confusion matrix.

Google's crash course describes the effect: raising the classification threshold usually reduces false positives and increases false negatives, and lowering it does the opposite. Precision rises as false positives fall, recall rises as false negatives fall, so improving one tends to worsen the other.

Models B and C in the example could be the same underlying classifier at different thresholds. B uses a strict cutoff: 150 flags, few false alarms, but 20 frauds slip through. C uses a loose cutoff: 400 flags, only 5 frauds missed, but 305 false alarms. Sliding along this dial traces out the whole trade-off, and the full set of thresholds is what ROC and precision-recall curves plot.

Picture a metal detector at an airport. Turn the sensitivity up and it catches every concealed item, along with belt buckles and keys. Turn it down and the line moves faster and a few real threats get through. No setting is correct in the abstract. The right setting depends on what a miss costs compared with what a false alarm costs.

Choosing the threshold by cost

Suppose, purely as a hypothetical, that a missed fraud costs $200 and a false alarm costs $5 in manual review. Total cost per 10,000 transactions:

Model	Missed frauds (FN)	False alarms (FP)	Cost
A	100	0	100 x $200 = $20,000
B	20	70	20 x $200 + 70 x $5 = $4,350
C	5	305	5 x $200 + 305 x $5 = $2,525

Under those costs, Model C is the best of the three, even though it has the lowest precision and the lowest accuracy among the models that catch fraud. Change the costs (a false alarm now loses a customer worth $500) and the answer flips. The cost numbers are invented for illustration, but the lesson holds generally: the metric that matches the real-world cost structure matters more than any universal score, and the threshold should be tuned on validation data, not left at the default 0.5.

F1, F-beta, and when F1 hides things

People often want one number that combines precision and recall. The standard answer is F1, the harmonic mean of the two. Scikit-learn gives the general F-beta form as (1 + beta squared) times precision times recall, divided by (beta squared times precision plus recall), and notes that with beta equal to 1, F1 weights the two equally. With beta below 1 precision is favored, and above 1 recall is favored.

For the worked example, F1 = 2TP / (2TP + FP + FN), the form scikit-learn uses to avoid division by zero:

• Model A: 0.
• Model B: 160 / (160 + 70 + 20) = 160 / 250 = 0.64.
• Model C: 190 / (190 + 305 + 5) = 190 / 500 = 0.38.

F1 ranks B above C, the reverse of the cost-based ranking above. Nothing is wrong with the arithmetic. F1 assumes a false alarm and a miss are equally bad, and in our hypothetical a miss is forty times worse. Using F2 (beta equal to 2, which favors recall) gives 5TP / (5TP + FP + 4FN): B scores 400 / 550 = 0.73 and C scores 475 / 800 = 0.59, still favoring B, which tells you that even F2 is not weighting misses as heavily as the hypothetical costs do. No F-score replaces writing down real costs.

Why a harmonic mean

The harmonic mean punishes lopsided pairs. A model that flags every transaction as fraud gets recall 1.0 and precision of 1%. The arithmetic mean of those is about 0.505, which sounds respectable. The harmonic mean is 2 x 0.01 x 1.0 / 1.01, about 0.02, which correctly says the model is nearly worthless. That sensitivity to the weaker of the two numbers is the reason F1 exists.

What F1 leaves out

F1 has three blind spots worth knowing:

1. It ignores true negatives. TN never appears in the formula. That is a feature for rare-event problems and a bug if correctly ignoring negatives matters to you.
2. It depends on which class you call positive. Swap the labels and you get a different F1. On balanced data this matters little; on imbalanced data it can matter a lot.
3. It assumes equal weighting of errors. As shown above, real costs rarely are equal.

Scikit-learn also provides balanced accuracy, the macro-average of per-class recall, equal to accuracy when classes are balanced. For the always-negative Model A it gives (0 + 1) / 2 = 0.5, exactly chance level, which is the honest verdict. Scikit-learn's documentation says balanced accuracy avoids inflated performance estimates on imbalanced datasets, and that if conventional accuracy is above chance only because the classifier exploits an imbalanced test set, balanced accuracy drops to 1 divided by the number of classes. For Models B and C, balanced accuracy is about 0.90 and 0.96.

ROC curves vs precision-recall curves on rare classes

Two curves summarize the threshold trade-off across all cutoffs.

The ROC curve (receiver operating characteristic) plots the true positive rate (recall) against the false positive rate (FP divided by FP plus TN) as the threshold sweeps. The precision-recall curve plots precision against recall. A summary number for the first is ROC AUC, the area under it; for the second, scikit-learn's average precision (AP) is a weighted sum of precisions at each threshold, where AP for random predictions equals the fraction of positive samples.

Tom Fawcett's 2006 guide "An introduction to ROC analysis" in Pattern Recognition Letters explains ROC graphs and their common misconceptions and pitfalls. One pitfall shows up in our example. Model B's operating point has a true positive rate of 80% and a false positive rate of 70 out of 9,900, or 0.7%. On a ROC plot that is a point near the top-left corner, the region that looks excellent. Yet its precision is 53%: nearly half of the alarms are false.

The explanation is the denominator. The false positive rate divides by the huge pool of negatives, so even a large number of false alarms looks like a small rate. Precision divides by the flags raised, so it feels every false alarm directly.

What the research says

Takaya Saito and Marc Rehmsmeier's 2015 paper in PLoS ONE, "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," argues that ROC plots can be misleading on imbalanced data because their visual appeal can be deceptive about reliability, as specificity is commonly misread, while precision-recall plots evaluate the fraction of true positives among positive predictions and give a more accurate view of future performance. In their literature review, 33 of the 58 research articles they analyzed met their criteria for binary SVM studies with large, imbalanced datasets.

There is a nuance. Jesse Davis and Mark Goadrich showed at ICML 2006 that a curve dominates in ROC space if and only if it dominates in precision-recall space, so the two views agree on which of two models is better when one is better at every threshold. The difference is in how the picture reads and in how the summary areas behave. A ROC curve can look nearly perfect for a model whose precision at useful recall is poor, and a precision-recall curve will show it.

Two practical rules follow:

• Compare against the right baseline. On a ROC plot, a random classifier is the diagonal, a baseline that does not depend on class balance. On a precision-recall plot, a random classifier sits at a horizontal line at the prevalence of the positive class (1% in our example), so a model with AP of 0.30 is thirty times better than random, not "bad."
• Mind interpolation. Scikit-learn's documentation notes that its average precision does not use trapezoidal interpolation on the precision-recall curve, because work by Davis and Goadrich and by Flach and Kull argues that linear interpolation of precision-recall points gives an overly optimistic measure.

Precision depends on how rare the positives are

A final subtlety separates precision from recall. Recall and the false positive rate describe the classifier given each class, so they stay put when prevalence changes. Precision does not. Take Model B's rates (recall 80%, false positive rate about 0.7%) and apply them to populations with different shares of positives, 10,000 cases each:

Positive share	Positives	TP	Negatives	FP	Precision
0.1%	10	8	9,990	about 71	about 10%
1%	100	80	9,900	70	53%
10%	1,000	800	9,000	about 64	about 93%
50%	5,000	4,000	5,000	about 35	about 99%

The same classifier, with the same recall and the same false positive rate, goes from a precision of about 10% to about 99%, purely because positives became more common. This is Bayes' theorem at work, and it is why a test with impressive-sounding sensitivity and specificity can still produce mostly false alarms when screening for something rare. It also means a model evaluated on a rebalanced test set will report a flattering precision that does not hold in production. Evaluate on data that has the real-world class mix, or report the prevalence next to the metric.

Handling imbalance: weighting, resampling, and their traps

When the minority class is rare, the training process itself may neglect it, since a model minimizing overall error has little incentive to learn a class that is 1% of the data. Haibo He and Edwardo Garcia's 2009 review "Learning from Imbalanced Data" in IEEE Transactions on Knowledge and Data Engineering surveys the problem, framing it as how learning algorithms perform when some classes are underrepresented and class distributions are severely skewed. Common remedies fall into three groups.

1. Class weights. Tell the learning algorithm that errors on the minority class count more. Many scikit-learn estimators accept a class_weight argument for this. It requires no change to the data, and it shifts the model toward flagging more positives, which raises recall and lowers precision. It is closely related to moving the threshold.

2. Resampling. Change the training data instead. Random oversampling duplicates minority rows; random undersampling throws away majority rows. SMOTE, the Synthetic Minority Over-sampling Technique from Nitesh Chawla, Kevin Bowyer, Lawrence Hall, and Philip Kegelmeyer (Journal of Artificial Intelligence Research, 2002), creates new synthetic minority examples and combines this with undersampling the majority. The authors reported better classifier performance in ROC space than undersampling alone.

3. Threshold moving. Train normally, then choose the threshold on validation data to hit a target recall or cost, as in the cost example above. It is simple and often as effective as the other two.

Traps that spoil the evaluation

Resampling comes with traps that quietly inflate results.

• Resample after splitting, never before. If you oversample first and then split, copies of the same minority row land in both training and test sets, and the model is graded on data it has seen. This is the leakage described in why data leakage breaks AI evaluations, and the separation rules in why training and test data must be separated apply with extra force. The test set should keep the real class mix.
• Resampling distorts probabilities. A model trained on a 50/50 rebalanced set will output probabilities that overstate how common positives are in reality. If you need calibrated probabilities (for expected-cost decisions, say), you must correct for this or use weights and thresholds instead.
• Synthetic points can be unrealistic. SMOTE interpolates between neighbors, which can create examples in regions where no real positives exist, particularly with noisy data or categorical features.
• Overfitting to duplicates. Plain oversampling can make a model memorize the few minority rows it sees repeatedly, a version of the problem in overfitting and underfitting.

Tree ensembles such as random forests and gradient-boosted models like XGBoost accept sample weights, and XGBoost exposes a parameter for scaling the weight of positive examples. Their default behavior still optimizes overall error, so on rare classes it pays to set the weight, tune the threshold, and evaluate with the metrics here instead of accuracy.

Multi-class and other extensions

Everything above is binary. With more than two classes, you compute precision and recall per class and then average them. A macro average treats every class equally, so a rare class counts as much as a common one, which suits imbalanced problems. A micro average pools all the counts first, so large classes dominate. A weighted average weights each class by how many true examples it has. Scikit-learn's balanced accuracy is the macro-average of per-class recall, and its ROC AUC supports one-vs-one and one-vs-rest macro and weighted averaging (micro only for one-vs-rest).

The choice of average is a statement about what you care about. If missing a rare category is costly, macro averaging will show it. If overall volume matters most, micro averaging reflects that. Report which one you used, because the numbers differ.

The same logic applies when evaluating systems outside classical classification. For language models, evaluation sets are often skewed toward easy cases, and a headline score can hide failures on the rare, hard ones, a point developed in LLM evaluation explained.

Common misconceptions

"High accuracy means a good model." Only if classes are balanced and errors cost the same. With 1% positives, 99% accuracy is the baseline for doing nothing.

"Precision and recall are fixed properties of a model." They are properties of a model at a threshold, on a particular data distribution. Change either and the numbers change.

"F1 is the neutral choice." F1 encodes the assumption that false alarms and misses are equally costly. It is a choice, and often a poor match to real costs.

"ROC AUC of 0.95 means the alarms are usually right." It does not speak to precision at all. A high ROC AUC is compatible with low precision when positives are rare.

"Balancing the test set makes evaluation fairer." It makes the numbers look better and answers a different question. Precision in particular will not transfer to the real class mix.

"Resampling always helps." It can help recall, and it can also distort probabilities, leak information, or add unrealistic samples. Threshold tuning and class weights are simpler first tries.

Frequently asked questions

Q: Why is accuracy misleading on imbalanced data? A: Accuracy counts correct predictions across all cases, so when one class dominates, predicting that class every time scores high. A model that always predicts negative scores 99% when positives are 1% of the data, while catching none of them.

Q: What is the difference between precision and recall? A: Precision is the share of the model's positive predictions that were correct (TP divided by TP plus FP). Recall is the share of actual positives the model found (TP divided by TP plus FN). Precision measures how trustworthy an alarm is, and recall measures how much of the problem is covered.

Q: When should I use F1 instead of accuracy? A: Use F1 when the positive class is rare and you care about both false alarms and misses roughly equally. F1 ignores true negatives, so it will not be inflated by a large pool of easy negatives. If misses and false alarms have very different costs, use F-beta or an explicit cost calculation.

Q: Is a ROC curve or a precision-recall curve better for imbalanced data? A: Saito and Rehmsmeier (2015) argue the precision-recall plot is more informative on imbalanced datasets, because ROC plots can look strong when many false alarms are hidden behind a large pool of negatives. Davis and Goadrich (2006) showed the two curves agree on which model dominates, so they differ mainly in how clearly they show the problem.

Q: How do I choose the classification threshold? A: Choose it on validation data, not by default. Decide which error is costlier, then pick the threshold that meets a target recall, maximizes an F-beta score that reflects the cost ratio, or minimizes total expected cost. Re-check it when the class mix in production changes.

Q: Should I use SMOTE or class weights? A: Neither is universally better. Class weights are simpler and keep the data intact, and threshold tuning is simpler still. SMOTE, introduced by Chawla and colleagues in 2002, can help some models, but it must be applied only to training data after the split and can distort probabilities, so test it against the simpler options.

Q: What is a good precision-recall AUC or average precision? A: It depends on prevalence. For random predictions, scikit-learn notes that average precision equals the fraction of positive samples, so 0.30 is excellent when positives are 1% of the data and weak when they are 40%. Always compare to the prevalence baseline.

The takeaway

Accuracy answers "how often was the model right?" and that question stops being useful once one outcome swamps the rest. The confusion matrix breaks the answer into the four things that can happen, and precision and recall read it from two sides: how much the alarms can be trusted, and how much of the real problem was found. The threshold is the dial that trades one for the other, and F1, ROC AUC, and average precision are only compressed views of that dial.

The worked example carries the main lesson. A model with 99.0% accuracy caught nothing, a model with 96.9% accuracy caught 95 of 100 frauds, and which of the two working detectors was better depended on prices that no metric supplies. Report prevalence, name the costs, tune the threshold on held-out data with the real class mix, and treat any single number, including F1, as a summary of a decision you still have to make. For the wider evaluation toolkit, the complete machine learning guide places these metrics in the full modeling workflow.

Sources

• Google, Machine Learning Crash Course, "Classification: Accuracy, recall, precision, and related metrics": https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall
• scikit-learn User Guide, "Metrics and scoring: quantifying the quality of predictions": https://scikit-learn.org/stable/modules/model_evaluation.html
• Saito and Rehmsmeier, "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," PLoS ONE 10(3): e0118432, 2015: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432
• Chawla, Bowyer, Hall, and Kegelmeyer, "SMOTE: Synthetic Minority Over-sampling Technique," Journal of Artificial Intelligence Research 16, 2002 (arXiv:1106.1813): https://arxiv.org/abs/1106.1813
• Davis and Goadrich, "The Relationship Between Precision-Recall and ROC Curves," ICML 2006 (DOI 10.1145/1143844.1143874): https://doi.org/10.1145/1143844.1143874
• Fawcett, "An introduction to ROC analysis," Pattern Recognition Letters 27(8), 2006 (DOI 10.1016/j.patrec.2005.10.010): https://doi.org/10.1016/j.patrec.2005.10.010
• He and Garcia, "Learning from Imbalanced Data," IEEE Transactions on Knowledge and Data Engineering 21(9), 2009 (DOI 10.1109/TKDE.2008.239): https://doi.org/10.1109/TKDE.2008.239