Common ML Mistakes Beginners Make — How to Avoid Them

Most beginners fail at the same mistakes. This guide helps you skip them.

---

Mistake 1: Overfitting (The Silent Killer)

What: Model memorizes training data instead of learning patterns.

Example:

Training accuracy: 99%
Testing accuracy: 45%

Your model memorized train data. Doesn't generalize.

Why it happens:

• Too complex model
• Too little data
• Too many features

How to catch it:

• Always check test accuracy
• If train >> test = overfitting

How to fix:

# 1. Simplify model
model = DecisionTreeClassifier(max_depth=5) # Limit depth

# 2. Add regularization
model = LogisticRegression(C=0.1) # C is strength (lower = more)

# 3. Get more data
# 4. Remove irrelevant features

---

Mistake 2: Using Wrong Metric

What: Measuring success with irrelevant metric.

Example:

Problem: Detect rare disease (1% of population)

Dumb metric: Accuracy
"Always predict no disease" → 99% accuracy ✅

Smart metric: F1-score, Precision, Recall
"Always predict no disease" → 0 F1-score ❌

Choose right metric:

Problem	Metric
Balanced classes	Accuracy
Imbalanced classes	F1-score, AUC
Rare events	Precision, Recall
Regression	RMSE, R²
Ranking	MAP, NDCG

---

Mistake 3: Data Leakage

What: Train data contains information from test data (cheating).

Example (Bad):

# Scale all data first
X_scaled = scaler.fit_transform(X)

# Then split
X_train, X_test = train_test_split(X_scaled)

# WRONG! Scaler saw test data mean during fit()

Example (Good):

# Split first
X_train, X_test = train_test_split(X)

# Scale separately
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test) # Use fit_transform from train

# Scaler never saw test data

How to avoid:

1. Split train/test FIRST
2. Build pipeline using only train data
3. Apply to test data

---

Mistake 4: No Baseline Model

What: Build complex model, 85% accuracy. Is that good?

Don't know! You need baseline.

Baseline ideas:

# Baseline 1: Always predict most common class
baseline = y_train.value_counts().idxmax()
baseline_accuracy = (y_test == baseline).mean()

# If your model isn't better than baseline, fail

# Baseline 2: Simple model
from sklearn.tree import DecisionTreeClassifier
baseline = DecisionTreeClassifier(max_depth=1) # Very simple

Rule: If your complex model isn't better than simple baseline, scrap it.

---

Mistake 5: Ignoring Class Imbalance

What: 95% class A, 5% class B.

Problem:

Model: Predict A 95% of time (cheating)
Accuracy: 95% (seems good!)
But: Never predicts class B (useless)

How to fix:

Option 1: Balanced weights

model = RandomForestClassifier(class_weight='balanced')

Option 2: Resampling

from imblearn.over_sampling import SMOTE

smote = SMOTE()
X_balanced, y_balanced = smote.fit_resample(X_train, y_train)
model.fit(X_balanced, y_balanced)

Option 3: Use right metric

f1 = f1_score(y_test, y_pred) # Better than accuracy

---

Mistake 6: Using Test Data Multiple Times

What: Tune hyperparameters on test set.

Problem:

for alpha in [0.001, 0.01, 0.1, 1.0]:
 model = LogisticRegression(C=alpha)
 model.fit(X_train, y_train)
 score = model.score(X_test, y_test) # BAD!
 if score > best:
 best = score
 best_alpha = alpha

# You're overfitting to test set!

How to fix: Use validation set

X_train, X_val, X_test = split in 60/20/20

for alpha in [0.001, 0.01, 0.1, 1.0]:
 model = LogisticRegression(C=alpha)
 model.fit(X_train, y_train)
 score = model.score(X_val, y_val) # GOOD
 if score > best:
 best_alpha = alpha

# Evaluate ONCE on test set (final check)
final_score = model.score(X_test, y_test)

---

Mistake 7: Not Checking Data Quality

What: Garbage data → garbage model.

Things to check:

# Missing values
print(X.isnull().sum())

# Outliers
print(X.describe()) # Look at min/max

# Class imbalance
print(y.value_counts())

# Duplicates
print(X.duplicated().sum())

# Leaks (features that contain target)
print(X.corr(with_target))

---

Mistake 8: Over-optimizing One Metric

What: Maximize accuracy, ignore interpretability/speed/fairness.

Example:

Loan approval model:
- Complex model: 95% accuracy, can't explain decisions (bias lawsuit)
- Simple model: 92% accuracy, can explain (lenders like it)

Choose simple.

Optimize for:

• Accuracy
• Speed
• Interpretability
• Fairness
• Computational cost

---

Mistake 9: Not Documenting

What: Build model, forget how it works.

Better:

"""
Model: RandomForest for house price prediction
Features: Square footage, bedrooms, location
Data: 1000 homes (2025 dataset)
Accuracy: 0.87 R²
Preprocessing: StandardScaler, OneHotEncoder
Known issues: Overfits on luxury homes
Last updated: 2026-05-12
"""

---

Mistake 10: Insufficient Testing

What: Trust model on new data without testing edge cases.

Test cases:

# Edge case 1: Very small house
small = [[100, 1, 'bad_location']]
print(model.predict(small))

# Edge case 2: Very large house
large = [[10000, 10, 'best_location']]
print(model.predict(large))

# Edge case 3: Missing features
missing = [[None, 5, 'ok_location']]
print(model.predict(missing))

# Edge case 4: Realistic but unusual
unusual = [[5000, 1, 'rural']]
print(model.predict(unusual))

If predictions seem wrong, investigate.

---

Checklist: Avoid These Mistakes

• Check train accuracy >> test accuracy (overfitting?)
• Use right metric for problem
• Split train/test BEFORE scaling/engineering
• Compare to baseline
• Check for class imbalance
• Tune hyperparameters on validation set, not test
• Check data quality (missing, outliers, duplicates)
• Optimize multiple metrics (not just accuracy)
• Document your model
• Test edge cases

---

Advanced Debugging Techniques

When basic checks don't explain poor performance, dig deeper.

Mistake 11: Ignoring Statistical Significance

Your model: 85% accuracy Baseline: 82% accuracy

Is 3% improvement real or random noise?

Solution: Confidence intervals

from sklearn.metrics import accuracy_score
from scipy import stats

accuracies = []
for i in range(100):
    indices = np.random.choice(len(y_test), len(y_test), replace=True)
    y_true_boot = y_test[indices]
    y_pred_boot = y_pred[indices]
    accuracies.append(accuracy_score(y_true_boot, y_pred_boot))

ci_lower = np.percentile(accuracies, 2.5)
ci_upper = np.percentile(accuracies, 97.5)

print(f"Accuracy: {np.mean(accuracies):.1%} [95% CI: {ci_lower:.1%}-{ci_upper:.1%}]")

# If baseline 82% is outside CI, improvement is statistically significant

Mistake 12: Not Testing on Time-Shifted Data

Your model trained on 2024 data. You test on 2024 data. Result: 90% accuracy.

Deploy to 2025. Result: 70% accuracy.

Why? Data distribution shifted (new trends, seasonality, behavior changes).

Solution: Temporal validation

# Don't random split; time-based split
cutoff = '2024-12-01'

X_train = data[data['date'] < cutoff].drop('date', axis=1)
y_train = data[data['date'] < cutoff]['target']

X_test = data[data['date'] >= cutoff].drop('date', axis=1)
y_test = data[data['date'] >= cutoff]['target']

model.fit(X_train, y_train)
accuracy_real_world = model.score(X_test, y_test)

Always use temporal splits for time-series/forecasting problems.

Mistake 13: Threshold Tuning Confusion

Binary classification model outputs probabilities (0-1).

Default threshold: 0.5 (if prob > 0.5 → predict positive).

But optimal threshold might be 0.3 or 0.7.

Solution: Plot precision-recall curve

from sklearn.metrics import precision_recall_curve, roc_curve
import matplotlib.pyplot as plt

precisions, recalls, thresholds = precision_recall_curve(y_test, y_pred_proba)

plt.plot(recalls, precisions)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.show()

# Pick threshold maximizing your objective (recall vs precision tradeoff)

---

Model Selection Strategy

Don't just build one model. Compare systematically.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

models = {
    'Logistic Regression': LogisticRegression(),
    'Random Forest': RandomForestClassifier(n_estimators=100),
    'Gradient Boosting': GradientBoostingClassifier(n_estimators=100),
    'SVM': SVC(probability=True),
    'KNN': KNeighborsClassifier(n_neighbors=5)
}

results = {}
for name, model in models.items():
    model.fit(X_train, y_train)
    train_acc = model.score(X_train, y_train)
    test_acc = model.score(X_test, y_test)
    results[name] = {'train': train_acc, 'test': test_acc}

# Print comparison
for name, scores in results.items():
    print(f"{name}: train={scores['train']:.3f}, test={scores['test']:.3f}, "
          f"gap={scores['train'] - scores['test']:.3f}")

Example output:

Logistic Regression: train=0.850, test=0.835, gap=0.015 (good, low gap)
Random Forest: train=0.998, test=0.840, gap=0.158 (bad, overfitting)
Gradient Boosting: train=0.920, test=0.855, gap=0.065 (ok)
SVM: train=0.880, test=0.850, gap=0.030 (good)
KNN: train=0.900, test=0.810, gap=0.090 (ok)

Winner: Logistic Regression or SVM (low overfitting, good test accuracy).

---

When to Give Up on a Model

Sometimes, no amount of tuning helps.

Signs to abandon:

1. Test accuracy < random baseline

   • Random guessing: 50% (binary classification)
   • Your model: 45%
   • Likely: Features don't correlate with target. Get better data.

2. Train accuracy = test accuracy = 50%

   • Model didn't learn anything.
   • Likely: Features too weak or target noise.

3. Accuracy plateaus after week of tuning

   • Law of diminishing returns applies.
   • Pivot: Get better features, more data, or different problem.

4. Inference cost > business value

   • Model: 95% accuracy, but takes 10 seconds per prediction
   • Business: Needed in 1 second
   • Solution: Use simpler model (lower accuracy, faster)

---

Frequently Asked Questions

Q: Is 99% accuracy always good?
A: No. If classes imbalanced (1% positive), 99% accuracy = always predict negative (useless). Use F1-score or AUC instead.

Q: How do I spot overfitting early?
A: Epoch 1-10: train loss drops, test loss stays same → overfitting starting. Stop training.

Q: Can I fix data leakage after the fact?
A: Not fully. Re-run your entire pipeline with corrected split. Leakage invalidates all prior metrics.

Q: Should I always use cross-validation?
A: Yes, for small datasets (<10K samples). For large datasets, single train/val/test split sufficient.

Q: What if I only have 100 samples?
A: Use 5-fold cross-validation (80/20 per fold). Use simple model (e.g., logistic regression). Be humble about accuracy claims.

Q: How long should training take?
A: Scikit-learn: seconds to minutes. Neural networks: minutes to hours (on GPU). If much longer, likely bug.

---

Read next:

• How to Build Your First ML Model — Step-by-Step
• ML Algorithms Explained — Linear Regression, Decision Trees
• Feature Engineering in ML — What It Is & Why It Matters

---

Bottom Line

Most mistakes are about overfitting or bad evaluation.

Build simple models first. Check everything (train vs test, metrics, data quality).

90% of ML work is not the model, it's avoiding these mistakes.

Last verified: May 2026