How to Handle Missing Data — Imputation Strategies

Real data is messy. In any real-world dataset, you will find missing values. A survey respondent skipped a question, a sensor dropped a reading, a user left a signup field blank — gaps show up in almost every dataset a data scientist touches, from spreadsheets with a few hundred rows to production pipelines processing millions of events a day.

How you handle those gaps matters more than most beginners realize. The wrong strategy can bias your model, distort your statistics, or silently corrupt downstream results in ways that are hard to detect after the fact. This guide walks through the full toolkit — deletion, statistical imputation, time-series fills, and model-based imputation — with working Python code, real-world case studies, and a clear decision framework for when to use each one.

---

Why Missing Data Is a Problem, Not Just an Inconvenience

Most machine learning algorithms cannot operate on missing values directly. Pass a DataFrame containing NaN (Not a Number, pandas' placeholder for a missing entry) into a Scikit-learn estimator's .fit() method and you get a ValueError before training even starts. Tree-based libraries like XGBoost and LightGBM can handle missingness natively by learning a default split direction, but that is the exception, not the rule — linear regression, logistic regression, SVMs, and neural networks all require a fully numeric, gap-free matrix.

Beyond the mechanical error, missing values distort the statistics you rely on to understand your data. A column's mean, standard deviation, and correlation with other variables are all computed only from the rows that have values, so if the missingness itself is not random, those summary statistics describe a biased subset of your population, not the population you actually care about. A income survey where high earners are less likely to disclose their salary will produce a mean income that is too low, no matter how large the sample is.

The statistical framework for reasoning about this precisely comes from Donald Rubin's foundational 1976 paper in Biometrika, "Inference and Missing Data," which formalized three mechanisms of missingness that every data scientist should be able to name and diagnose:

• MCAR (Missing Completely At Random): the probability that a value is missing has no relationship to any variable in the dataset, observed or unobserved. A lab technician accidentally deletes a random row from a spreadsheet. This is the easiest case — deletion or simple imputation introduces no systematic bias, only a loss of statistical power.
• MAR (Missing At Random): the probability of missingness depends on other observed variables, but not on the missing value itself. Older survey respondents might be less likely to complete an online form, but once you condition on age, the missingness of the remaining fields is unrelated to their content. This is the most common real-world case, and it is what most imputation methods, including KNN and iterative imputers, are designed for.
• MNAR (Missing Not At Random): the probability of missingness depends on the value that is missing. People with very high or very low incomes are systematically less likely to report them. This is the hardest case — no imputation method can fully correct for it because the information needed to predict the missing value is, by definition, the thing that caused it to go missing. It requires domain knowledge, sensitivity analysis, or explicit modeling of the missingness mechanism itself (Rubin, 1976, Biometrika; overview at University of Melbourne Statistical Consulting Centre).

Distinguishing these three cases is not academic. It is the single decision that determines whether your chosen imputation strategy will reduce bias or quietly bake more of it into your model.

Donald Rubin's 1976 taxonomy of MCAR, MAR, and MNAR missingness has shaped nearly five decades of statistical practice — and most working data scientists never formally check which category their missing data falls into before choosing an imputation method, even though the choice changes whether that method helps or hurts.

---

Step 0: Diagnose Before You Treat

Before picking a strategy, investigate the shape and pattern of your missing data. Skipping this step is the single most common mistake in data preprocessing — treating a missing-data problem as purely mechanical ("just fill the NaNs") when it is really a statistical inference problem in disguise.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv('data.csv')

# Count missing values per column
print(df.isnull().sum())

# Percentage missing per column
missing_pct = df.isnull().mean() * 100
print(missing_pct[missing_pct > 0].sort_values(ascending=False))

# Visualize missing pattern (helps spot MCAR vs structured missingness)
sns.heatmap(df.isnull(), yticklabels=False, cbar=False, cmap='viridis')
plt.title("Missing Data Pattern")
plt.show()

# Check whether missingness correlates with other columns
# (a rough proxy for distinguishing MCAR from MAR)
for col in df.columns[df.isnull().any()]:
    flag = df[col].isnull().astype(int)
    correlations = df.select_dtypes(include=np.number).corrwith(flag)
    print(f"\nCorrelation of {col}_missing with other numeric columns:")
    print(correlations.sort_values(ascending=False).head(5))

This tells you three things: which columns are missing, how much of each column is missing, and whether the pattern of missingness lines up with other variables. If missingness in one column correlates strongly with the value of another (for example, income_missing correlating with age), you are almost certainly looking at MAR, not MCAR — which changes which tools are appropriate later.

A useful visual aid for this step is the missingno library, which produces matrix and dendrogram views of missingness patterns across a whole dataset in a few lines, making it faster to spot whether gaps cluster by row, by time, or by respondent group.

---

Strategy 1: Delete Rows (Listwise Deletion)

The simplest approach is to drop any row that contains a missing value.

df_clean = df.dropna()  # Remove all rows with any missing values

# Or only drop where a specific column is null:
df_clean = df.dropna(subset=['email', 'user_id'])

Pros: Simple, makes no assumptions about the missing values, and preserves the true distribution of the data that remains.

Cons: You lose data. If you have 10,000 rows and 2,000 have at least one missing value, listwise deletion throws away 20 percent of your dataset even if only one field per row was actually missing.

When to use: Missing data affects less than 5 percent of rows, the dataset is large enough that losing those rows does not meaningfully shrink your training set, and you have reason to believe the missingness is MCAR.

Red flag: If the rows being deleted share a pattern — say, every deleted row belongs to users under 25, or every deleted row comes from a specific region — deletion is not neutral. You are not removing random noise, you are removing a subpopulation, and your model will systematically underperform on that subpopulation once deployed.

Pairwise Deletion — a lesser-known alternative

Instead of dropping a whole row because one column is missing, pairwise deletion computes each statistic (say, a correlation matrix) using only the rows that have both relevant columns present. Pandas' .corr() does this by default. It preserves more data than listwise deletion but can produce a covariance matrix that is not internally consistent (not positive semi-definite), which breaks some downstream algorithms like PCA. Use it for exploratory correlation checks, not for building a training matrix.

---

Strategy 2: Delete Columns (If Too Sparse)

# Keep columns where at least 50% of values are present
threshold = int(0.5 * len(df))
df_clean = df.dropna(axis=1, thresh=threshold)

# Or manually drop specific columns
df_clean = df.drop(columns=['sparse_feature', 'mostly_empty'])

Pros: Removes columns so sparse that any imputation strategy applied to them is mostly guessing, and that guessing can hurt more than it helps.

Cons: You lose whatever real signal that column might have carried, even if the signal was sparse rather than absent.

When to use: A column has more than roughly 50 percent missing values. There is no universal cutoff — 50 percent is a common rule of thumb, but a column that is 70 percent missing can still be worth keeping if the 30 percent that is present is highly predictive and MNAR (its very absence is informative).

Tip: Before deleting a sparse column outright, check whether the fact that it's missing carries signal on its own. A column like reported_income being null might correlate with the outcome you are predicting. Capture that with a binary flag before dropping the raw values.

df['income_was_missing'] = df['income'].isnull().astype(int)
df = df.drop(columns=['income'])  # then drop the original

This pattern — encode missingness as its own feature, then decide separately what to do with the underlying value — is one of the most underused techniques in applied machine learning. It lets a tree-based model learn "people who didn't report income behave differently" without you having to explicitly model why.

---

Strategy 3: Fill With Statistics

This is the most common approach for numeric data, largely because it is fast, interpretable, and easy to explain in a model card or a stakeholder meeting.

Mean Imputation

df['age'] = df['age'].fillna(df['age'].mean())

Use this when the data is roughly normally distributed and the missingness is MCAR. Mean imputation shrinks the variance of the column — every imputed row now sits exactly at the average, which artificially tightens the distribution and can weaken correlations with other variables if a large share of the column is imputed this way.

Median Imputation (Usually the Better Default)

df['income'] = df['income'].fillna(df['income'].median())

Median is more robust to outliers. If your data has extreme values — and income, real estate prices, and transaction amounts almost always do — mean imputation pulls every imputed value toward those extremes. Median is usually the safer default for skewed numeric columns.

Mode Imputation (Categorical)

df['country'] = df['country'].fillna(df['country'].mode()[0])

For categorical columns, fill with the most frequent category. This is a reasonable default when one category dominates, but it becomes a weak choice fast in high-cardinality columns where no single value represents a meaningful plurality — filling every missing country with "United States" because it happens to be the mode can quietly distort a global dataset.

Group-Based Imputation (Smarter Than Global Statistics)

# Instead of a global median, use the median within each group
df['salary'] = df.groupby('department')['salary'].transform(
    lambda x: x.fillna(x.median())
)

This is far more accurate than a global statistic because it conditions on context. A software engineer's median salary is different from a marketing manager's, and filling both with one company-wide number erases a real signal your model could otherwise use. Group-based fills are effectively a lightweight version of MAR-aware imputation — you are using an observed variable (department) to inform the estimate for a missing one (salary).

Forward Fill and Backward Fill (Time Series)

df['temperature'] = df['temperature'].fillna(method='ffill')  # Use previous value
df['temperature'] = df['temperature'].fillna(method='bfill')  # Or next value

Pros: Works well for time series where the previous observation is genuinely the best available estimate of the current one — a reasonable assumption for slow-moving physical quantities.

Cons: Creates artificial flat periods in the data and performs badly when the underlying signal changes quickly (a sensor gap during a rapid temperature spike gets filled with a stale, misleading value).

When to use: Sensor telemetry, stock prices, weather readings — any sequential series where consecutive values are strongly correlated and gaps are short relative to how fast the signal typically changes.

Linear Interpolation — Between Forward Fill and Modeling

df['temperature'] = df['temperature'].interpolate(method='linear')

# Time-aware interpolation respects unevenly spaced timestamps
df = df.set_index('timestamp')
df['temperature'] = df['temperature'].interpolate(method='time')

Interpolation draws a straight line (or a more complex spline, with method='spline' or method='polynomial') between the last known value and the next known value, which usually beats forward fill for continuous physical measurements because it captures the trend across the gap instead of freezing it. It is a poor fit for categorical data or for series with abrupt regime changes, like a machine that switches states rather than drifting smoothly.

---

Strategy 4: ML-Based Imputation (Advanced)

KNN Imputation

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
df_imputed = pd.DataFrame(
    imputer.fit_transform(df),
    columns=df.columns
)

How it works: For each row with a missing value, KNN Imputation finds the five most similar rows based on the other, non-missing features (using Euclidean distance by default) and fills the gap with a weighted average of their values.

Pros: Uses relationships between all features simultaneously, which typically produces noticeably more accurate imputations than a single global or group-level statistic, especially when features are correlated with each other.

Cons: Computationally expensive at scale — the distance calculation is roughly quadratic in the number of rows unless you use an approximate nearest-neighbor structure — and sensitive to feature scaling. Features on different scales (say, age in years and salary in dollars) will let the larger-magnitude column dominate the distance metric unless you standardize first.

When to use: Medium-sized datasets (roughly tens of thousands of rows, not tens of millions) where features are meaningfully correlated and accuracy matters more than raw speed.

Iterative Imputation / MICE (Most Statistically Grounded)

from sklearn.experimental import enable_iterative_imputer  # noqa
from sklearn.impute import IterativeImputer
from sklearn.ensemble import RandomForestRegressor

imputer = IterativeImputer(
    estimator=RandomForestRegressor(n_estimators=100),
    max_iter=10,
    random_state=42
)
df_imputed = pd.DataFrame(
    imputer.fit_transform(df),
    columns=df.columns
)

This is the approach with the deepest statistical pedigree in the toolkit. Scikit-learn's IterativeImputer models each column with missing values as a function of all the other columns, iterating in a round-robin fashion until the imputed values converge, and its design was explicitly inspired by the mice package in R — Multivariate Imputation by Chained Equations — first published by Stef van Buuren and Karin Groothuis-Oudshoorn and formalized in their widely cited 2011 paper in the Journal of Statistical Software (van Buuren & Groothuis-Oudshoorn, 2011, JSS). The mice R package itself traces back to an S-PLUS library released in 2000, making chained-equation imputation one of the longest-running, most battle-tested approaches in applied statistics.

There is an important subtlety scikit-learn documents explicitly: IterativeImputer by default returns a single, deterministic imputation. True multiple imputation — running the process several times with different random seeds to produce several plausible completed datasets, then pooling the results — requires setting sample_posterior=True and repeating the fit with different random_state values (scikit-learn IterativeImputer documentation). Multiple imputation is preferable whenever you need honest uncertainty estimates, because a single imputed value pretends to know something you do not actually know, while a pooled set of five or ten imputations propagates that uncertainty into your final standard errors.

Pros: Because it conditions each missing column on every other column using a real regression or classification model, it generally produces the most accurate imputations of any method covered here, and it is the standard against which other quick-fix approaches get benchmarked in the missing-data literature.

Cons: Slowest approach by a wide margin, especially with a complex estimator like a random forest and a high max_iter. It can also be unstable on datasets with very high missingness rates or highly non-linear relationships, where the round-robin regressions struggle to converge.

When to use: Whenever imputation accuracy genuinely matters for the downstream decision — a medical dataset, a financial risk model, an academic research dataset that will be scrutinized during peer review — and the dataset is small enough that the runtime cost is tolerable.

Deep Learning Imputation (GAIN and Autoencoders)

For very high-dimensional or image-like data, neural approaches such as Generative Adversarial Imputation Nets (GAIN) and denoising autoencoders learn a latent representation of the full data distribution and use it to reconstruct missing entries. These methods can outperform KNN and iterative imputation on complex, high-dimensional tabular data, but they require substantially more data to train reliably and are harder to explain to a non-technical stakeholder — a real cost in regulated industries where imputation logic sometimes needs to be auditable. For most tabular business datasets under a few hundred thousand rows, iterative imputation with a tree-based estimator captures most of the achievable accuracy without the added complexity.

---

Case Study 1: Electronic Health Records and MNAR Lab Values

Clinical datasets are a textbook example of why the MCAR/MAR/MNAR distinction is not academic. In electronic health records, a lab test is often ordered because a clinician already suspects an abnormal result — meaning the decision to measure a value is correlated with the value itself. A patient's blood glucose is far more likely to be tested if they show symptoms consistent with diabetes. That makes the missingness pattern MNAR by construction: healthier patients are systematically less likely to have the measurement at all.

Naively applying mean or KNN imputation in this setting silently assumes the untested patients look statistically similar to the tested ones, which understates disease prevalence in the imputed data and can bias any model trained on it toward under-flagging risk in patients who were never tested in the first place. The standard mitigation in clinical informatics is to encode "was this lab ever ordered" as its own feature (the missingness-flag trick from Strategy 2 above) so the model can learn from the decision to test as a signal in its own right, separate from whatever value gets imputed for patients who were not tested.

Case Study 2: Time-Series Sensor Networks in Industrial IoT

Manufacturing plants and utility grids depend on networks of thousands of physical sensors, and gaps are routine — a transmitter briefly loses connectivity, a device reboots after a firmware update, or a maintenance window takes a sensor offline for an hour. In this setting, missingness is close to genuinely MCAR (network hiccups are not correlated with the physical quantity being measured), which makes forward fill and time-aware linear interpolation reasonable defaults for gaps measured in minutes.

The failure mode here is different from the clinical case: it is treating a long gap the same as a short one. A five-minute sensor dropout during stable operating conditions is safely forward-filled. A six-hour dropout during a known equipment fault is not — the value genuinely changed during that window, and forward-filling it manufactures a flat, misleading signal that can mask exactly the anomaly a predictive maintenance model is supposed to catch. Production IoT pipelines commonly cap forward fill at a maximum gap length (for example, fill gaps up to 15 minutes, flag or drop anything longer) rather than filling indiscriminately.

Case Study 3: Survey Non-Response in Social Science Research

Large-scale social surveys, from national census supplements to academic panel studies, routinely see 10 to 30 percent item non-response on sensitive questions like income, drug use, or political affiliation. This is the classic use case that motivated the original development of multiple imputation methodology. Rather than a single best-guess fill, survey statisticians generate several plausible completed datasets (typically five to twenty, following recommendations refined since Rubin's original work), run the planned analysis on each one separately, then pool the resulting estimates and standard errors using Rubin's combination rules. The pooled standard error is deliberately larger than what you would get from a single imputed dataset, because it honestly reflects the extra uncertainty introduced by not actually knowing the missing values. This is the methodological ancestor of what IterativeImputer(sample_posterior=True) approximates in a machine learning pipeline, and it is worth understanding even if your day-to-day work stays inside scikit-learn, because it explains why a single deterministic imputation is a simplification, not the full statistical picture.

---

Real Example: End-to-End Pipeline

import pandas as pd
import numpy as np
from sklearn.impute import KNNImputer

# Simulate a messy dataset
np.random.seed(42)
df = pd.DataFrame({
    'age': [25, np.nan, 35, 40, np.nan, 28, 55, np.nan],
    'salary': [50000, 60000, np.nan, 80000, 90000, np.nan, 120000, 75000],
    'department': ['Eng', 'Eng', 'Mkt', 'Mkt', np.nan, 'Eng', 'Mkt', 'Eng']
})

# Step 1: Inspect
print("Missing values:")
print(df.isnull().sum())

# Step 2: Categorical — fill with mode
df['department'] = df['department'].fillna(df['department'].mode()[0])

# Step 3: Create "was missing" flags for important columns
df['salary_was_missing'] = df['salary'].isnull().astype(int)

# Step 4: Numeric — group-based median for salary
df['salary'] = df.groupby('department')['salary'].transform(
    lambda x: x.fillna(x.median())
)

# Step 5: KNN for age (uses the salary relationship)
knn = KNNImputer(n_neighbors=3)
df[['age', 'salary']] = knn.fit_transform(df[['age', 'salary']])

print("\nClean dataset:")
print(df)

This pipeline layers strategies deliberately: mode for the low-cardinality categorical column, a group-aware statistic for salary, an explicit missingness flag to preserve the MNAR-relevant signal, and KNN for age because it correlates with salary. Stacking simple and advanced methods by column, rather than applying one blanket strategy to the whole DataFrame, is standard practice in production feature-engineering pipelines.

---

Decision Guide

Situation	Recommended strategy	Why
Time series with short gaps	Forward fill or time-aware interpolation	Consecutive values are correlated; recent history is the best available estimate
Column >50% missing	Delete column, consider a _was_missing flag first	Imputation is mostly guessing beyond this threshold
<5% missing rows, large dataset, MCAR	Delete rows (listwise deletion)	Minimal information loss, no assumptions required
Categorical column, low cardinality	Mode imputation or group-based mode	Fast, interpretable, works well when one category dominates
Numeric column with outliers or skew	Median imputation (or group median)	Robust to extreme values that would distort a mean
Features are correlated with each other	KNN imputation	Uses cross-feature relationships a single statistic cannot capture
Accuracy is critical, dataset is small-to-medium	Iterative imputation (MICE-style)	Most statistically grounded approach, models each column conditionally
Need honest uncertainty estimates	Multiple imputation, pooled per Rubin's rules	Single imputed values understate uncertainty; pooling propagates it correctly
Suspected MNAR	Missingness flag + domain-specific correction	No generic imputer fixes bias caused by the value itself driving the gap

Table 1. Decision matrix for missing-data strategies, matched to data type, missingness mechanism, and accuracy requirements.

---

Common Mistakes to Avoid

1. Imputing before the train/test split. Always split first, then fit the imputer on the training data only. Fitting on the full dataset lets statistics from the test set — means, medians, nearest neighbors — leak into training, which inflates your validation metrics and produces a model that looks better offline than it will perform in production.

from sklearn.model_selection import train_test_split

X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)

imputer = KNNImputer()
X_train_clean = imputer.fit_transform(X_train)  # Fit on train only
X_test_clean = imputer.transform(X_test)         # Transform test with train statistics

2. Ignoring the "why" behind the missingness. If data is MNAR, no imputation strategy fully removes the resulting bias, because the information required to correct it is the same information that caused the gap. Document the limitation explicitly in your model card or analysis write-up rather than presenting imputed MNAR data as equivalent to observed data.

3. Using the mean on skewed data. Income, housing prices, transaction amounts, and age at diagnosis are commonly right-skewed. Always plot the distribution — a simple histogram takes one line of code — before deciding between mean and median.

4. Treating all gaps in a time series the same way. A ten-minute sensor dropout and a ten-hour outage are not the same problem, and filling both with the same forward-fill logic can hide exactly the anomaly a monitoring system is meant to catch. Cap the maximum gap length you are willing to fill automatically, and flag longer gaps for manual review or a different treatment entirely.

5. Forgetting that imputation is not free of assumptions. Every fill strategy encodes a belief about what the missing value probably was. Mean imputation assumes "probably typical." Forward fill assumes "probably unchanged." KNN assumes "probably similar to comparable rows." Naming that assumption out loud, even in a code comment, makes it much easier to catch when it is wrong for your specific dataset.

6. Applying the same imputer object to unrelated batches of new data without re-checking distributions. A KNNImputer or IterativeImputer fit on data from one time period can degrade silently if the underlying data distribution shifts (a phenomenon known as data drift). Periodically re-validate imputation quality on fresh data rather than assuming a fitted imputer stays accurate indefinitely.

---

FAQ

Q: Which imputation strategy is best overall? A: There is no single best strategy — it depends on the missingness mechanism and the data type. Default to median for skewed numeric columns and mode for low-cardinality categorical columns. Reach for KNN or iterative imputation when features are meaningfully correlated and the dataset is under roughly 100,000 rows, since both methods get computationally expensive at larger scale.

Q: Should I impute before or after the train/test split? A: Always after the split. Fit the imputer on the training data only, then use that same fitted imputer to transform both the training and test sets. Fitting on the full dataset before splitting leaks test-set information into training and inflates your evaluation metrics.

Q: What should I do if the data is not missing at random (MNAR)? A: Create an explicit is_missing binary feature to capture the pattern, then impute a value for the underlying column so the model can still use it. No generic imputation method fully corrects for MNAR bias, because the information that would let you predict the missing value accurately is the same information whose absence caused the gap in the first place.

Q: How much missing data is too much to impute? A: There is no universal number, but a widely used rule of thumb treats columns above roughly 50 percent missing as unreliable for standard imputation. Beyond that threshold, decide whether the column carries enough unique signal to justify keeping it (perhaps only as a missingness flag) or whether it should be dropped entirely.

Q: What is the difference between single imputation and multiple imputation? A: Single imputation, like filling with a median or a single KNN pass, replaces each missing value with one best guess and then proceeds as though that guess were the true value. Multiple imputation generates several plausible completed datasets, analyzes each one, and pools the results using Rubin's combination rules, which produces standard errors that honestly reflect the uncertainty introduced by the missing data rather than understating it (van Buuren & Groothuis-Oudshoorn, 2011, Journal of Statistical Software).

Q: Can forward fill work for anything other than time series? A: It can technically run on any ordered column, but it only makes statistical sense when adjacent rows are genuinely related, which is almost always a property of time-ordered or spatially ordered data. Applying forward fill to a randomly shuffled dataset just propagates whatever value happens to sit above the gap, with no underlying justification.

Q: Does KNN imputation require scaling my features first? A: Yes, in almost all cases. KNN Imputation relies on a distance metric, typically Euclidean distance, and features with larger numeric ranges will dominate that distance calculation unless every column is standardized or normalized first. Skipping this step is one of the most common reasons KNN imputation underperforms a simpler group-based median.

Q: Is iterative imputation the same as the R mice package? A: They share the same conceptual foundation — modeling each incomplete column as a function of the others in a round-robin fashion — and scikit-learn's IterativeImputer explicitly credits mice as its inspiration. They differ in default behavior: mice was built from the ground up for multiple imputation with pooled uncertainty, while IterativeImputer returns a single imputation by default unless you explicitly set sample_posterior=True and repeat the process with different random seeds.

Q: Should I always add a "was missing" flag column? A: Not always, but it is a low-cost, high-value habit for any column where you suspect the missingness itself carries information (a MAR or MNAR situation), such as income, medical test results, or optional survey fields. For genuinely MCAR columns like a random sensor dropout, the flag usually adds noise rather than signal, so it is worth checking correlation with your target variable before keeping it.

Q: How do I choose the number of neighbors for KNN imputation, or the number of iterations for iterative imputation? A: For KNNImputer, n_neighbors between 3 and 10 is a common starting range, with smaller datasets favoring the lower end to avoid diluting the estimate with dissimilar rows. For IterativeImputer, max_iter=10 is the scikit-learn default and is usually sufficient for convergence, but you can monitor convergence directly by tracking how much the imputed values change between rounds and stopping once that change falls below a small threshold.

---

Related Articles

• Data Cleaning: 80% of Data Scientist's Job
• Best Data Science Tools — 2026 Tech Stack
• Python vs R for Data Science

---

The Takeaway

Missing data is not a single problem with a single fix — it is a family of related problems that all demand the same first step: figure out why the data is missing before deciding how to fill it. MCAR data tolerates simple deletion or basic statistics with minimal consequence, MAR data rewards conditional and model-based imputation like group medians, KNN, or iterative imputation, and MNAR data resists full correction no matter which tool you reach for, which is exactly why the missingness-flag habit exists. The methods with the longest statistical track record, multiple imputation and its chained-equations implementation in mice and IterativeImputer, exist specifically to keep your uncertainty estimates honest rather than pretending a filled-in value is as good as a real one. Default to the simplest defensible method for your data type, document every assumption you make along the way, and always fit your imputer on training data only.