What Data Cleaning Actually Involves

Data cleaning is the process of finding and correcting errors, inconsistencies, and gaps in a dataset before it's used for analysis or modeling. It's a well-documented pattern across industry surveys of data scientists that this preparation work consumes a large share of total project time, often more than the modeling itself, though exact percentages vary by survey and role. What follows is a look at what that work actually consists of, why it changes results rather than merely tidying them, and where the honest limits of "fixing" data sit.

Why cleaning eats so much of a project's time

Ask a data scientist what they spend most of their week doing, and the answer is rarely "building models." It's reconciling three different spellings of the same city name, chasing down why a sensor logged negative temperatures in Celsius when the range should be zero to 40, or figuring out whether a blank field means "zero," "not applicable," or "we forgot to ask."

This pattern shows up consistently, if inexactly, across the industry. Surveys of working data scientists — including practitioner polls run by analytics vendors and professional associations over the past decade — have repeatedly found that respondents report spending a disproportionate share of project hours on data preparation relative to modeling and analysis. The specific numbers in these surveys differ depending on methodology, sample, and how "cleaning" is defined, so any single figure should be treated as an approximation of a broader, well-documented pattern rather than a fixed constant. What's consistent across nearly every account, whether from a Kaggle community poll, a corporate data-team retrospective, or an academic case study, is the direction of the finding: preparation dominates.

The reason isn't laziness or inefficiency. Raw data reflects the messiness of how it was collected, not the tidiness analysts would prefer. A single company might pull customer records from a point-of-sale system, an email marketing platform, and a support ticketing tool, each of which stores names, dates, and identifiers in its own format. Merging those sources without reconciling the differences produces a dataset that looks complete but is quietly wrong in ways that only surface once someone tries to count something precisely.

There's also a compounding effect. Cleaning one column often reveals problems in another. Standardizing a "country" field might expose that the "currency" field wasn't adjusted to match, since a record listed as "Japan" but priced in dollars suggests the currency was left at a default value rather than corrected during entry. Each fix opens a new question, which is part of why the work rarely wraps up as quickly as project timelines assume.

The four core categories of data quality problems

Nearly every data cleaning task falls into one of four broad categories, and recognizing which one you're facing changes the fix. These categories map loosely onto the data-quality dimensions long used in information-systems research: accuracy, completeness, consistency, and timeliness, a taxonomy that traces back to foundational data-quality management literature from the 1990s and remains standard in data governance frameworks today.

Missing values are the most familiar problem: a field that should contain a value but doesn't. A survey respondent skips the income question, a sensor loses connectivity for an hour, a form field is optional and left blank. How consequential this is depends heavily on why the value is missing, not just how many are missing, a distinction covered in detail below.

Inconsistent formatting covers cases where the same real-world fact is represented differently across records. Dates written as "03/04/2024" in one system and "2024-04-03" in another look similar but parse to different calendar days depending on whether the format is day-first or year-first. State abbreviations, currency symbols, capitalization of category labels, and unit systems (kilometers versus miles, Celsius versus Fahrenheit) all fall into this bucket.

Duplicate records happen when the same real-world entity appears more than once, usually with small variations, a customer signing up twice with a middle initial added the second time, or a product entered separately by two warehouses using slightly different SKU formats. Duplicates inflate counts, distort averages, and can cause a machine learning model to overweight whatever pattern the duplicated rows represent.

Encoding and structural errors are the least glamorous but often the most disruptive: a text field that should contain a number, a date stored as a string that sorts alphabetically instead of chronologically, a character-encoding mismatch that turns an accented name into garbled symbols, or a column that silently shifted because a delimiter appeared inside a text field during a CSV export. These errors are easy to miss because the data often still "looks" plausible at a glance.

The four data-quality dimensions most commonly used in data governance — accuracy, completeness, consistency, and timeliness — originated in information-systems research from the 1990s and remain the standard vocabulary analysts use to categorize what's wrong with a dataset before deciding how to fix it.

A structured view of common cleaning tasks

Problem type	Typical cause	Common handling approach
Missing values	Optional form fields, sensor dropouts, respondent non-response	Imputation, listwise deletion, or explicit "missing" flagging
Duplicate records	Repeated sign-ups, merged data sources, re-scraped datasets	Fuzzy matching and deduplication rules
Inconsistent formatting/units	Multiple source systems, regional defaults, manual entry	Standardization to one controlled format or unit
Outliers / implausible values	Data-entry typos, unit mismatches, genuine rare events	Validation rules, manual review, robust statistical methods
Encoding errors	Character-set mismatches, delimiter collisions, type coercion	Re-parsing with correct encoding, schema validation

Each row represents a distinct failure mode with a distinct fix; treating them interchangeably is one of the more common mistakes in early-career data work.

Missing data is a modeling decision, not a cleanup step

The instinct when facing missing values is to either delete the incomplete rows or fill the gap with something reasonable, like the column average. Both choices are legitimate in some circumstances and quietly damaging in others, and the difference comes down to why the data is missing, not how much is missing.

Statistician Donald Rubin formalized this distinction in the 1970s with a framework still taught in applied statistics courses today. Missing completely at random (MCAR) means the chance a value is absent has nothing to do with the value itself or with any other variable in the dataset, a rare and convenient case where deleting incomplete rows introduces no systematic bias, only a loss of sample size. Missing at random (MAR) means the missingness is related to other observed variables but not to the missing value itself, for instance, older survey respondents being less likely to fill in an optional email field regardless of what that email would have been. Missing not at random (MNAR) is the most dangerous case: the absence is tied to the very value that's missing, such as people with very high incomes being less likely to report their income at all. Deleting rows or naively imputing values under MNAR conditions can systematically bias results in a direction that's hard to detect after the fact.

Given that framework, three broad handling strategies come up repeatedly in practice, each with real tradeoffs rather than a clear winner.

Listwise deletion removes any row with a missing value in a field the analysis needs. It's simple and doesn't introduce artificial values, but it throws away every other piece of information in that row, and if missingness correlates with anything meaningful, it introduces bias while looking clean. A customer database where high-value clients were onboarded manually by a sales team, and therefore have more incomplete self-reported fields, would produce a systematically skewed analysis if those rows were simply dropped.

Single imputation, most commonly mean or median imputation, fills every gap with one estimated value. It preserves sample size and is computationally trivial, but it has a well-documented side effect: every imputed value sits exactly at the average, artificially shrinking the reported variability of the dataset and, in some cases, distorting the relationships between variables that a later model is supposed to learn.

Multiple imputation generates several plausible estimates for each missing value, based on relationships with other observed variables, analyzes each completed version of the dataset separately, then pools the results together in a way that reflects the genuine uncertainty about what the missing value might have been. Rubin's 1987 book on multiple imputation for survey nonresponse remains a standard reference for this approach, and it's generally considered more statistically honest than single imputation, though it requires more computation and more careful setup than filling in an average.

Why can't messy data just be deleted instead of cleaned?

Deletion is tempting because it's fast, but it only avoids bias when missingness is unrelated to anything of interest in the dataset, the MCAR case described above. In most real datasets, missingness correlates with something, a customer segment, a time period, a device type, which means deleting those rows changes what the remaining sample represents, not just how big it is.

How duplicate detection actually works

Exact duplicates, where every field matches identically, are trivial to remove with a simple deduplication rule. The harder and far more common case is the near-duplicate: two records that represent the same real-world entity but differ slightly because of a typo, a formatting difference, or a partial update.

Fuzzy matching techniques compare records based on similarity rather than exact equality, using methods like edit distance (how many character changes separate two strings), phonetic matching (do two names sound alike even if spelled differently), or matching on a combination of fields, such as name plus birth date plus zip code, when no single field is reliable alone. These methods reduce false negatives (missed duplicates) but introduce a new risk: false positives, where two genuinely different people or products get merged into one because they happened to look similar on the fields being compared.

This tradeoff means deduplication thresholds are a judgment call, not a fixed formula. A hospital record system merging two patients who share a similar name and birth year is a far more serious error than an e-commerce system merging two entries for the same T-shirt listed in slightly different colors. The acceptable error rate depends entirely on what happens downstream if the match is wrong, which is why deduplication rules should be documented and reviewed rather than applied as a one-time script and forgotten.

Outliers, unit mismatches, and the problem of implausible values

An outlier is a value that sits far from the rest of the distribution, and one of the trickiest parts of cleaning is deciding whether that value represents a genuine rare event or an error that needs correcting. A recorded human age of 200 years is almost certainly a data-entry error. A recorded income of ten times the median in a sample of company executives might be entirely real. Treating both the same way, either by automatically capping extreme values or automatically trusting every recorded number, produces different but equally serious problems.

Unit mismatches are a specific and common source of implausible values. A shipment weight recorded in pounds inside a column expecting kilograms will look like an outlier, not an error, unless someone checks the units against the source system. A temperature reading of 98.6 sitting in a column of otherwise Celsius values is a strong clue that a subset of records came from a source using Fahrenheit. These errors are especially dangerous because the values often still fall within a technically plausible range, they just represent a different scale than the rest of the column, which means simple range-check validation won't catch them.

Robust statistical methods, ones less sensitive to extreme values than a simple mean, are often used alongside manual review rather than as a replacement for it. A deeper look at how a small number of extreme values can distort a summary statistic is covered in Why Outliers Can Change Averages, which walks through the mechanics of why a single very large or very small number can pull a mean far from what most of the data actually looks like.

Why cleaning decisions change the analysis, not just tidy it

Every choice made during cleaning is, in effect, a modeling assumption made before the model exists. Deciding to drop rows with missing values assumes those rows aren't systematically different from the rows that remain. Deciding to impute a missing value with a column average assumes the true distribution of that variable is well represented by its center, an assumption that quietly fails whenever the variable is skewed or clustered.

This matters because a model trained on cleaned data doesn't announce which assumptions were baked in during preparation. It produces confident output regardless of whether the underlying assumptions held. A retailer's demand forecast built on data where returning customers were accidentally counted twice due to a failed deduplication step will look internally consistent even though its baseline is inflated. A hospital readmission model trained on a dataset where records with missing follow-up visits were simply dropped, and those missing follow-ups correlate with patients who couldn't afford continued care, will underestimate risk for exactly the population it most needs to flag. The failure isn't loud. It's a quietly biased answer that looks as confident as a correct one.

This is also why data cleaning connects directly to broader questions about how conclusions drawn from data can mislead even when every individual number is technically accurate. How Survivorship Bias Hides Failures covers a related failure mode, where the data that's missing entirely from a dataset, not just incomplete within it, shapes conclusions in ways that are easy to overlook. And once a cleaned dataset is used to produce an estimate, understanding how much uncertainty that estimate carries becomes its own question, one addressed in What A Confidence Interval Really Means.

What data cleaning cannot fix

It's worth being direct about the limits here. Cleaning can correct known errors, standardize formats, and make reasonable, documented decisions about ambiguous cases. It cannot recover information that was never collected in the first place, and it cannot fully undo a biased collection process.

If a survey was only distributed to people with reliable internet access, no amount of downstream cleaning will make the resulting dataset representative of people without it. If a company's sales records only exist for transactions that were logged correctly, and a subset of transactions failed to log due to a system bug correlated with a specific store location, cleaning the dataset that remains cannot recreate the transactions that vanished. Data cleaning fixes what's visibly wrong within the data that exists. It's not a substitute for a sound data collection design, and treating it as one is a common source of analyses that look rigorous on the surface but rest on an uncorrectable gap underneath.

The takeaway

Data cleaning is not a chore that happens before the real analytical work starts, it's analytical work in its own right, because every choice about how to handle a gap, a duplicate, or an inconsistent label changes what the final numbers actually represent. The time practitioners spend here, consistently reported across surveys as a disproportionate share of total project effort even if the exact figure varies, reflects how much of an analysis's accuracy is decided before a single model gets trained. Treating cleaning decisions as documented, revisitable choices, rather than a one-time script run at the start of a project, is what separates a dataset that's merely tidy from one that's actually trustworthy.

---

Read next:

• Why Outliers Can Change Averages
• How Survivorship Bias Hides Failures
• What A Confidence Interval Really Means

FAQ

Q: Why do data scientists spend so much time cleaning data?

Real-world data is collected from multiple systems, people, and time periods, which introduces inconsistent formats, missing values, and duplicate records that have to be reconciled before analysis is reliable. Industry surveys of practitioners consistently report that data preparation consumes a disproportionate share of total project time relative to modeling, though the exact percentage varies by survey and methodology.

Q: What is the difference between data missing at random and data missing not at random?

Data missing at random means the chance of a value being absent relates to other observed variables but not to the missing value itself. Data missing not at random means the absence is tied to the value that would have been recorded, such as high earners being less likely to report income, which can seriously bias results if handled carelessly.

Q: Is it ever acceptable to just delete rows with missing data?

It's acceptable when the missing data is a small fraction of the dataset and appears to be missing completely at random, meaning its absence has nothing to do with any variable in the data. Deletion becomes risky when missingness concentrates in a specific group, since removing those rows changes what the remaining sample represents.

Q: What is multiple imputation and why is it considered better than simple averaging?

Multiple imputation generates several plausible estimates for each missing value based on relationships with other variables, analyzes each completed version separately, then pools the results to reflect genuine uncertainty. Unlike filling every gap with a single average, it avoids artificially shrinking the dataset's natural variability.

Q: How is a data-entry error different from a genuine outlier?

A data-entry error is an implausible value caused by a mistake, like a unit mismatch or mistyped digit, while a genuine outlier is an accurate but extreme value that legitimately belongs in the dataset. Distinguishing the two usually requires checking the value against its source system or domain knowledge rather than relying on a statistical rule alone.

Q: Can data cleaning fix a biased dataset?

No. Cleaning can correct known errors and standardize formats within the data that was collected, but it cannot recover information that was never captured or reverse a biased collection process, such as a survey that only reached a specific subgroup. Fixing that kind of gap requires redesigning how the data is collected, not further cleaning of what already exists.