Data cleaning : n°1 pain point for data scientists
Data cleaning is one of the biggest bottlenecks in data science. In Anaconda’s State of Data Science 2020, respondents report spending 45% of their time just loading (19%) and cleaning (26%) data, before modeling even starts. [1]
It is also widely seen as excessive. The CrowdFlower Data Scientist Report 2016 found that 3 out of 5 data scientists spend most of their time cleaning and organizing data, and nearly 60% say they spend too much time on it. [2]
And the cost is not just internal frustration. Gartner cites research estimating that poor data quality costs organizations at least $12.9M per year on average, while 59% of organizations do not measure data quality at all. [3]
Yet despite the time and money involved, teams often do not even agree on what “data cleaning” means. Many reduce it to formatting, filling nulls, and removing duplicates. That creates a dangerous illusion, because data can look clean while drifting further away from reality.
This post is the first in a 3 part series. Here we define data cleaning from first principles, explain why it exists, clarify who must be involved, and surface the misconceptions that quietly break projects. The next article will focus on best practices, the final article will walk through a concrete case study.
It is less about the data, more about the reality
Data science creates value by capturing patterns in data that reflect real patterns in the world.
Reality is the target. Customers, systems, business processes, physical devices, and human behavior are what we are trying to understand.
Data is only the medium that carries traces of that reality into an information system. Models and analytics are tools, their validity depends on whether the data is a faithful proxy for the real system.
That leads to a definition if data cleaning from its reason to exists.
Data cleaning is the activity whose objective is to bring data closer to reality by identifying and correcting distortions, biases, or corruptions introduced by data collection and data processing systems.
Notice what this definition does.
First, it is model agnostic. You are not cleaning for a specific algorithm. You are cleaning so that the dataset represents the real world.
Second, the reference point is reality, not convenience. Clean data is not data that looks tidy or easy to query. It is data that correctly describes what happened, as far as you can know.
Data cleaning is important because it allows data science algorithms to capture patterns of the real world instead of bugs and artifacts of the information system.
Why data cleaning is often necessary
We need data cleaning when the mapping from reality to data is imperfect.
Distortions happen for two main reasons.
The first reason is technical. Data collecting and processing pipelines could be imperfects, obsoletes or simply operational mistakes.
Systems drop events. Logs change format. Pipelines join tables on the wrong key. Time zones shift. A field that used to mean “delivery date” becomes “planned delivery date” after a product update, and nobody tells the analytics team.
The second reason is structural. Data is also often reused. It was collected for billing, not for churn prediction. It was collected for compliance, not for operational optimization. The same dataset is asked to answer questions it was never designed to answer.
More often than not, new data science project works on a new use case of the data. In the business point of view, before the use case is proven, it is hard to justify investing in a new data collection pipeline.
In short, data cleaning is required when there are important distortions between the data and the reality.
Data cleaning is a team sport
Data scientists can spot anomalies in the data. But deciding whether an anomaly reflects real behavior or a data artifact usually requires going upstream, into the systems that produced the data, and talking to the people who run the process.
Most intuition about reality lives in people. Operations knows what really happens when things are busy. Product knows what changed in the last release. Finance knows the edge cases that trigger reversals. Customer support knows why a category suddenly spikes.
Take a churn project. During exploration, you see a churn spike on the 5th of every month. Is it real or is it a data issue? The dataset alone cannot tell you. It could be an IT artifact, for example a monthly batch update. Or it could be real behavior, for example customers leaving right after billing. You need system context and business knowledge to decide.
Good data cleaning is a team effort.
Data teams explore the dataset, quantify issues, and test hypotheses.
Business stakeholders provide the map of what reality should look like, what is plausible, and what matters.
IT and engineering teams understand how data is collected and transformed.
That also makes cleaning iteration is much longer than modeling since it requires access to other teams and systems.
Why data cleaning is hard
A key reason data cleaning is hard is that you cannot do it alone.
When an organization treats data cleaning as “the data team’s job,” it puts data scientists in a role they cannot fully do. They do not own the source systems. They do not control the pipelines. They cannot validate reality alone. The result is wasted time and quiet assumptions. You get a dataset that looks consistent, but is still wrong.
This is why data cleaning feels like such a pain. It is not mainly technical. It is organizational and mental. You keep hitting issues you cannot resolve by yourself.
Finding an anomaly is the easy part. The hard part is getting answers. You have to track down the right people, dig through outdated documentation, and wait for new extracts. Then the new extract arrives and reveals a new issue. That loop drains energy, slows projects, and hurts productivity.
Four misconceptions that keep breaking data science projects
1. Cleaning is a mechanical formatting task.
Formatting and types helps technical implementations, but projects fail due to distortions and structural biases.
True data cleaning should work on patterns and hypotheses, not just simply format data.
2. Clean data is data that looks clean.
The reality is often messy. Trying to get a tidy, academic dataset can lead to destructive transformations that distort the data further from the reality.
Dropping missing values, replacing outliers should not be automatic. The consequence can be biased modeling, wrong insights, bad evaluation.
The truth don’t care about looking nice.
3. Cleaning is the data scientist’s job.
Good data cleaning requires knowledge and skills across teams. Treating cleaning as a solo activity leads to incorrect assumptions and fragile outcomes.
Having a cross functional squad with clear organization will greatly improve quality and efficiency of the project.
4. Data cleaning and data preprocessing are the same.
Cleaning is getting data closer to the truth.
Preprocessing is making data works well with an algorithm (logistic regression, neural network, etc).
Mixing them, in best cases, create confusion and inefficiency; in worst case, generate wrong insights and bad modeling.
Separate clearly 2 phases: data cleaning to get closer to reality, preprocessing and feature engineering bring data closer to algorithms.
Conclusion, the point of cleaning is trust
Data cleaning is not about making data tidy. It is about restoring the connection between data and the real world you are trying to understand.
If you take that seriously, a few implications follow. You stop treating missing values as a default to fill. You stop deleting “ugly” rows without asking what they represent. You involve business stakeholders earlier, because reality lives outside the dataset. You document decisions, because cleaning is a chain of reasoning, not a one time script.
In the next article, we move from first principles to practice. We will cover the best habits that make data cleaning better, faster and more enjoyable, how exploration drives the investigation, how to validate changes without false confidence, and how documentation should speed up cleaning instead of coming at the end.
Sources
[1] Anaconda, State of Data Science 2020, time spent on data loading and data cleaning.
[2] CrowdFlower, Data Scientist Report 2016, time spent cleaning and organizing data.
[3] Gartner, “Data Quality” topic page, estimated cost of poor data quality and share of organizations not measuring data quality.