Session 3: Data Quality

Learning goals¶

• Data Quality Dimensions

• Data Quality Checks

• Data Profiling

• Missing Values

  • Realistic Missingness Patterns

  • Imputation

• Outlier detection

• Redaction

Why Data Quality¶

• Poor data quality costs the US economy ~$3.1 trillion/year (IBM, 2016)

  • Causes: Wasted time, Operational inefficiencies, Trust issues

• ML models inherit and amplify data defects Sculley et al. (2015)

• Legal Perspective (personal rights / privacy)

  • GDPR Regulation (EU) 2016/679 of the European Parliament and of the Council (2016) (EU)

  • HIPAA Health Insurance Portability and Accountability Act of 1996 (1996) (US)

• Data Quality Standards

  • ISO 8000 mandate International Organization for Standardization (ISO) (2022)

  • Artificial intelligence — Data quality for analytics and machine learning (ML) International Organization for Standardization (2023)

  • Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Measurement of data quality International Organization for Standardization (2015)

• Data Quality is difficult to automate (Biessmann et al. (2021) and Schelter et al. (2018))

GDPR and HIPAA¶

ISO Data Quality Standards¶

• ISO/IEC 25024:2015 – Data Quality Measurement International Organization for Standardization (2015)

  • Framework to quantitatively measure various aspects of data quality

    • Accuracy, completeness, consistency, and timeliness

• ISO 8000 Series – Data Quality Management and Exchange International Organization for Standardization (ISO) (2022)

  • Requirements for managing and exchanging master data to improve data quality across organizations.

  • Emphasizes the importance of data quality in enhancing business processes and decision-making.

• ISO/IEC 5259 Series – Data Quality for Analytics and Machine Learning (ML) International Organization for Standardization (2023)

  • Focuses on ensuring data quality for AI and ML applications

  • Provides frameworks and measurable characteristics to help organizations effectively manage their data quality in analytical workflows

ISO/IEC 25024:2015 – Data Quality Measurement¶

Example: Consider a company that collects customer data for its marketing campaigns. Using ISO/IEC 25024:2015, the company can measure:

• Accuracy: By comparing customer addresses in their database against a reliable external database (e.g., postal service data), they can determine the percentage of addresses that are valid.

• Completeness: The company can calculate the percentage of customer records that have complete information, such as name, address, email, and phone number. If they find that only 70% of records have a valid phone number, they can work on strategies to gather this missing data.

• Timeliness: The company may assess how current their data is by checking the age of records. If most records are over two years old, they may need to implement a regular update process to ensure data remains relevant.

ISO 8000 Series – Data Quality Management and Exchange¶

Example: A manufacturing company that manages a complex supply chain can apply the ISO 8000 standards to improve the quality of their product master data:

• Data Provenance: The company can implement tracking for each product’s origin, including details about the supplier, production date, and batch number. This ensures that all stakeholders have access to accurate historical data, which is critical for quality control and compliance.

• Interoperability: When exchanging data with suppliers and customers, the company can adopt standardized formats as per ISO 8000. For instance, using a standardized product identifier (like GTIN) allows seamless integration of data between their systems and those of suppliers, reducing errors in product information.

• Accuracy and Consistency: They can regularly audit their product data against sales records and inventory systems, ensuring that product descriptions, specifications, and prices are consistent across all platforms. If discrepancies are found, they can initiate corrective actions to maintain high data quality standards.

ISO/IEC 5259 Series – Data Quality for Analytics and ML¶

Example: A financial institution uses machine learning algorithms to detect fraudulent transactions. Applying the ISO/IEC 5259 standards, the institution can:

• Establish a Data Quality Model: Define what constitutes high-quality transaction data (e.g., accuracy, consistency, and relevance) and create a scorecard to evaluate the quality of incoming transaction data.

• Assess Completeness: Before inputting data into their ML model, they can check for missing fields, such as the merchant category code (MCC), which could be crucial for identifying fraud patterns. If many transactions lack this information, they may need to implement a validation step to ensure all necessary fields are populated.

• Monitor Data Consistency: They can regularly review data from different sources for discrepancies. If one data source lists a transaction amount as 100 Eur while another source lists it as 90 Eur, the institution can flag this for review, ensuring that only consistent data is used in their fraud detection models.

6 Data Quality Dimensions Wang & Strong (1996)¶

Wang & Strong (1996)

adapted from Sculley et al. (2015)

Data Quality Monitoring¶

Let’s consider a simple dataset of employee information and insert some outliers:

Automated Profiling with ydata-profiling¶

Formerly known as pandas-profiling — generates a comprehensive HTML report covering all quality dimensions in one call.

Data Quality Dimensions¶

Completeness¶

Metric:

Uniqueness / Duplicates¶

Metric:

Validity & Consistency¶

Values must conform to domain rules and referential constraints.

Examples:

• age must be in [18, 80]

• salary must be > 0

• department must be in a known set

Aggregated Quality Score¶

• Sometimes it can be helpful to aggregate data quality metrics

• Pipino et al. (2002) propose a composite score:

  $\text{DQ Score} = \frac{1}{|D|} \sum_{d \in D} w_d \cdot m_d$

  where $w_d$ is the weight for dimension $d$ and $m_d \in [0, 1]$ is its metric value.

Accuracy / Outlier Detection¶

• The accuracy dimension of data quality refers to statistical properties of data

• Simple outlier detection: parametric tests (z-scoring)

• Statisticians / ML researchers develop anomaly / outlier detection methods

Declarative Data Quality Test¶

Similiar to how unit tests work for software, we can define data quality tests to automatically check for issues in our datasets.

• Testing all data quality dimensions is difficult

• Some are easy to test

• Libraries provide declarative language to define tests

  • https://github.com/awslabs/deequ

  • https://github.com/great-expectations/great_expectations

• Defining tests remains manual work

deequ¶

val verificationResult = VerificationSuite()
    .onData(data)
    .addCheck(
        Check(CheckLevel.Error, "unit testing my data")
            .hasSize(_ == 5) // we expect 5 rows
            .isComplete("id") // should never be NULL
            .isUnique("id") // should not contain duplicates
            .isComplete("name") // should never be NULL
            // should only contain the values "high" and "low"
            .isContainedIn("priority", Array("high", "low"))
            // at least half of the descriptions should contain a url
            .containsURL("description", _ >= 0.5)
            // half of the items should have less than 10 views
            .hasApproxQuantile("numViews", 0.5, _ <= 10))
    .run()```

@schelter2018automating

deequ Workflow:

great_expectations¶

Similar to deequ (Schelter et al) for Scala/Spark, the python library Great Expectations allows you to define, document, and validate data contracts declaratively.

Conceptually:

EXPECT column X TO HAVE values between A and B
EXPECT column Y TO NOT BE NULL
EXPECT table TO HAVE exactly N columns

Schelter et al. (2018)

Anomaly Detection¶

• Anomaly detection is an important task

• Anomaly detection is difficult:

• We usually don’t know what is normal/anomalous

• If we have examples of anomalies, they are usually very rare

• For evaluation, we can use classification metrics but we need labels for that

There are great libraries for anomaly detection:

• scikit-learn anomaly detection

• pyOD

Simple Anomaly Detection¶

Statistical outliers can indicate data entry errors or sensor faults.

Z-score method Iglewicz & Hoaglin (1993) :

IQR method (more robust to non-normal distributions):

Another view: Parametric Distributions¶

Consider univariate data $x\in R^1$ drawn from a normal distribution $\mathcal{N}(\mu,\sigma^2)$.

What’s an anomalous data point?¶

The likelihood of each sample under this distribution is given by

$p(x) = {\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}{2}}\left({\frac {x-\mu }{\sigma }}\right)^{2}}$

So you can compute the likelihood of any given data point as:

Defining Anomaly¶

Given a data point and a (not necessarily generative) model of your data, you can define a threshold for what’s an anomaly.

For instance, a common threshold is to assume that data points for which $p(x)< 0.05$ or $p(x)< 0.01$ are anomalous.

Multivariate Anomaly Detection¶

• Simple Parametric Approach: Assuming Gaussian Data

• Sophisticated Non-parametric: Isolation Forest

Example taken from Alexandre Gramforts sklearn tutorial

Missing Values¶

There are three main options how to deal with missing values:

• drop rows with missing values

  • probably ok, if you have enough data

  • but can introduce biases in your data that bias your downstream models

• replace missing values with placeholder symbol

  • probably ok, if you have a nonlinear downstream model and the right type of missingness (MAR or MNAR)

  • but sometimes you are really interested in the missing values

• impute missing values with some ML model

Dropping rows¶

Easy with boolean indexing in pandas or numpy

Filling with Placeholder¶

Filling with Placeholder in sklearn¶

This way you’ll be able to replace missing values in ML Pipelines

Imputation¶

• You can also use more sophisticated imputation methods

This also works for Tabular Data¶

Biessmann et al. (2019)

Biessmann et al. (2018)

Biessmann et al. (2018)

Realistic Missingness Patterns¶

1. Missing Completely at Random (MCAR):

   • Definition: Missingness is entirely random and unrelated to both observed and unobserved data.

   • Example: Sensor noise

2. Missing at Random (MAR):

   • Definition: Missingness is related to observed data but not to the missing data itself.

   • Example: Older patients not reporting pain levels, but their pain levels are unrelated to whether they reported.

3. Missing Not at Random (MNAR):

   • Definition: Missingness is related to the unobserved data itself.

   • Example: Higher-income individuals not reporting their income.

Realistic Missingness Patterns as generic error models¶

We can use these ideas not only for missingness, but for all kinds of errors

• Modelling realistic errors in tables Jung et al. (2025)

• Detecting realistic error patterns in tables Jung et al. (2025)

• Testing ML systems with realistic error patterns

  • Jäger et al. (2021) and Jäger & Biessmann (2024)

  • Schelter et al. (2021)

  • Rukat et al. (2020)

  • Schelter et al. (2020)

Why should we model Realistic Errors?¶

• Errors are important:

  • Evaluation of cleaning methods / robustness

  • Regularization (adding noise to reduce overfitting)

  • Augmentation (training contrastive learning models, Computer Vision)

  • Privacy (adding noise to hide sensitive data)

• Most error models are oversimplified

  • Better error models can

    • Improve training (regularization, augmentation, privacy)

    • Improve evaluation (more realistic evaluations of robustness)

    • Yield insights in error generation / provenance

Realistic errors with tab-err¶

• Reference implementation of realistic error mechanisms Towards Realistic Error Models for Tabular Data Jung et al. (2025).

• Perturbation of Pandas DataFrames using MCAR/MNAR/MAR for various error types

GitHub: here.

Install tab-err with the command: pip install tab-err

Load the data

Helper function to display the results

APIs¶

Tab-err includes three APIs depending on the level of control desired.

• Low-level API: This API allows for perturbations in one column with a given configuration.

• Mid-level API: This API allows for perturbations in multiple columns with a given configuration.

• High-level API: This API allows for perturbations in multiple columns and needs only an error rate and the data.

All APIs return a dataframe with the perturbed data and a dataframe with a binary error mask.

Low-Level API¶

• Detailed control

  • Select the column to perturb

  • Error mechanism (ECAR, ENAR, EAR)

  • Error rate

  • Error type.

In this example, we utilize missing values with the ECAR error type and a 50% error rate on the sepal_width column.

Mid-Level API¶

• MidLevelConfig allows detailed but scalable control

In our scenario, we will apply two error models to the sepal_width column and one error model to the class column. In the error applied to the class column, we will use the EAR error mechanism and condition on the petal_width column.

Configuration Specification

Application of Mid-Level API

High-Level API¶

• Least user interaction required

• Expects data and an error rate.

• Applies one error model to each column with a random (but valid) error type and error mechanism.

We will perturb the entire dataframe with a rate of 50%.

Perturbing and Imputing¶

Imputing data¶

Imputation improves downstream ML model performance¶

Jäger et al. (2021)

Privacy and redaction¶

• LLMs are popular

• Privacy concerns with personally identifying information

• Named-Entity Recognition models can redact sensitive information

• redakto demo Saha & Biessmann (2025)

• redakto demo screencast

Summary¶

• Data Quality is important

  • For researchers

  • For society

  • For businesses

• Data quality monitoring tools help with testing

• Automating DQ checks remains challenging

• Anna Karenina Principle of Data Quality (due to Charles Sutton):

All healthy data is healthy in the same way. But each broken data set is broken in its own way