What is data quality?

If you haven’t been living under a rock for the last couple of years, you’ve probably heard about the importance of data quality in training AI models. It’s often summarized by the GIGO principle (Garbage In – Garbage Out), which is a cornerstone of AI model development and performance. But what does it really mean? What are the characteristics of data quality, and how can we measure if our data is good enough?

In pharmaceutical GxP environments, we have the universal “ALCOA++” framework for ensuring data integrity, so we should be fine, right? Well… Yes. And no. It’s complicated, so let’s dive into the nitty-gritty of this problem.

ALCOA++ and GxP Documentation

At this point, most people familiar with GxP have heard of ALCOA acronym and what it stands for: Attributable, Legible, Contemporaneous, Original and Acurate, etc. But the real question is: why are we obliged to use it? It doesn’t just apply to our computerized systems; it also governs all paper-based, GxP-relevant documentation. There must be a concept behind this. And there is.

GxP documentation is basically a time machine. Since we don’t live in a world where all our actions are automatically recorded 24/7 (not yet, anyway), we must keep an accurate representation of our actions in document form. These documents can later be used for many purposes—like regulatory compliance, product recalls, and inspections. At the end of the day, it all boils down to patient safety. Whether we’re showing the evidence to the product quality or investigating a deviation that led to a product recall, patient is always at the center.

ALCOA++ Principles

Here’s what ALCOA++ stands for:

In typical GxP settings, Computerized System Validation (CSV) usually covers the testing, validation, and production environments. The development environment isn’t as critical from a strict compliance viewpoint because it doesn’t normally hold or process data relevant to patient safety. (Of course, there can be exceptions, but this is the usual situation.)

The AI/ML Twist

Things change when we integrate AI models into GxP processes. Developing an AI system relies on iterative cycles of model training, testing, and validation—work that often falls outside the traditional CSV scope. A model “learns” from the input data you provide, so data quality has a direct impact on model performance.

Here, our main concern is not traveling back in time to check if data was properly recorded, but rather ensuring the data is accurate and free from bias so the model can learn efficiently. ALCOA++ rules focus on data integrity in a regulatory sense, which is essential but does not cover every aspect of data quality needed for robust, fair, and high-performing AI models.

That’s why we need a different standard specifically dedicated to providing the data quality needed for AI/ML model development, ensuring patient safety, product quality, and overall data integrity (not just the integrity of training data, but of the entire system).

ISO/IEC 5259-2:2024 for AI Data Quality

Enter the ISO standards. ISO/IEC FDIS 5259-2:2024, which has been under discussion for several years, was approved in 2024. Its purpose is to define a data quality model, data quality measures, and guidance on reporting data quality in analytics and machine learning contexts. When you look at this standard, it’s clear that ALCOA++ omits several critical data quality factors crucial for building robust and ethical ML models.

Four Categories of Data Quality Characteristics

ISO/IEC 5259-2:2024 outlines four main categories of data quality characteristics:

1. Inherent Data Quality Characteristics
   Accuracy, Completeness, Consistency, Credibility, Currentness
2. Inherent and System-Dependent Data Quality Characteristics
   Accessibility, Compliance, Efficiency, Precision, Traceability, Understandability
3. System-Dependent Data Quality Characteristics
   Availability, Portability, Recoverability
4. Additional Data Quality Characteristics
   Auditability, Balance, Diversity, Effectiveness, Identifiability, Relevance, Representativeness, Similarity, Timeliness

Comparing ISO vs. ALCOA++

Below is a side-by-side comparison of how ALCOA++ aligns (or doesn’t) with the ISO 5259-2 list:

1. Accuracy (ISO Inherent) vs. Accurate (ALCOA++)
   Both emphasize correctness and freedom from errors.
2. Completeness (ISO Inherent) vs. Complete (ALCOA++)
   Both require no missing data, no untracked repeats, and no selective reporting.
3. Consistency (ISO Inherent) vs. Consistent (ALCOA++)
   Both address internal coherence, such as timestamps and logical data sequencing.
4. Traceability (ISO Inherent & System-Dependent) vs. Traceable (ALCOA++)
   Both demand linking data points to their source, user, date/time of capture, and the entire data life cycle.
5. Availability (ISO System-Dependent) vs. Available (ALCOA++)
   Both stress ready data retrieval for review or audit.
6. Credibility, Currentness (ISO Inherent)
   ALCOA++ does not explicitly mention “credibility” or “currentness,” though there’s an indirect link via “Original” and “Contemporaneous.” In ISO, “Currentness” means data is up to date, whereas ALCOA++ focuses more on recording data at the time of action rather than ensuring it’s still “fresh.”
7. Accessibility, Efficiency, Precision, Understandability (ISO)
   These do not appear in ALCOA++. ALCOA++ is primarily about data integrity, whereas ISO includes how data can be effectively processed and understood.
8. Compliance (ISO)
   ALCOA++ itself stems from compliance requirements (FDA, EMA, MHRA, etc.), so it doesn’t list “compliance” as a separate category—it’s assumed.
9. Legible & Enduring (ALCOA++)
   ISO touches on readability and longevity in terms of “Accessibility,” “Understandability,” and “Availability,” but ALCOA++ is more explicit with “Legible” and “Enduring.”
10. Balance, Diversity, Representativeness, Similarity (ISO Additional)
    ALCOA++ does not address whether a dataset is diverse or representative. ISO explicitly points to these factors, which are crucial for ML fairness and performance.
11. Timeliness (ISO Additional) vs. Contemporaneous (ALCOA++)
    ISO’s “Timeliness” is about having data available when needed, while ALCOA++ focuses on recording data at the moment of action. They overlap but address slightly different concerns.
12. Auditability (ISO Additional) vs. Traceable / Attributable (ALCOA++)
    Both emphasize the importance of audits. ALCOA++ ensures each record is linked to a person and time, which is fundamental for audits.
13. Relevance, Effectiveness, Identifiability (ISO Additional)
    ALCOA++ does not explicitly address whether the data is relevant or fit for a particular AI/ML purpose. ISO highlights the importance of data suitability for specific business or analytical goals.

Gaps When Applying ALCOA++ to AI/ML

1. Model Fit, Representativeness, and Bias
   ALCOA++ does not mention whether the dataset is representative, balanced, or relevant to the population or problem. In AI/ML, data diversity and representativeness are vital. ISO’s “Balance,” “Diversity,” and “Representativeness” fill this gap.
2. Data Quality Beyond Integrity
   ALCOA++ ensures data integrity, traceability, and compliance of records, but AI/ML also needs labeling and annotation quality, bias checks, completeness vs. noise assessment, and real-time data timeliness.
3. Lifecycle vs. Pipeline
   ALCOA++ primarily looks at data acquisition, transformation, storage, and archival. AI/ML, however, involves iterative pipelines: data ingestion, cleaning, feature engineering, model training, validation, and ongoing monitoring. Additional guidelines on versioning data, model artifacts, and parameter logs may be required.
4. Fairness, Transparency, and Explainability
   ISO includes characteristics like “Understandability” and “Traceability,” but the AI/ML community also emphasizes fairness, interpretability, and broader ethical considerations—topics that go beyond what ALCOA++ was originally designed to handle.

Conclusion

While ALCOA++ remains a solid foundation for data integrity in GxP environments, it doesn’t fully cover the additional demands of AI and machine learning. ISO/IEC 5259-2:2024 closes these gaps by providing a broader set of data quality characteristics—such as balance, diversity, relevance, and timeliness—that are crucial for building ethical, robust, and high-performing AI models. Together, ALCOA++ and the ISO framework can help ensure that both compliance and real-world AI performance requirements are met, keeping patient safety, product quality, and data integrity at the forefront of innovation.