Synthetic data generation

Synthetic data - what is it really?

Synthetic data is artificially generated data that replicates the structure and statistical properties of real-world information.

In life sciences we're constantly dealing with strict compliance requirements, data scarcity, and increasing focus on patient privacy, which makes collecting real-world, high-quality datasets increasingly challenging. Therefore synthetic data is rapidly becoming an essential asset in AI development, especially in pharmaceutical GxP environments. It enables organizations to rapidly scale AI model development, safely explore innovative scenarios, and significantly reduce privacy risks.

Synthetic data can bridge critical gaps, driving faster innovation while still respecting regulatory boundaries - making it not just relevant, but indispensable to the future of Pharma.

How do we generate it?

There are multiple practical methods to create synthetic data, each designed to address specific challenges and data types effectively. They include:

o    Generative Adversarial Networks (GANs):
GANs use two neural networks (a generator and a discriminator) trained together adversarially. The generator creates synthetic data and sends it to the discriminator, which tries to distinguish fake from real. If the discriminator succeeds, the generator is updated; if not, the discriminator gets an update to improve its accuracy. The generator’s goal is to fool the discriminator — minimizing its ability to tell the difference. Through this competitive loop, the generated data gradually becomes indistinguishable from real data.

o    Variational Autoencoders (VAEs):
VAEs encode real-world data into a compressed representation and then decode it back into new (synthetic) data. This encoding-decoding mechanism ensures synthetic data captures essential characteristics of real datasets while allowing for flexibility and variation.

o    Diffusion Models:
Diffusion models work by gradually adding controlled noise to real data and then learning to reverse this process, reconstructing clear, realistic data from noisy inputs. They're particularly effective in generating high-quality synthetic images or complex scenarios such as medical imaging scans, significantly enhancing realism and detail.

o    Transformers:
Transformers handle sequential data such as text or personal (e.g. patient) records by learning context and relationships within the data. Once trained, they can generate synthetic narratives or patient histories that resemble real data to a very high degree.

o    Digital Twins:
Digital Twins are virtual replicas of physical systems or processes updated in real-time. In Pharma manufacturing, Digital Twins can model production lines or equipment, enabling teams to generate synthetic datasets representing various operational scenarios, performance conditions, or failure modes without impacting real production.

Depending on the business case an assessment should be done to determine what kind of data do we need to train our model. Each method of generating synthetic data has its own characteristics, making it more or less useful for the intended purpose.

However, it is important to note that none of the above generative models holds the crown for synthesizing bias-free data. In fact, there are studies showing that all major model types (GANs, VAEs, Transformers, and diffusion models) have demonstrated bias in their outputs when trained naively.

For example, this NeurIPS 2023 evaluation found that unmodified state-of-the-art generators – including a GAN (StyleGAN2), a transformer-based GAN (StyleSwin), and a diffusion model (Stable Diffusion) – all exhibited considerable bias in generated data.

However, for each of the above methods, additional mechanisms and benchmarks can be set up in order to boost the fairness of the generated datasets. For example, GAN architecture can be modified in such a way, that one discriminator ensures the generated data resembles real data, while a second discriminator ensures the generated features and outcomes are statistically independent of the sensitive attribute​ (example can be found here).

It is also crucial to understand that none of these techniques will help if our models are using real-world biased data, especially if the bias is hidden. This is because synthetic data generators (like GANs, VAEs, or LLMs) aim to mimic the distribution of the training data. If bias is embedded in the real data, the synthetic data will reflect the same statistical properties, including the bias (sometimes in even more amplified ways).

How can we ensure that our dataset is bias-free?

What are the benefits of using synthetic data?

What are the risks?

What are the do’s and don’ts?

We will explore all of it in next articles!