CDISC's goal is that "By the end of 2030, all CDISC standards are digital, connected, and easily consumable by users and systems." AI agents are playing a growing part in the automation those standards make possible.
But how does the industry verify that a digital data flow is built correctly and fit-for-purpose? Testing automated pipelines on synthetic data exposes defects, but only if the synthetic data accurately models a real clinical pipeline. Is your synthetic data good enough to meet this challenge?
In my talk at the Interchange on October 6, I proposed seven axes for defining synthetic data quality. Each axis supports a different kind of testing; together, they reveal exactly which downstream tests a synthetic study can reliably support.
You can now apply these metrics to your own synthetic SDTM studies using the Fidelity Scorer. It runs entirely in your browser, ensuring your files never leave your computer.
The Seven Axes
The method explains why each axis matters, and the scoring specification dictates exactly what each one checks. Here is what each axis asks, what it allows you to test, and an example of what it uncovers:
Structural conformance. Does the data satisfy the published rules for its standard? The scorer reads the report generated by CORE, CDISC's open-source conformance engine.
- Tests: Mapping, the define, and any tool that expects conformant input.
- Catches: A required variable left out.
Referential integrity. Are subject, visit, and parent-record references consistent across datasets?
- Tests: Cross-form checks, merges, and joins.
- Catches: An adverse event for a subject with no demographic record.
Temporal coherence. Is the study's timeline mathematically and logically consistent?
- Tests: Date and visit-window edit checks, and visit derivations.
- Catches: An assessment dated before the subject consented.
Clinical plausibility. Would a clinician believe the values?
- Tests: Medical review listings, lab units, and coding checks.
- Catches: An oxygen saturation of 101%.
Longitudinal realism. Do each subject's values change over time as a single person's would?
- Tests: Change-from-baseline derivations, shift tables, and subject profiles.
- Shows: A blood pressure generated independently at each visit. It may be plausible every time, but unrelated from one visit to the next—resulting in an intraclass correlation near zero.
Cross-domain clinical consistency. Are the parts of each subject's record clinically consistent with each other?
- Tests: Safety listings across datasets, and medications against indications.
- Catches: Dosing recorded after the end of an adverse event for which the drug was permanently withdrawn.
Deliverable completeness. Is everything the define declares actually delivered, and does the define match the data?
- Tests: The define and the package.
- Catches: A dataset listed in the define that was never delivered.
Scoring a Study
The step-by-step guide walks you through the process. You need the SDTM datasets (XPT, Dataset-JSON, or CSV files) in a folder or ZIP file. The define and a CORE report are optional, though the guide will build the CORE command for you. (Without a CORE report, axis 1 is not scored).
Drop the folder onto the scorer in a desktop browser. There is nothing to install and no account to create. Because the page processes your files locally, no data is sent to Clinventive or anyone else. The guide shows you how to verify this privacy using your browser's developer tools.
Reading the Result
Each axis is reported on its own. There is no weighted overall total, and a check that could not run is never counted as a pass. The guide explains how to read each label on the result cards.
When evaluating synthetic data, a falling count of violations can be misleading. The count drops when the data improves, but it also drops when the data is sparse. If a field is left empty, no check that relies on that field can run. For this reason, a better score only counts if it was computed on at least as many values as the score it is compared with.
This dynamic is critical when an AI system generates the data. A system rewarded simply for minimizing violations will often learn to remove data instead of correcting it. Our scoring exposes this by ensuring every result reports the population it was computed on.
An Interest to Declare
Full disclosure: Verdatic, the software I created, generates synthetic SDTM studies from a study design and the CDISC standards. Therefore, this method can be used to score its output. Verdatic uses CORE's rules as constraints during generation, runs the CORE engine on the finished datasets, and feeds the results back into the next round of generation.
Against that commercial interest, I have made the method, the scoring code, and the reference ranges entirely open to inspection. Anyone can run the scorer. You should not have to take my word for any score.
Comment on the Axes and Their Scorers
The purpose of these axes is to know exactly which kinds of testing a synthetic study can support before real data arrives. However, a definition of synthetic data quality requires broad consensus before it can become an industry standard. Today, only axis 1 has an independent engine. Reaching an agreement on how the remaining six are scored requires input from across the field.
The method and its scoring specification are currently in Draft 0.2, and I want your feedback. Which axis would you define or score differently?
Use the forms below to drop a comment on the overall methodology or on a specific axis. (Your comments go directly to me and will not be published publicly).
You can also message me directly on LinkedIn, or start a conversation with Clinventive about testing your next study build on synthetic data.