Synthetic Study Data Fidelity Benchmark · draft 0.3

Synthetic Study Data Fidelity Benchmark: the method

Draft 0.3, published by Clinventive, LLC, for comment. The exact statistic for every check is in the scoring specification, and the Fidelity Scorer applies both to a study in your browser.

1. What this is

Synthetic study data can be used to test the transfers, mappings, derivations, and programs that process clinical study data, before any real data exists. Poor synthetic data can let a check pass, a program run, or a file load, when the same check, program, or file would fail on real data.

Other work measures particular aspects of synthetic clinical data, among them statistical distance, disclosure risk, and the plausible-patient methods developed in quantitative systems pharmacology. This method proposes seven axes for judging a synthetic study as a whole, each scored on its own, and a scorer anyone can run.

It is a proposal: open for comment, expected to change, and versioned so the changes are visible. It is not a standard, and no standards body has endorsed it.

A conflict of interest. The author created Verdatic, software that generates synthetic clinical study data, and this method can be used to score its output. Weigh the axes with that in mind. To offset that, the method, the scoring code, and the reference ranges are all open to inspection, and anyone can run the scorer, so nobody has to take the author's word for a result.

2. Two kinds of property, never added together

The seven axes are properties of a dataset, and they can be computed from the files. Two other properties matter for synthetic data and cannot be computed from the files: disclosure risk needs the source data the synthetic data was derived from, and reproducibility needs running the generator twice. Both are properties of the generator, not of the dataset. This method does not score them, and they are never combined with the seven axes into one figure.

3. Three rules that apply to every axis

Each check has three outcomes. A single check on a single record passes, fails, or is not run. A check that is not run is never counted as a pass, and it says why: there was nothing to check, a file was not supplied, a record had no value, a date was too imprecise, the method does not run the check in that case, or CORE did not run the rule.

There is no total. Each axis reports its own results, and there is no overall score. A single figure can combine a low result on one axis with a high result on another, so that neither is visible.

A score is compared only with one computed on at least as many values. A dataset can avoid failures by carrying fewer values: a field that is left empty silences every check that needs a value in it. So every result reports the population it was computed on, and a sparse dataset reads as sparse, not as clean.

4. The seven axes

Axis 1: Structural conformance

The question. Does the data satisfy the published conformance rules for the standard it claims to follow?

What is measured. The published rules for the declared standard and version, run by CDISC CORE, CDISC's open-source conformance engine, on the datasets and their define. The result is the count of rules passed, failed, and not run. A rule that CORE skipped or could not run is reported as not run, never as a pass.

The CORE report names the CORE version but not the build. Each result also carries a rule set fingerprint (a short code computed from the ID and version of each rule in the CORE report). A different fingerprint means the rule sets differ. The same fingerprint does not show that two runs used the same engine code, because the fingerprint does not change when only the engine's code changes. Compare axis 1 counts only between runs with the same CORE version and the same fingerprint. Different CORE releases can run different rules, or run the same rules differently, so their counts on the same data can differ.

Why it matters. It is the only axis scored by an independent, publicly maintained engine, so its result does not depend on this method's own code.

Boundary. It tests the data against a rule set, not against reality. Data that passes has shown that it is well formed, not that it is realistic. Declaring a narrow scope avoids the rules that would fail, which is why every count carries its denominator and why axis 7 exists.

Axis 2: Referential integrity

The question. Are subject, visit, and parent-record references consistent across datasets?

What is measured. The share of references that resolve, for each kind of reference: every subject identifier used in any dataset has a demographics (DM) record; visit references resolve to visits the schedule defines (TV); every supplemental qualifier has its parent record; and relationships declared in RELREC point at records that exist. Each kind is reported separately, because each fails for different reasons.

TV lists only the planned visits, so an unscheduled visit is not in it. A visit that is not in TV resolves when the subject's visit records (SV) hold it and do not mark it as planned. These references are counted as resolved and reported separately. A visit that neither TV nor SV holds is unresolved.

Why it matters. Without it, the generated tables do not form a coherent study, and each broken reference can create reconciliation work downstream.

Boundary. This axis does not check values. Every reference can resolve while every value is wrong.

Axis 3: Temporal coherence

The question. Is the study's timeline consistent?

What is measured.

Why it matters. Timing errors can trigger edit checks in a real system. Checks that fire for errors the generator made hide the defects a test is meant to find.

Boundary. Real studies have late and out-of-window visits, so the method reports window adherence as a rate and does not reward a perfect one. Ordering, study bounds, and visit dates are defects whenever they fail.

Axis 4: Clinical plausibility

The question. Would a clinician believe the values?

What is measured.

Reading the reference-range share. This share describes the data. It is not a pass or fail target. The ranges are published adult reference ranges and limits. Blood pressure is compared with a normal adult resting range, 90/60 to 120/80 mmHg (MedlinePlus). The glucose range is a fasting range, so a glucose result is compared with it only when its record says the sample was fasting (LBFAST = Y). Otherwise the comparison is not run, and the scorer says why.

Reference data. Published sources only. Each entry in the reference bundle cites its source and was checked against a copy of that source, and the bundle ships with the scorer under its own version number. A reference range describes healthy people, so it is never used as a limit: an abnormal result is not a defect, and a dataset with none is itself unrealistic.

Every entry except oxygen saturation carries a lower age limit of 18 years, including the vital sign, ECG and body size entries. A result is compared with an entry only when the subject's age, read with its unit (AGEU), meets the entry's limit, so a child's results, vital signs included, are not compared with these ranges. The oxygen saturation range applies at any age. An age with no AGEU, or with an AGEU other than YEARS, MONTHS, WEEKS, DAYS or HOURS, is treated as unknown. A subject whose age is unknown is not assumed to be an adult: the comparison is not run, and the scorer says why.

Boundary. This axis does not check that a value is right. It would pass a dataset in which every value is believable and no subject makes sense as a person. Axes 5 and 6 address that.

Axis 5: Longitudinal realism

The question. Do each subject's values change over time as one person's would?

What is measured, with no target:

Why it matters. Change-from-baseline and every other within-subject analysis depend on it. A subject whose values are generated independently at each visit has a plausible value at every visit and no relationship between one visit's value and the next.

PK concentrations (PC) get no ICC, because they are sampled across each dosing interval and change with the time since the dose. PK parameters (PP) are kept apart by analyte (PPCAT), so two analytes that share a test code are not pooled.

Boundary. These measures show that structure exists and has a plausible shape. They do not judge whether a trajectory is right for the indication, and the method publishes no threshold, because the expected values depend on the measure, the indication, and the length of the study. Read them by comparison with other studies.

Axis 6: Cross-domain clinical consistency

The question. Are the parts of each subject's record clinically consistent?

What is measured. Four relationships in draft 0.3, each with its rule stated openly so that it can be disputed:

Why it matters. A record that contradicts itself can remain after every other check has passed. Data can be conformant, linked, correctly ordered, plausible, and consistent over time, and still describe a subject whose record contradicts itself.

Boundary. The rule set is small on purpose: it holds only relationships that can be stated precisely enough for two readers to agree. Much of clinical coherence cannot be stated that way, and this axis does not reach it.

Axis 7: Deliverable completeness

The question. Is everything the define declares actually delivered?

What is measured. A checklist, with each item marked yes, no, partly, or could not tell:

The items are not combined into a percentage, because one item cannot stand in for another.

Why it matters. A generator can score well on the other axes by delivering less. A single clean dataset can score higher than a complete study on the axes that score data values, while being of no use for the testing the study needs.

Boundary. This axis does not check the data's quality. A complete package of poor data scores well here and poorly on the other axes, which is why this axis is reported beside them and not folded into them.

5. What this method does not do

6. How the method is run and changed