1. What this is
Synthetic study data can be used to test the transfers, mappings, derivations, and programs that process clinical study data, before any real data exists. Poor synthetic data can let a check pass, a program run, or a file load, when the same check, program, or file would fail on real data.
Other work measures particular aspects of synthetic clinical data, among them statistical distance, disclosure risk, and the plausible-patient methods developed in quantitative systems pharmacology. This method proposes seven axes for judging a synthetic study as a whole, each scored on its own, and a scorer anyone can run.
It is a proposal: open for comment, expected to change, and versioned so the changes are visible. It is not a standard, and no standards body has endorsed it.
A conflict of interest. The author created Verdatic, software that generates synthetic clinical study data, and this method can be used to score its output. Weigh the axes with that in mind. To offset that, the method, the scoring code, and the reference ranges are all open to inspection, and anyone can run the scorer, so nobody has to take the author's word for a result.
2. Two kinds of property, never added together
The seven axes are properties of a dataset, and they can be computed from the files. Two other properties matter for synthetic data and cannot be computed from the files: disclosure risk needs the source data the synthetic data was derived from, and reproducibility needs running the generator twice. Both are properties of the generator, not of the dataset. This method does not score them, and they are never combined with the seven axes into one figure.
3. Three rules that apply to every axis
Each check has three outcomes. A single check on a single record passes, fails, or is not run. A check that is not run is never counted as a pass, and it says why: there was nothing to check, a file was not supplied, a record had no value, a date was too imprecise, the method does not run the check in that case, or CORE did not run the rule.
There is no total. Each axis reports its own results, and there is no overall score. A single figure can combine a low result on one axis with a high result on another, so that neither is visible.
A score is compared only with one computed on at least as many values. A dataset can avoid failures by carrying fewer values: a field that is left empty silences every check that needs a value in it. So every result reports the population it was computed on, and a sparse dataset reads as sparse, not as clean.
4. The seven axes
Axis 1: Structural conformance
The question. Does the data satisfy the published conformance rules for the standard it claims to follow?
What is measured. The published rules for the declared standard and version, run by CDISC CORE, CDISC's open-source conformance engine, on the datasets and their define. The result is the count of rules passed, failed, and not run. A rule that CORE skipped or could not run is reported as not run, never as a pass.
The CORE report names the CORE version but not the build. Each result also carries a rule set fingerprint (a short code computed from the ID and version of each rule in the CORE report). A different fingerprint means the rule sets differ. The same fingerprint does not show that two runs used the same engine code, because the fingerprint does not change when only the engine's code changes. Compare axis 1 counts only between runs with the same CORE version and the same fingerprint. Different CORE releases can run different rules, or run the same rules differently, so their counts on the same data can differ.
Why it matters. It is the only axis scored by an independent, publicly maintained engine, so its result does not depend on this method's own code.
Boundary. It tests the data against a rule set, not against reality. Data that passes has shown that it is well formed, not that it is realistic. Declaring a narrow scope avoids the rules that would fail, which is why every count carries its denominator and why axis 7 exists.
Axis 2: Referential integrity
The question. Are subject, visit, and parent-record references consistent across datasets?
What is measured. The share of references that resolve, for each kind of reference: every subject identifier used in any dataset has a demographics (DM) record; visit references resolve to visits the schedule defines (TV); every supplemental qualifier has its parent record; and relationships declared in RELREC point at records that exist. Each kind is reported separately, because each fails for different reasons.
TV lists only the planned visits, so an unscheduled visit is not in it. A visit that is not in TV resolves when the subject's visit records (SV) hold it and do not mark it as planned. These references are counted as resolved and reported separately. A visit that neither TV nor SV holds is unresolved.
Why it matters. Without it, the generated tables do not form a coherent study, and each broken reference can create reconciliation work downstream.
Boundary. This axis does not check values. Every reference can resolve while every value is wrong.
Axis 3: Temporal coherence
The question. Is the study's timeline consistent?
What is measured.
- Ordering: each record's end (--ENDTC) on or after its start (--STDTC). A start and an end on the same day are in order unless their times say otherwise.
- Study bounds: every date inside the subject's own participation, from informed consent (RFICDTC) to the end of participation (RFPENDTC). Dates in MHSTDTC, MHENDTC, CMSTDTC, CMENDTC, PRSTDTC, PRENDTC and GFDTC that fall before consent are reported, not scored, because these variables can record events or specimens from before the study.
- Visit dates: each finding dated on or between the start and end dates of its visit in SV. A finding at a visit with no dated SV record is counted as not run.
- Window adherence: when visit windows are supplied, how many visit records fall inside them. SDTM has no numeric variables for windows (TV holds them only as text), so they come from a facts file. This is reported with no target.
Why it matters. Timing errors can trigger edit checks in a real system. Checks that fire for errors the generator made hide the defects a test is meant to find.
Boundary. Real studies have late and out-of-window visits, so the method reports window adherence as a rate and does not reward a perfect one. Ordering, study bounds, and visit dates are defects whenever they fail.
Axis 4: Clinical plausibility
The question. Would a clinician believe the values?
What is measured.
- Results that cannot be a measurement at all, such as a negative concentration or an oxygen saturation above 100%.
- Results outside a published plausibility limit, only for tests where such a limit is cited. No test in reference bundle 0.3 has a cited limit yet.
- Units that are valid for the test. Conventional and SI units are both accepted where the bundle states the conversion between them, such as mg/dL and umol/L for creatinine.
- Coded values that belong to the codelists the study's own define assigns, value-level first.
- Precision that stays consistent within a series (a dataset and test code, plus PPCAT in PP): each result has the same number of decimal places, or the same number of significant figures, as the most common count among results of that series.
- How often results fall outside the reference range, reported with no target.
Reading the reference-range share. This share describes the data. It is not a pass or fail target. The ranges are published adult reference ranges and limits. Blood pressure is compared with a normal adult resting range, 90/60 to 120/80 mmHg (MedlinePlus). The glucose range is a fasting range, so a glucose result is compared with it only when its record says the sample was fasting (LBFAST = Y). Otherwise the comparison is not run, and the scorer says why.
Reference data. Published sources only. Each entry in the reference bundle cites its source and was checked against a copy of that source, and the bundle ships with the scorer under its own version number. A reference range describes healthy people, so it is never used as a limit: an abnormal result is not a defect, and a dataset with none is itself unrealistic.
Every entry except oxygen saturation carries a lower age limit of 18 years, including the vital sign, ECG and body size entries. A result is compared with an entry only when the subject's age, read with its unit (AGEU), meets the entry's limit, so a child's results, vital signs included, are not compared with these ranges. The oxygen saturation range applies at any age. An age with no AGEU, or with an AGEU other than YEARS, MONTHS, WEEKS, DAYS or HOURS, is treated as unknown. A subject whose age is unknown is not assumed to be an adult: the comparison is not run, and the scorer says why.
Boundary. This axis does not check that a value is right. It would pass a dataset in which every value is believable and no subject makes sense as a person. Axes 5 and 6 address that.
Axis 5: Longitudinal realism
The question. Do each subject's values change over time as one person's would?
What is measured, with no target:
- How consistent each subject's values are from visit to visit, for each measure (the intraclass correlation, or ICC).
- A second ICC, computed on the logarithm of each value and reported beside the first. It is computed only when the ICC is computed and every value it uses is above zero. On the log scale a very high value is closer to the rest, so a few very high values, such as one grade 3 liver enzyme result among normal ones, tend to lower the log-scale ICC less than they lower the ICC. A planned change that every subject goes through, such as a treatment nadir, still lowers it.
- The shape of each subject's series: moving in one direction, returning toward a level, or without structure.
- The pattern of missing visits: subjects whose visits stop early, have gaps, or are complete.
- How many subjects have a record at each scheduled visit.
Why it matters. Change-from-baseline and every other within-subject analysis depend on it. A subject whose values are generated independently at each visit has a plausible value at every visit and no relationship between one visit's value and the next.
PK concentrations (PC) get no ICC, because they are sampled across each dosing interval and change with the time since the dose. PK parameters (PP) are kept apart by analyte (PPCAT), so two analytes that share a test code are not pooled.
Boundary. These measures show that structure exists and has a plausible shape. They do not judge whether a trajectory is right for the indication, and the method publishes no threshold, because the expected values depend on the measure, the indication, and the length of the study. Read them by comparison with other studies.
Axis 6: Cross-domain clinical consistency
The question. Are the parts of each subject's record clinically consistent?
What is measured. Four relationships in draft 0.3, each with its rule stated openly so that it can be disputed:
- CD-01: a treatment recorded for an indication has that indication in the subject's medical history or adverse events. Only case, extra spaces and punctuation are ignored in the comparison. CD-01 checks that the condition a treatment was given for is recorded, and it is not run for an indication that names prophylaxis or prevention. An unmatched indication that is close to one of the subject's terms, by spelling, spacing or wording, is listed as a near match with a count. It is still counted as unmatched: the scorer does not decide that two different terms agree.
- CD-02: a discontinuation attributed to an adverse event has an adverse event recorded on or before it.
- CD-03: study treatment withdrawn because of an adverse event is not given after that event ends.
- CD-04: an event term is compatible with the subject's recorded sex.
Why it matters. A record that contradicts itself can remain after every other check has passed. Data can be conformant, linked, correctly ordered, plausible, and consistent over time, and still describe a subject whose record contradicts itself.
Boundary. The rule set is small on purpose: it holds only relationships that can be stated precisely enough for two readers to agree. Much of clinical coherence cannot be stated that way, and this axis does not reach it.
Axis 7: Deliverable completeness
The question. Is everything the define declares actually delivered?
What is measured. A checklist, with each item marked yes, no, partly, or could not tell:
- the observation classes the datasets cover;
- the trial design datasets;
- a define;
- value-level metadata;
- every dataset the define declares, supplied.
The items are not combined into a percentage, because one item cannot stand in for another.
Why it matters. A generator can score well on the other axes by delivering less. A single clean dataset can score higher than a complete study on the axes that score data values, while being of no use for the testing the study needs.
Boundary. This axis does not check the data's quality. A complete package of poor data scores well here and poorly on the other axes, which is why this axis is reported beside them and not folded into them.
5. What this method does not do
- It does not rank tools. It reports each axis and does not add them up.
- It does not certify anything, and no result should be described as a certification, a validation, or an endorsement by any standards body.
- It does not measure fitness for a specific purpose. Data adequate for testing one edit check may be inadequate for anything else.
- It does not measure privacy or disclosure risk, which are properties of the generator (section 2).
- It says nothing about whether synthetic data is appropriate for a given use. That is a question for the person accountable for the use.
6. How the method is run and changed
- The scoring code is open. The scorer runs in your browser at clinventive.com/fidelity/score, and the page serves its scoring code as readable files. A command-line version runs the same code; it is not published yet.
- Public reference data only, each entry cited (axis 4).
- No data is collected to produce a score. Scoring runs on your computer, and the page does not send your files anywhere.
- Comment is open to anyone. Each change is recorded with a new version number, and every result names the method, scorer, and reference-range versions it used.
- Under consideration, not offered: registering an intent to score a tool before seeing its results, so that a tool registered for scoring whose results were never reported is visible, and a way to report results, with claimed and independently checked results shown separately.