Synthetic Study Data Fidelity Benchmark · draft 0.3

Fidelity Scorer

Clinical study data passes through many automated steps between collection and analysis: transfers, mappings, derivations and programs. Testing those steps on synthetic data before real data arrives can show whether they work, but only if the synthetic data is good enough. Poor synthetic data can hide problems: a check passes or a program runs on the synthetic data, and the same check or program fails on real data.

The Fidelity Scorer reads a synthetic study in SDTM, the CDISC standard format for study datasets, and scores it on seven separate measures, called axes. For each axis it reports which checks the study passes, which it fails, and what could not be checked, so you can see which kinds of testing the study supports before you rely on it.

The seven axes

Each axis asks one question and is reported on its own. There is no total score, because a good result on one axis cannot make up for a poor result on another. Several axes refer to the define: the Define-XML file (define.xml) that describes a study's datasets.

AxisThe question it asksWhat it lets you test
1. Structural conformanceDoes the data satisfy the published rules for its standard?Mapping, the define, and any tool that expects conformant input
2. Referential integrityAre subject, visit and parent-record references consistent across datasets?Cross-form checks, merges and joins
3. Temporal coherenceIs the study's timeline consistent?Date and visit-window edit checks, visit derivations
4. Clinical plausibilityWould a clinician believe the values?Medical review listings, lab units, coding checks
5. Longitudinal realismDo each subject's values change over time as one person's would?Change-from-baseline derivations, shift tables, subject profiles
6. Cross-domain clinical consistencyAre the parts of each subject's record clinically consistent?Safety listings across datasets, medications against indications
7. Deliverable completenessIs everything the define declares delivered, and does the define match the data?The define and the package

Axis 1 is scored from a report by CDISC CORE, CDISC's open-source conformance engine. The scorer reads the report and does not run the rules itself. The other six axes are scored by the scorer, following the draft method.

What you need

If the study comes with a facts file (submission.json) in the format the scorer's command-line version uses, the scorer reads it too, and the facts it states take the place of what the scorer worked out from the files. The scorer says when it used one.

How to score a study

  1. Open the scorer. Go to clinventive.com/fidelity/score.
  2. Give it the study. Drag the study folder or its ZIP file onto the box, or click Choose files or a ZIP. In Chrome and Edge there is also Choose a folder. The browser then asks whether to let the site view the files in that folder; that permission lets the page read them on your computer. In other browsers, drag the folder onto the box. Dragging asks for no permission. Whichever you use, the page reads the files on your computer and does not send them anywhere (see What stays on your computer).
  3. Check what it found. The scorer lists the datasets it will score and whether it found a define, a CORE report and a visit schedule. If the same datasets are in two formats, it scores one copy and says which. If something you expected is missing, add the file and score again. If it finds no datasets at all, check that they are .xpt, .json or .csv files. If it stops with an error, send the message through the contact form.
  4. Read the results. Each axis has its own card. See How to read the results.
  5. Save the results. Save the scorecard as text saves a file you can read or share. Save the results for other software saves the same results as JSON. Both files are made on your computer.

How to read the results

Each axis card shows one of six labels.

No problems foundEvery check on this axis ran and none found a problem.
No problems in the checks that ranNone of the checks that ran found a problem, but some checks were not run. The card lists them as Not run and says why. Those parts of the axis are unknown, not passed.
Problems foundAt least one check found a problem. The card lists each check, how many times it ran, and how many problems it found, with examples.
Not scoredNothing on this axis could be checked with the files supplied. The card says what was missing, such as a CORE report for axis 1.
Measured, no targetMeasurements that are not pass or fail, such as how consistent each subject's values are from visit to visit (the intraclass correlation, ICC). There is no right answer. Compare them with other studies.
ChecklistAxis 7 lists what was delivered: yes, no, partly, or could not tell.

A check that was not run says why, on its own row. Each reason is one of six kinds.

Nothing to checkThe study has no records the check applies to. For example, the check on discontinuations attributed to an adverse event has nothing to check in a study where no subject discontinued for that reason.
Not suppliedA dataset or file the check reads was not in the files, such as the Subject Visits (SV) dataset or a define. In a CORE V1.0 report, it can also be a coding dictionary a rule needs. Some datasets are optional, and a study may simply have none.
No valueA record has no value the check needs, or a value that cannot be read. An event that is still ongoing, for example, has no end date.
Dates too impreciseA date is not precise enough for the check, for example a start recorded only as March 2022 and an end on 15 March 2022.
Not run by designThe method does not run the check in this case, for example a range check for a test with no published limit to cite, or a subject outside the reference ranges that ship with the scorer.
CORE did not run itCDISC CORE gave no pass or fail for the rule, because it skipped the rule or failed to run it. A CORE V1.0 report gives the reason for a skip, and the scorer shows it. A CORE 0.17.1 report does not.

Only Not supplied can be fixed from your side: add the file and score again. A dictionary goes to CORE and not to the scorer, so it means making a new report. The other kinds describe the data, the method or CORE. Variable names written with two dashes, such as --DTC, stand for the same variable in every dataset: LBDTC in LB, VSDTC in VS.

Three rules apply to every card.

Getting a CORE report (axis 1)

Axis 1 is scored from a CORE report. CDISC CORE does not run in a browser. It is a program you start by typing a command on your computer, so you make the report there and then add it to the scorer. If you do not have a CORE report, the other six axes are still scored and axis 1 shows Not scored.

Which CORE to use

CDISC's CORE page lists two versions of CORE, and they run different sets of rules. It lists CORE V1.0 for SDTMIG 3.2, 3.3 and 3.4 and the FDA Business Rules, and CORE 0.17.1 for TIG, USDM and SENDIG, which are other CDISC standards. CORE V1.0 is in Verisian's GitHub account, the location CDISC's page links to, and its page calls it Verisian's version of the CDISC rules engine. CORE 0.17.1 is in cdisc-org, CDISC's GitHub account. It still checks SDTM data with the rules it ships with.

CORE V1.0CORE 0.17.1
Use it ifYour datasets are XPT files and you can install PythonYou cannot install Python, or your datasets are Dataset-JSON
Rule packages CDISC released on November 3, 2026IncludedNot included
Dataset files it readsXPT (.xpt)XPT (.xpt) or Dataset-JSON (.json)
SetupInstall Python 3.12, then run two commandsUnzip a download

For SDTM data, use one of these two versions, not the v1.0.0 release in cdisc-org. Two GitHub releases carry the number 1.0.0, and they are different code. The one in cdisc-org is marked Latest on that account's releases page, and CDISC's CORE page does not list it.

In our test on one SDTMIG 3.4 study in November 2026, the cdisc-org release gave different results from CORE V1.0 on the same files.

Reports from both say version 1.0.0. If your report says 1.0.0 and you made it with a downloaded program instead of Python, make a new one with either version here.

Comparing axis 1 results. CORE V1.0 and CORE 0.17.1 run different sets of rules, so the same data can get different axis 1 counts from each. Results can also differ between two downloads with the same version number, and the CORE report does not name the build. Compare two axis 1 results only when they name the same CORE version and the same rule set fingerprint. A different fingerprint means the rule sets differ. The same fingerprint does not show that two runs used the same engine code, because the fingerprint does not change when only the engine's code changes.

Set up CORE V1.0

  1. Install Python 3.12 from the Python 3.12.10 page. Under Files, choose Windows installer (64-bit) or macOS 64-bit universal2 installer. On Windows, tick Add python.exe to PATH in the installer. Later 3.12 releases do not have installers. CORE V1.0 does not support Python 3.11 or older, or 3.13 or newer. You can keep other versions of Python installed. On Linux, see the CORE V1.0 instructions.
  2. Open the CORE V1.0 release page, which is on GitHub. Scroll down to Assets and download Source code (zip). Unzip it to a short path, such as C:\core-v1. In our test on Windows, the install in step 4 failed from a long folder path. The download includes the rules, and no GitHub account or CDISC Library key is needed.
  3. Find the folder that contains core.py, which is one level inside the unzipped folder, and open a command window there. On Windows, open that folder, type powershell in the address bar and press Enter. On macOS or Linux, open Terminal, type cd and a space, drag the folder onto the Terminal window, and press Enter.
  4. Run these two commands, one at a time. The first makes a folder called venv, a private copy of Python for CORE. The second downloads the software CORE needs into it. You do not repeat them for the next study.

    On Windows:

    py -3.12 -m venv venv
    .\venv\Scripts\python.exe -m pip install -r requirements.txt

    On macOS or Linux:

    python3.12 -m venv venv
    ./venv/bin/python -m pip install -r requirements.txt
  5. Go to Make the report.

Set up CORE 0.17.1

  1. Open the CORE 0.17.1 release page, which is on GitHub. Scroll down to Assets and download the file for your computer: core-windows.zip for Windows, core-mac-apple-silicon.zip or core-mac-intel.zip for a Mac, or core-ubuntu-latest.zip for Linux. Unzip it. The download includes the rules, and no GitHub account or CDISC Library key is needed.
  2. Open a command window in the unzipped folder. On Windows, open the folder that contains core.exe, type powershell in the address bar and press Enter. On macOS or Linux, open Terminal, type cd and a space, drag the unzipped folder onto the Terminal window, and press Enter.
  3. Go to Make the report.

Make the report

Before you run the command.

  • Use full paths, which start from the drive or the top folder, as the boxes ask. In our use of CORE 0.17.1, it did not read datasets through a relative path (one that starts from the current folder).
  • For CORE V1.0, the folder that holds the datasets must hold XPT files, with one file for each dataset, named after it (ae.xpt). CORE V1.0 skips other formats, such as Dataset-JSON, with a warning.
  • On macOS, if a security warning stops CORE 0.17.1 from opening, its quick-start guide gives this fix, run in its folder:
    xattr -rd com.apple.quarantine .
    chmod +x ./core
  1. Choose the CORE version you set up and your computer, then fill in the boxes below.
  2. Copy the command, paste it into a command window opened in the CORE folder, and press Enter. CORE V1.0 took up to about four and a half minutes on a 100-subject study in our tests.
  3. CORE saves core-report.json in the folder you chose. Go back to the scorer, add that file with the study files, and score again.

If the command does not work, you can still score the study without the report. To ask for help, send the error message through the contact form.


       
    

What stays on your computer

The scorer page reads your files in your browser and shows the results there. It does not send the files, or anything about them, to Clinventive or to anyone else.

The page is served with a security policy that tells the browser to block the ways a page sends data in the background: uploads, data requests and connections (connect-src 'none'), and form submissions (form-action 'none'). The page loads nothing from other sites, no fonts, images or scripts, and its own code makes no requests after the page has loaded. To check this yourself:

  1. Open the scorer, then open your browser's developer tools (F12 on Windows, or Option-Command-I on a Mac) and choose the Network tab.
  2. Score a study. No new requests appear in the list while the files are read and scored.

The code the page runs is the same scoring code as the command-line version, served as readable files rather than compressed into one, at /fidelity/score/lib/.

Questions

Can I score real study data?

The scorer reads SDTM datasets in the formats listed above, whatever their source, and the files stay on your computer. The axes were written to judge synthetic data.

Why is there no overall score?

A single number would have to trade a structural defect against a measure of realism, and any weighting of the two would be arbitrary. Each axis is reported on its own instead.

What does "draft 0.3" mean?

The method is a draft and may change. Each result names the versions it used, so you can tell which method produced it. The method explains each axis, and the scoring specification says exactly what is computed. Both are open for comment.

Where do the reference ranges for axis 4 come from?

From a reference bundle that ships with the scorer. Its version is printed with every result.

Is there a version for scripts and automated pipelines?

A command-line version runs the same scoring code. It is not published yet.

Who made this?

Clinventive, LLC. The seven axes were proposed by Ward Stetson Line at the 2026 CDISC US Interchange. He co-founded and led the CDISC SDS Conformance Rules team.

How do I send a question or a comment?

Use the contact form on clinventive.com.