Synthetic Study Data Fidelity Benchmark · draft 0.3

Synthetic Study Data Fidelity Benchmark: the scoring specification

Draft 0.3, published by Clinventive, LLC, for comment. The method says why each axis exists; this specification says exactly what is computed, so that two independent implementations produce the same numbers from the same inputs. The Fidelity Scorer implements its checks.

Changes in draft 0.3.

Changes in draft 0.2.

Output format. The checks below are what scorer 0.4.0 computes, and each axis's "Reported" list gives what it emits. Each check is reported as a group with its key, how many checks were answerable, how many failed, the rate, the not-run count with its causes, and up to ten examples. A "Reported" row named per_check or per_family lists the axis's groups, whose fields are named key (the check or family), answerable, violations, rate, and notEvaluable. Counts are emitted in the axis's population, versions and other text in its meta, and per-item rows (per class, per test, per measure, per rule, define findings) in its tables. The scorer's JSON output is the score object in §9.

0. Rules that bind every axis

R1. Three outcomes, never two, and this applies to an individual check, not to an axis. A single check on a single record resolves to pass, fail, or not evaluable: does this reference resolve, is this date before that one, is this term in that codelist. Those are binary; no threshold is needed to decide them. A check is not evaluable when its inputs were not supplied, and not-evaluable is never counted as either a pass or a fail; it is reported in its own column with the reason. A score always reports its not-evaluable count.

An axis never resolves to pass or fail. Axes report rates and inventories. How to read those rates is §0b, which exists because a rate on its own does not tell a reader whether it is good or bad.

R2. The denominator travels with the number. Every reported rate carries the count it was computed over. No rate is published without one.

R3. Population is reported alongside every axis. The counts each axis was computed on, so a sparse dataset reads as sparse. If a rate improves while population falls, both numbers are visible and the reader can see what happened.

R4. Absent input degrades, never fails. A domain that was not supplied makes checks into it not-evaluable. It does not make them failures. Counting absence as failure here would double-count axis 7, which measures scope.

R5. No composite. Axes are reported side by side. The scorer emits no total and no weighting, and the reference implementation has no code path that produces one.

R6. Determinism. Same inputs, same method version, same numbers. No sampling, no randomized subsetting, no time-dependent behavior.

R7. Every not-evaluable check says why (added 0.2.0-draft). Each group reports its not-evaluable count broken down by cause, and the causes sum to the count. Each cause has one of six kinds, shown to readers under the label in brackets: nothing_to_check (Nothing to check: the study has no records the check applies to), not_supplied (Not supplied: a dataset or file the check reads was not supplied; an absent dataset is always this kind, because the scorer cannot tell "the study has none" from "it was left out"), missing (No value: a record lacks a value the check needs, or the value cannot be read), imprecise (Dates too imprecise: a date is not precise enough for the check), declined (Not run by design: the method does not run the check in this case, for example no cited limit, a subject outside the reference ranges, or too few results), and engine (CORE did not run it: CORE skipped the rule for a reason of its own, or failed to run it). A group that did not run at all says so and why, instead of reporting zeros. A record the check does not apply to (a disposition that attributes nothing, a dataset-level RELREC row with no subject) is not counted as not-evaluable.


0b. How a rate is read: the three interpretation modes

The objection this answers: pass, fail, and not evaluable, with no further qualification, do not say where the boundary is or what percentage counts as success.

The objection is fair to draft 0.1. A rate such as referential_integrity = 0.983, reported alone, does not tell the reader whether that is excellent or disqualifying. A fixed cutoff does not solve this: a benchmark that declares 99% a pass and 98% a fail has no basis for the boundary, and it gives every tool a number to optimize against.

The resolution is that the axes are not all the same kind of measurement, and each states which kind it is. Every axis in §2–§8 carries a mode field, and the mode is what tells a reader how to read the number.

Mode A: Defect class. The target is zero.

For these, a violation is not a matter of degree: it is a defect that can surface downstream, and the answer to "what percentage is acceptable?" is none. A subject reference that does not resolve, an end date before its start date, and a coded value absent from its own codelist are each a defect.

Applies to: referential integrity (all four classes) · temporal ordering, study bounds and visit dates · reasonable range · physically impossible values · term validity · unit validity · precision consistency · cross-domain consistency · structural conformance failures.

Mode B: Distributional. No target is set.

For these there is no correct value in the abstract. The right intraclass correlation depends on the analyte. The right dropout curve depends on the indication and the study length. Real studies have late and out-of-window visits, so perfect adherence is not a goal.

Applies to: intraclass correlation and its log-scale form · trajectory shape · missingness pattern · dropout curve · visit window adherence · reference interval excursion.

Mode C: Inventory. Present, absent, or partial.

Nothing to threshold. Elements are not interchangeable, so they are listed rather than averaged.

Applies to: deliverable completeness.

What this changes about the "no composite" decision

Nothing. Refusing a total is not refusing to interpret. Mode A says plainly that non-zero is a defect. Mode B says plainly that the number means nothing alone and supplies what it should be read against. What the method still declines to do is collapse seven measurements of different kinds into one figure, because a total would have to weight a definitional defect against a distributional statistic, and no weighting of those two is defensible.


0c. Within-tool variance: why one study is not a score

One study is not enough to score a tool. Score any tool on one study and there is no way to tell whether the result is a property of the tool or of that study, or whether a gap between two tools means anything. If a tool scores 0.94 on one study and 0.89 on another, a second tool's 0.91 is inside that range, and the two cannot be ranked.

So, when results for a tool are reported:

This costs only running the scorer more than once.


1. Inputs

Input Required Used by
Datasets (XPT, Dataset-JSON, or CSV) yes all
CORE report (JSON) no (axis 1 is not scored without it) 1
Dataset-describing metadata (define) no (degrades) 4, 7
Visit schedule (TV), and visit windows from a facts file no (degrades) 2, 3, 5, 7

Declared scope. The define (or a facts file) says which datasets the study contains. Axis 7 compares that declaration with the datasets supplied; axes 1–6 measure what was supplied. A study declared with one domain is scored on that domain, and the narrow scope shows on axis 7.


2. Axis 1: Structural conformance

Inputs: datasets, declared standard/version, define if supplied, and a conformance report.

Design decision surfaced during implementation (2026-08-26). The official conformance engine does not run in a browser, and the specification forbids reimplementing it: a benchmark score and a product claim must not be able to disagree, which they would the moment two implementations of the same rules existed. Those two constraints leave one option: axis 1 is scored from a conformance report supplied with the study, produced by the official engine, which is freely available. Without a report the axis is not_evaluable with that reason, per R4. It is never estimated, and never approximated by a partial reimplementation.

The browser scorer and the command-line version both take the report as an input. This keeps the promise that nothing is uploaded, and keeps the rule corpus in one place.

Computation. Read each rule's status from the CORE report: a rule CORE passed is pass, a rule that reported issues is fail (with its count of findings), and a rule CORE skipped or could not run is not_evaluable. CORE V1.0 says why it skipped a rule, and the scorer reports that: a rule that applies to no dataset in the study is of kind nothing_to_check, a rule that needs a dictionary or file that was not given is not_supplied, and any other reason is engine, with CORE's words. CORE 0.17.1 does not say why, so its skipped rules are engine with no reason.

Reported.

rules_total, rules_pass, rules_fail, rules_not_evaluable per_class[]: { class, total, pass, fail, not_evaluable } findings_total, findings_per_1000_records meta: engine_name, engine_version, rule_corpus_version, rule_set_fingerprint, comparison, standard, ct_version, rule_classes

Notes. A rule-class breakdown is reported because an aggregate pass rate hides a tool that fails every rule of one kind. The engine's report carries no rule classes, so each rule is classed by the Rule Type its own CORE definition declares, read from the rule caches shipped with the CORE releases the scorer was built against (0.17.1 and 1.0.0); a rule those caches do not hold is unclassified. Most rules are of one type, Record Data (833 of the 981 rules in the CORE 0.17.1 rule cache, and 394 of the 430 in an SDTMIG 3.4 report), so the breakdown is uneven by construction. The report does not name a rule-corpus version, and the engine's rule cache can be updated without changing the CORE version, so rule_corpus_version is reported as not stated. Findings-per-1000-records accompanies the rule counts because a single rule failing on every record and a single rule failing once are not the same result.

rule_set_fingerprint. The report names the CORE version and no build. The fingerprint is computed from the report's rules:

  1. For each rule, one line <ID>@<version>, with the ID upper-cased, or <ID>@ when the report gives the rule no version.
  2. The lines sorted in character-code order and joined with a newline (\n), with no trailing newline.
  3. That text encoded as UTF-8 and hashed with SHA-256. The fingerprint is the first 16 hexadecimal characters of the hash.

It changes when the set of rules or any rule's version changes. It does not change when a rule is edited without a new version, or when only the engine's own code changes. A different fingerprint means the rule sets differ. The same fingerprint does not show that two runs used the same engine code. Where the report gives no rule versions, the meta text says so.

comparison states that axis 1 counts are compared only between runs with the same CORE version and the same fingerprint. Different CORE releases, and different builds of one release, can run different rules or run them differently. Their counts on the same data can differ.


3. Axis 2: Referential integrity

Inputs: datasets.

Four relationship classes, reported separately.

Class Check
subject_presence Each subject identifier appearing in any domain resolves to a record in the subject-level domain
visit_reference Each visit reference resolves to a visit the schedule (TV) defines, or to an unplanned visit the subject's SV records hold
parent_child Each supplemental qualifier record's parent record exists
declared_relationship Each explicitly declared relationship resolves at both ends

Computation. Per class: resolved / answerable, where answerable excludes references into domains that were not supplied (R4). Those are counted not_evaluable with the missing domain named.

Unplanned visits. TV lists only the planned visits. A visit reference that is not in TV resolves when SV has a record for the same subject and the same visit (VISIT, ignoring case) whose SVPRESP is not Y. The SV record itself is counted as resolved. These references are counted as resolved and are also listed in the group's breakdown as unplanned_visit:<DOMAIN>. A visit reference that is not in TV is unresolved when no SV record holds it for the subject, when the SV record marks it planned (SVPRESP = Y), or when SV was not supplied. SVPRESP is an SDTMIG 3.4 variable, so in SDTMIG 3.2 or 3.3 data, where SV has no SVPRESP, every visit SV holds that TV lacks resolves.

Reported.

per_class[]: { class, answerable, resolved, unresolved, not_evaluable, missing_targets, missing_targets_total } records_per_domain[]: { domain, records } population: subjects_in_subject_domain, visits_defined, unplanned_visit_records

Notes. missing_targets names the unresolved values, at most ten, and missing_targets_total gives how many there were. A reader should be able to tell "one subject is missing everywhere" from "every subject is missing one visit."


4. Axis 3: Temporal coherence

Inputs: datasets; schedule if supplied.

Four check families, reported separately.

Family Check Answerable when
ordering A record's end (--ENDTC) is on or after its start (--STDTC) Both dates present and parseable
study_bounds Activity falls within the subject's own participation period: DM.RFICDTC (informed consent) to DM.RFPENDTC (end of participation) The bound on that side was recorded for the subject
visit_dates A finding's collection date (--DTC) falls on or between its visit's SVSTDTC and SVENDTC (calendar days, inclusive; SVSTDTC alone when there is no end), matched by VISITNUM, else VISIT SV is supplied, the visit has an SV record with a start date, and the finding has a date. A finding at a visit with no dated SV record is not evaluable (missing)
window_adherence Visit dates fall within declared windows A schedule with windows was supplied (SDTM has no numeric window variables, so windows come from a facts file)

Computation. Per family: violations / answerable. Partial dates are answerable only to the precision both endpoints share; a comparison that the shared precision cannot decide is not_evaluable, not a pass.

study_bounds, as decided 2026-10-20 (§10 item 5). The participation period is DM.RFICDTC to DM.RFPENDTC, the SDTMIG variables for the date of informed consent and the date of end of participation. The reference period RFSTDTC–RFENDTC is not used as a bound. A missing bound is not replaced by another date: the side it would have checked is not evaluated for that subject, and the number of subjects affected is reported. Dates in MHSTDTC, MHENDTC, CMSTDTC, CMENDTC, PRSTDTC, PRENDTC and GFDTC that fall before consent are counted and reported, not scored, because these variables can record events or specimens from before the study. A history date after the end of participation is still a violation.

The bounds are inclusive calendar days. A record on the consent day or on the end-of-participation day is inside the period whatever its time of day: a first-day record can carry a time before the consent time, and that is not a violation. Where either side is coarser than a day (a month-only date against a full consent date), the shared-precision rule above still applies and an undecidable comparison is not_evaluable. Time of day is used at the epoch boundary instead. Dates inside the period and before first dose (RFSTDTC) are reported as the pre-treatment share and are never a defect; a record on the first-dose day counts as pre-treatment when both it and RFSTDTC carry a time and its time is earlier, and is reported as first_dose_day_untimed when either lacks one.

Reported.

per_family[]: { family, answerable, violations, rate, not_evaluable, breakdown? } population: dates_full, dates_partial, dates_missing meta: study_bounds_period (the bounds and the history variables used, as text)

breakdown for study_bounds counts each date by kind and place: before_consent:<DOMAIN>.<VAR> and after_participation_end:<DOMAIN>.<VAR> (the violations), history_before_consent:<DOMAIN>.<VAR>, before_first_dose:<DOMAIN> and first_dose_day_untimed:<DOMAIN> (reported, not scored).

Notes. window_adherence reports its rate without a target. Real studies have late and out-of-window visits, so the methodology does not reward 100% adherence (§0b).


5. Axis 4: Clinical plausibility

Inputs: datasets; reference bundle; define if supplied.

Six checks. (physiological_range was superseded by the decision of 2026-08-26. This table matches what the scorer computes.)

Check Mode Computation
reasonable_range A Per test code: proportion of results inside an explicitly cited plausibility bound (§5.2). Explicitly not the clinically-normal range: abnormal results are expected in a trial and are not defects
physically_impossible A Proportion of results that cannot be a measurement of the quantity at all (§5.3). Definitional, not clinical
reference_interval_excursion B How often results fall outside the published reference range. No target: a dataset in which nothing is ever abnormal is as implausible as one in which everything is. A range published for fasting samples is used only for results whose record says they were fasting (§5.4)
unit_validity A Proportion of results whose unit is admissible for that test. Conventional and SI units are both admissible where the bundle states the conversion between them
term_validity A Proportion of coded values present in the codelist the supplied define assigns. Where the define gives value-level metadata for the variable, a record is held to the codelist of the entry whose condition (EQ or IN) it meets, and to none if that entry assigns none; otherwise to the variable's codelist. Exact match, since submission values are case-sensitive. A value coded to an external dictionary (MedDRA, WHODrug) is declined, not passed
precision_consistency A Proportion of results whose precision is consistent within their series: a test code in one dataset, and in PP one analyte (PPCAT). A result is consistent when it has the series' most common number of decimal places or its most common number of significant figures (§5.4)

5.1 Interval entries and threshold entries

Added 2026-09-14. A reference entry declares what its source actually published:

kind Meaning reasonable_range reference_interval_excursion
interval (default) An interval estimated from a healthy reference population. The central-95% convention applies to its width declines unless a cited bound is present scored
threshold A decision threshold or diagnostic category boundary, such as a guideline treatment target, a category cut-point, a regulatory limit declines unless a cited bound is present scored

Superseded 2026-09-15 (see open item 7). kind no longer decides mode-A eligibility; a cited bound does, for both kinds. kind remains required reading because it records what the source published, which is what makes the excursion reading interpretable and the classification contestable.

The distinction exists because widening a threshold as though its width implied a standard deviation produces bounds that are wrong in both directions. Measured on the transcribed vitals and ECG entries: an ACC/AHA treatment target of 90–130 mmHg derives to 69.6–150.4, so a hypertensive subject at 160 scores as a data defect; a WHO saturation threshold of 95–100% derives to a ceiling of 102.6%, so a physically impossible 101% passes. Fever, obesity and QT prolongation fail the same way.

A declined row is not_evaluable with the reason and the test codes named. It is never omitted and never counted as conforming. Declined entries remain fully scored in the mode-B excursion reading, which is what such sources are published for.

5.2 Only a cited bound scores in mode A

Decided 2026-09-15, closing open item 7. reasonable_range scores a result only where its entry carries an explicit plausibleLow/plausibleHigh and a plausibleSourceCitation. A bound is never derived from a reference interval for scoring purposes.

The reason is that a healthy-population interval does not describe a patient population. Widened by the default multiple, it still marks ordinary trial findings as data defects. An oral-temperature interval of 36.1–37.2 °C widens to 35.5–37.8, so a fever of 38.5 is scored as a defect; a fasting-glucose interval widens in the same way. No multiple can be defended for every test, so none is used.

An explicit bound is therefore a sourced clinical value in its own right, and rule 1 applies to it on its own account: a bound without its own citation does not load. The entry survives for the excursion reading and the uncited bound is reported as dropped.

The derivation of a bound from a reference interval remains in the code, unused by scoring.

This result is intended. Mode-A range coverage starts near zero and grows only as citable bounds are added. Rule 3 says coverage is reported, not maximized, and plausibility is scored only where a bound can be cited.

kind is decided by what the source publishes, not by clinical judgment about values, and each entry carries a kindBasis note so the call can be contested. Classify conservatively: calling an interval a threshold costs coverage, which is reported; calling a threshold an interval invents defects in real data, which the reader cannot detect.

5.3 Definitional limits: physically_impossible

Added 2026-09-15. A separate mode-A group that asks whether a number can be a measurement of its quantity at all, rather than whether it is clinically usual. A negative mass. A saturation above full. Violations are data errors, which can come from a unit confusion, a transposed digit, a sentinel written into a numeric field, so the target is zero and it needs neither a citation nor a clinical opinion.

Limits come from two places:

Source Supplies Requires
The unit (physical.ts, published with the code) Non-negativity only, and only for units denoting a non-negative magnitude: length, mass, area, volume, duration, rate, absolute pressure, concentration, catalytic activity, cell count nothing; it follows from the unit
The entry (physicalLow / physicalHigh) Ceilings and anything test-specific a physicalBasis stating why the limit holds. Without one the bound does not load

Two units are deliberately absent. Celsius and Fahrenheit are interval scales on which negative values are meaningful, so no bound follows from the unit. And % carries no definitional ceiling: a percent-predicted spirometry value can exceed 100 and a percent change can be negative, so a blanket rule on the unit would invent defects in both. An oxygen saturation is capped at 100, but that is a property of the quantity, declared on its entry, not of the percent sign.

A quantity with no known limit is not_evaluable with that reason. Nothing is guessed.

This is the group that catches what the old derived range let through: a saturation of 101% passes every clinical-range test and is impossible.

5.4 Age, fasting status, units and precision

Age. An entry's ageLow and ageHigh are in years. A subject's age is DM.AGE read with DM.AGEU: YEARS as given, MONTHS divided by 12, WEEKS times 7 / 365.25, DAYS divided by 365.25, HOURS divided by (24 x 365.25). An age with no AGEU, or with a unit outside that list, is not known. An entry limited by age applies only to a subject whose age is known and inside its limits; a subject whose age is not known is never assumed to be an adult. In bundle 0.3 every entry except oxygen saturation (OXYSAT) carries ageLow 18, and the laboratory entries carry an ageBasis saying why. A result with no applicable entry because the age is not known is not_evaluable, kind missing. A result whose subject is outside every entry's ages or sex is not_evaluable, kind declined.

Fasting. An entry with fastingOnly describes fasting samples only. In bundle 0.3 that is glucose, whose cited range is the fasting plasma glucose range. A result is compared with such an entry only when its --FAST (LBFAST) is Y. With --FAST blank the comparison is not_evaluable, kind missing; with any other value it is not_evaluable, kind declined. The scorer does not check the specimen type (LBSPEC), so the plasma glucose range is applied to any GLUC record marked fasting.

Reading the excursion share. meta.reference_interval_reading states how the share is read. The share describes the data and is not a pass or fail target. The ranges are published adult reference ranges and limits. Blood pressure (SYSBP, DIABP) is compared with a normal adult resting range of 90/60 to 120/80 mmHg (MedlinePlus). Glucose is compared with its fasting range only for fasting samples.

Units. Bundle 0.3 adds SI units, with their conversions, for creatinine, uric acid, urea nitrogen, cholesterol, triglycerides, sodium, potassium, chloride, bicarbonate, calcium, phosphorus, magnesium, iron, total protein, MCHC, ferritin, platelets, neutrophils, lymphocytes, eosinophils, and the enzymes ALP, GGT, LDH and CK. Each conversion states its factor and why the factor holds: an identity (mEq/L and mmol/L for a monovalent ion), a decimal prefix, a definitional SI relation (1 ukat/L = 60 U/L), or a molar mass from standard atomic weights, which the conversion names. Triglyceride is a mixture, so its factor uses the molar mass of triolein. A result is converted into the entry's unit before any comparison; a unit with no stated conversion is not_evaluable, kind declined, never compared across units.

Precision. For each series, the scorer finds the most common number of decimal places and the most common number of significant figures, the latter taken from results whose count is unambiguous. A whole number ending in zeros (120) has an ambiguous count, two or three, and matches when the most common count is either. A zero has no significant figures and is judged by decimal places alone. A result fails only when it matches neither.

Computation. Test codes absent from the reference bundle are not_evaluable and named. Coverage (what proportion of supplied results the bundle could judge) is reported alongside, because an apparently perfect score over 5% of the data is a different claim from one over 95%.

Reported.

per_check[]: { check, answerable, violations, rate, not_evaluable } population: tests_supplied, tests_in_bundle, results_judged, results_total (reference coverage) per_test[]: { test, unit, n, in_reference_bundle, judged_against_reference, outside_reference, outside_reference_rate, outside_plausibility_limit, physically_impossible, unit_not_accepted, precision_mismatches } meta.reference_bundle_version <- the bundle's version string meta.reference_interval_reading <- how the excursion share is read (§5.4) population.reference_bundle_entries <- how many entries it carried

per_test has one row per dataset and test code (LB.ALT). unit is the standard unit most of its results carry. In population, results_judged is the number of results compared with a reference range.

population is numeric by contract, so the bundle version travels in the axis's meta map instead. Both are reported: the version identifies which bundle produced the numbers, and the entry count says how many entries it had. The command-line version accepts another bundle, and this is what stops two scores computed against different bundles from being mistaken for comparable.

Notes. per_test is reported because an aggregate hides a single badly parameterized analyte.


6. Axis 5: Longitudinal realism

Inputs: datasets with repeated measures per subject.

The primary statistic is the intraclass correlation. For each repeatedly measured quantity, decompose variance into within-subject and between-subject components and report the ICC.

This is the axis's discriminator: a generator that redraws each visit independently from one shared distribution produces an ICC near zero, however plausible each value is. A share of total variance is expected to be between-subject for many clinical measurements, but the expected share varies by analyte and for some it is low. Hence the reporting rule below: per measure, with no target attached.

Three further measures.

Measure Computation
trajectory_shape Number of subject series that are monotone, mean-reverting, or unstructured (pooled across measures in the group, per measure in the table)
missingness_pattern Subjects whose visits stop and do not resume, have intermittent gaps, or are complete
dropout_curve Subjects with a record at each scheduled visit, reported as a series

Reported.

per_measure[]: { measure, subjects, subjects_with_repeats, observations_per_subject_median, icc, icc_ci95_low, icc_ci95_high, icc_log, icc_log_ci95_low, icc_log_ci95_high, icc_log_not_run } trajectory_shape[]: { measure, monotone, reverting, unstructured, too_short } population: measures_with_icc, measures_with_log_icc, missingness_terminal, missingness_intermittent, missingness_none dropout_curve[]: { visit, retained, n } meta: icc_estimator, icc_log, icc_not_run_by_design, measure_key

Measures. A measure is a dataset and test code (LB.ALT). In PP it is also the category (PP.CMAX/<PPCAT>), because PP can hold parameters for several analytes under one test code, and pooling an analyte reported in ug/mL with one reported in ng/mL would put two scales in one series. PK concentrations (PC) get no ICC, because they are sampled across each dosing interval and change with the time since the dose. The group is not_evaluable, kind declined, with that reason.

The estimator. ICC(1), the one-way random-effects form, computed from the subjects with at least two results for the measure. Where subjects have unequal numbers of results, k, the number of results per subject, is replaced by k0, the mean number corrected for the inequality: k0 = (N - sum of n_i squared / N) / (s - 1), for s subjects and N results. The 95% interval comes from the F distribution of the ratio of the between-subject to the within-subject mean square (MSB/MSW), as Shrout and Fleiss (1979) give it for ICC(1,1), with k0 in place of k; with unequal numbers of results the interval is approximate. A measure with fewer than two such subjects, or whose results do not vary, has no ICC and no interval, and both are reported as null. A measure whose results vary between subjects but never within one has an ICC of 1 and no interval.

The log-scale ICC. icc_log is the same estimator, with its interval, computed on the natural logarithm of each value. It is reported beside icc and changes nothing about it. It is computed only when the ICC is computed and every value it uses is above zero. When it is not computed, icc_log and its interval are null and icc_log_not_run says why: the measure has no ICC, a value is zero or below, the logarithms do not vary, or the measure is a PK concentration. On the log scale a very high value is closer to the rest, so a few very high values beside ordinary results tend to lower icc_log less than they lower icc. A planned change that every subject goes through, such as a treatment nadir, still lowers it.

Notes. ICC is reported per measure and never averaged across measures, because heart rate and hemoglobin are different quantities. No target value is published, because the expected ICC depends on the analyte; a reader who knows the measure judges it.


7. Axis 6: Cross-domain clinical consistency

Inputs: datasets across two or more domains.

Each relationship rule is published individually, in this form:

id, statement, antecedent_domain, consequent_domain, answerable_when, agreement_when

Draft 0.3 rule set (0.3.0), deliberately small: it holds only relationships that can be stated precisely enough for two readers to agree.

id Statement
CD-01 A treatment recorded for an indication (CMINDC) has that indication present in the subject's own history or event records (MHTERM, MHDECOD, AETERM, AEDECOD). Rule set 0.3.0: the comparison ignores only case, extra spaces and punctuation (below)
CD-02 A discontinuation attributed to an adverse event (DSDECOD or DSTERM) has an adverse event recorded for the subject, starting on or before it (calendar days)
CD-03 Study treatment withdrawn because of an adverse event (AEACN = DRUG WITHDRAWN, treatment permanently stopped for that event) is not given after that event ends: the subject's last exposure (latest EXENDTC, else EXSTDTC) is on or before AEENDTC. An event with no end date is not evaluable (missing). Rule set 0.2.0
CD-04 An event term is compatible with the subject's recorded characteristics (a condition of one sex not recorded for another)

Computation. Per rule: violations / answerable, where answerable requires both sides present.

CD-01 matching. Both sides are upper-cased. An apostrophe (' or ’) is removed, so CROHN'S and CROHNS match. Every other Unicode punctuation character becomes a space, so GASTRO-OESOPHAGEAL and GASTRO OESOPHAGEAL match. Symbols such as + < = ^ are kept. Runs of spaces become one space, and leading and trailing spaces are dropped. The two must then be equal. CD-01 checks that the condition a treatment was given for is recorded, and it is not run for an indication containing the word PROPHYLAXIS, PROPHYLACTIC, PREVENTION or PREVENTIVE. Such a record is not_evaluable, kind declined.

CD-01 near matches. An unmatched indication is compared with each of the subject's terms. A term is a near match of one of three kinds:

Terms are tried in alphabetical order of their match form, and for each term the kinds are tried in the order listed. The first hit is the near match. The scorer does not judge whether the two terms mean the same thing. The record is still a violation, and the near match is listed so a reader can see the likely cause.

Reported.

per_rule[]: { id, answerable, agreements, violations, rate, not_evaluable } near_matches[]: { indication, nearest_term, how, records } population: cd01_near_matches meta: rule_set_version

The CD-01 group's breakdown counts the near matches by kind, as near_match:<how>. A near-match row holds the two terms and a count, and no subject identifier.

Notes. The rule set is expected to change through comment, and every change is versioned. A rule that commenters dispute stays published, with the dispute noted, so that readers can see it.


8. Axis 7: Deliverable completeness

Inputs: everything supplied, plus the declared scope.

An inventory, not a percentage. Each element is present, absent, or partial, and the elements are not interchangeable so they are never averaged.

Element Present when
domains_by_class Domains supplied, grouped by observation class, against the declared scope
dataset_metadata A define supplied
define_consistency The supplied define and the data agree (below). partial when there is any finding; not evaluable without a define
value_level_metadata Value-level metadata present in the define
trial_design Trial-design information supplied

Reported.

inventory[]: { element, state, detail } declared_scope, scope_met consistency_findings[]: { check, dataset, variable, count, detail }

define_consistency compares the datasets and variables the define declares (each ItemGroupDef and the ItemDefs it references) with the datasets supplied. Each finding names its check, the dataset, the variable where there is one, and how many values it covers:

Check Finding
dataset_not_in_define A supplied dataset the define does not declare
variable_not_in_define A column in a supplied dataset that the define does not list for it
variable_not_in_data A variable the define lists that the supplied dataset has no column for
type_mismatch Values that are not whole numbers in a variable the define declares integer, or not numbers in one it declares float
length_exceeded Text values longer than the variable's declared Length

A declared dataset that was not supplied is the declared_scope element's finding, not this one's. A supplied dataset with no records has no columns to compare. Date and time variables are not checked against their DataType here, because the Define-XML date types differ by precision and the partial-date comparison belongs to axis 3.


9. The score object

The scorer's JSON output (the command line's --json, and the results file the browser page saves). It is built where the scoring runs and is not sent anywhere; no service receives score objects. The score object contains no data values and no record-level content.

method_version, scored_at_utc, scorer_version tool: { name, version, declared_by } study: { label, provenance, therapeutic_area, phase, subjects, duration_visits } inputs: { formats[], domains[], subjects, records_total, define_supplied, schedule_supplied } axes: { axis_1..axis_7 as specified above, each carrying its mode: A | B | C, and warnings where there are any } population: { populated_cells_by_domain[] } content_hash

An axis may carry warnings, a list of plain sentences a reader should see before the axis's numbers, for example, a sentence giving the number of rules CORE could not run. A warning restates a count the axis already reports and changes no count. The field is absent when an axis has no warning.

tool is what the person scoring states about the tool that made the data; declared_by is person scoring when they state it and null when they do not. The scorer cannot check it.

study.provenance is one of authoring_system_import, standards_native_definition, synthesized_without_source, or other: the path the study design arrived by. It is null when the person scoring does not state it, and such a score cannot take part in a comparison across provenances, because §0c depends on it: a spread across provenances is what distinguishes a property of the tool from a property of one study. label is a name the person scoring the study chooses and identifies no person. therapeutic_area and phase are the TS values for THERAREA and TPHASE when TS supplies them, subjects counts the subjects in DM, and duration_visits counts the planned visits in the schedule. populated_cells_by_domain gives, for each dataset, the cells holding a value and the cells in all.

Each axis carries its mode, so a consumer of the score object can render Mode A results as defects and Mode B results as comparisons without re-deriving which is which.

content_hash is a hash of the score object, not of the data, and it identifies nobody. It is the SHA-256 of the object's canonical JSON (every object's keys sorted, no whitespace) with scored_at_utc and content_hash left out, so the same inputs and versions give the same hash whenever they are scored (R6). Recomputing it shows whether a saved score was edited after it was computed. It cannot show that the inputs were what the label says.


10. Open items in this specification

The body refers to these by number.

Open:

Closed: