Changes in draft 0.3.
- Axis 1: each result carries a fingerprint of the rules in the CORE report and a note on comparing results (§2).
- Axis 2: a visit that is not in TV resolves when SV holds it as an unplanned visit; such references are reported separately (§3).
- Axis 3: prior procedure dates (PRSTDTC, PRENDTC) and specimen collection dates (GFDTC) before consent are reported, not scored (§4).
- Axis 4: a reference range applies only to subjects of the ages its source covers; age is read with AGEU; the glucose range is used only for fasting samples; SI units and their conversions are added; precision accepts consistent significant figures (§5.4).
- Axis 5: a log-scale ICC is reported beside the ICC; PK concentrations get no ICC; PP parameters are kept apart by PPCAT (§6).
- Axis 6: CD-01 ignores punctuation, is not run for prophylaxis, and lists near matches, which are still counted as unmatched. The rule set is now 0.3.0 (§7).
- Score object: each axis may carry warnings (§9).
Changes in draft 0.2.
- The benchmark scores data and its metadata only. The two program elements of axis 7 and the program input are removed.
- Axis 3:
interval_plausibility, which had no defined span list, is replaced byvisit_dates, a check that follows from what SV and--DTCrecord. - Axis 3:
orderingcounts a start and an end on the same calendar day as in order. - Axis 4:
term_validityreads the codelists the supplied define assigns, value-level first. - Axis 5: the ICC estimator and its interval are stated (§6).
- Axis 7:
define_consistencycompares the supplied define with the data (§8). - Output: scorer 0.3.0 emits every field named in the "Reported" lists, inside the score object of §9.
- Axis 6: CD-02 compares dates by calendar day, so an event that starts on the day of the discontinuation counts as preceding it; an order the dates cannot decide is not evaluable.
- Axis 6: CD-03 (a severe event against a related measurement), which needed an event-to-measurement mapping that draft 0.1 never defined, is replaced by a rule that follows from the CDISC meaning of DRUG WITHDRAWN.
- R7: every check that does not run says why, by kind and cause, within its group.
Output format. The checks below are what scorer 0.4.0 computes, and each axis's "Reported" list gives what it
emits. Each check is reported as a group with its key, how many checks were answerable, how many failed, the rate, the
not-run count with its causes, and up to ten examples. A "Reported" row named per_check or per_family lists the
axis's groups, whose fields are named key (the check or family), answerable, violations, rate, and
notEvaluable. Counts are emitted in the axis's population, versions and other text in its meta, and per-item
rows (per class, per test, per measure, per rule, define findings) in its tables.
The scorer's JSON output is the score object in §9.
0. Rules that bind every axis
R1. Three outcomes, never two, and this applies to an individual check, not to an axis. A single check on a single record resolves to pass, fail, or not evaluable: does this reference resolve, is this date before that one, is this term in that codelist. Those are binary; no threshold is needed to decide them. A check is not evaluable when its inputs were not supplied, and not-evaluable is never counted as either a pass or a fail; it is reported in its own column with the reason. A score always reports its not-evaluable count.
An axis never resolves to pass or fail. Axes report rates and inventories. How to read those rates is §0b, which exists because a rate on its own does not tell a reader whether it is good or bad.
R2. The denominator travels with the number. Every reported rate carries the count it was computed over. No rate is published without one.
R3. Population is reported alongside every axis. The counts each axis was computed on, so a sparse dataset reads as sparse. If a rate improves while population falls, both numbers are visible and the reader can see what happened.
R4. Absent input degrades, never fails. A domain that was not supplied makes checks into it not-evaluable. It does not make them failures. Counting absence as failure here would double-count axis 7, which measures scope.
R5. No composite. Axes are reported side by side. The scorer emits no total and no weighting, and the reference implementation has no code path that produces one.
R6. Determinism. Same inputs, same method version, same numbers. No sampling, no randomized subsetting, no time-dependent behavior.
R7. Every not-evaluable check says why (added 0.2.0-draft). Each group reports its not-evaluable
count broken down by cause, and the causes sum to the count. Each cause has one of six kinds, shown to
readers under the label in brackets: nothing_to_check (Nothing to check: the study has no records
the check applies to), not_supplied (Not supplied: a dataset or file the check reads was not
supplied; an absent dataset is always this kind, because the scorer cannot tell "the study has none"
from "it was left out"), missing (No value: a record lacks a value the check needs, or the value
cannot be read), imprecise (Dates too imprecise: a date is not precise enough for the check),
declined (Not run by design: the method does not run the check in this case, for example no cited
limit, a subject outside the reference ranges, or too few results), and engine (CORE did not run
it: CORE skipped the rule for a reason of its own, or failed to run it). A group that did
not run at all says so and why, instead of reporting zeros. A record the check does not apply to (a
disposition that attributes nothing, a dataset-level RELREC row with no subject) is not counted as
not-evaluable.
0b. How a rate is read: the three interpretation modes
The objection this answers: pass, fail, and not evaluable, with no further qualification, do not say where the boundary is or what percentage counts as success.
The objection is fair to draft 0.1. A rate such as referential_integrity = 0.983, reported alone, does not tell
the reader whether that is excellent or disqualifying. A fixed cutoff does not solve this: a benchmark that declares
99% a pass and 98% a fail has no basis for the boundary, and it gives every tool a number to optimize against.
The resolution is that the axes are not all the same kind of measurement, and each states which
kind it is. Every axis in §2–§8 carries a mode field, and the mode is what tells a reader how to
read the number.
Mode A: Defect class. The target is zero.
For these, a violation is not a matter of degree: it is a defect that can surface downstream, and the answer to "what percentage is acceptable?" is none. A subject reference that does not resolve, an end date before its start date, and a coded value absent from its own codelist are each a defect.
- The threshold is zero because it is not a judgment call: it follows from the check being definitional rather than statistical.
- The rate still matters, for magnitude, not for verdict. One unresolved reference and forty thousand are both defects and are not the same result.
- Reported as: the count, the rate, the denominator, and enough detail to tell a systematic failure from a scattered one.
Applies to: referential integrity (all four classes) · temporal ordering, study bounds and visit dates · reasonable range · physically impossible values · term validity · unit validity · precision consistency · cross-domain consistency · structural conformance failures.
Mode B: Distributional. No target is set.
For these there is no correct value in the abstract. The right intraclass correlation depends on the analyte. The right dropout curve depends on the indication and the study length. Real studies have late and out-of-window visits, so perfect adherence is not a goal.
- No threshold is published, ever. A benchmark that declared the correct ICC for hemoglobin would be making up clinical fact.
- Interpretation is by comparison: the value beside other studies scored on the same axis, and beside the spread the same tool shows across different studies (§0c).
- Reported as: the value, its uncertainty where it has one, and the comparison set it should be read against.
Applies to: intraclass correlation and its log-scale form · trajectory shape · missingness pattern · dropout curve · visit window adherence · reference interval excursion.
Mode C: Inventory. Present, absent, or partial.
Nothing to threshold. Elements are not interchangeable, so they are listed rather than averaged.
Applies to: deliverable completeness.
What this changes about the "no composite" decision
Nothing. Refusing a total is not refusing to interpret. Mode A says plainly that non-zero is a defect. Mode B says plainly that the number means nothing alone and supplies what it should be read against. What the method still declines to do is collapse seven measurements of different kinds into one figure, because a total would have to weight a definitional defect against a distributional statistic, and no weighting of those two is defensible.
0c. Within-tool variance: why one study is not a score
One study is not enough to score a tool. Score any tool on one study and there is no way to tell whether the result is a property of the tool or of that study, or whether a gap between two tools means anything. If a tool scores 0.94 on one study and 0.89 on another, a second tool's 0.91 is inside that range, and the two cannot be ranked.
So, when results for a tool are reported:
- The tool is scored across multiple studies, and its results are shown as a spread with the per-study values, never as a single figure.
- Study provenance travels in the score object (§9), because a tool may score differently depending on where the study came from: a design imported from an authoring system, a standards-native definition, or a study synthesized without a source at all. That difference is a finding about the tool, not noise to be averaged away.
- A difference between two tools is only reported as a difference if it exceeds the within-tool spread. Otherwise it is reported as indistinguishable.
This costs only running the scorer more than once.
1. Inputs
| Input | Required | Used by |
|---|---|---|
| Datasets (XPT, Dataset-JSON, or CSV) | yes | all |
| CORE report (JSON) | no (axis 1 is not scored without it) | 1 |
| Dataset-describing metadata (define) | no (degrades) | 4, 7 |
| Visit schedule (TV), and visit windows from a facts file | no (degrades) | 2, 3, 5, 7 |
Declared scope. The define (or a facts file) says which datasets the study contains. Axis 7 compares that declaration with the datasets supplied; axes 1–6 measure what was supplied. A study declared with one domain is scored on that domain, and the narrow scope shows on axis 7.
2. Axis 1: Structural conformance
Inputs: datasets, declared standard/version, define if supplied, and a conformance report.
Design decision surfaced during implementation (2026-08-26). The official conformance engine does not run in a browser, and the specification forbids reimplementing it: a benchmark score and a product claim must not be able to disagree, which they would the moment two implementations of the same rules existed. Those two constraints leave one option: axis 1 is scored from a conformance report supplied with the study, produced by the official engine, which is freely available. Without a report the axis is
not_evaluablewith that reason, per R4. It is never estimated, and never approximated by a partial reimplementation.The browser scorer and the command-line version both take the report as an input. This keeps the promise that nothing is uploaded, and keeps the rule corpus in one place.
Computation. Read each rule's status from the CORE report: a rule CORE passed is pass, a rule that reported
issues is fail (with its count of findings), and a rule CORE skipped or could not run is not_evaluable. CORE V1.0
says why it skipped a rule, and the scorer reports that: a rule that applies to no dataset in the study is of kind
nothing_to_check, a rule that needs a dictionary or file that was not given is not_supplied, and any other reason
is engine, with CORE's words. CORE 0.17.1 does not say why, so its skipped rules are engine with no reason.
Reported.
rules_total, rules_pass, rules_fail, rules_not_evaluable
per_class[]: { class, total, pass, fail, not_evaluable }
findings_total, findings_per_1000_records
meta: engine_name, engine_version, rule_corpus_version, rule_set_fingerprint, comparison, standard, ct_version,
rule_classes
Notes. A rule-class breakdown is reported because an aggregate pass rate hides a tool that fails every rule of
one kind. The engine's report carries no rule classes, so each rule is classed by the Rule Type its own CORE definition
declares, read from the rule caches shipped with the CORE releases the scorer was built against (0.17.1 and 1.0.0); a
rule those caches do not hold is unclassified. Most rules are of one type, Record Data (833 of the 981 rules in the CORE
0.17.1 rule cache, and 394 of the 430 in an SDTMIG 3.4 report), so the breakdown is uneven by construction. The report does not name a rule-corpus
version, and the engine's rule cache can be updated without changing the CORE version, so rule_corpus_version
is reported as not stated. Findings-per-1000-records accompanies the rule counts because a single
rule failing on every record and a single rule failing once are not the same result.
rule_set_fingerprint. The report names the CORE version and no build. The fingerprint is computed from the
report's rules:
- For each rule, one line
<ID>@<version>, with the ID upper-cased, or<ID>@when the report gives the rule no version. - The lines sorted in character-code order and joined with a newline (
\n), with no trailing newline. - That text encoded as UTF-8 and hashed with SHA-256. The fingerprint is the first 16 hexadecimal characters of the hash.
It changes when the set of rules or any rule's version changes. It does not change when a rule is edited without a new version, or when only the engine's own code changes. A different fingerprint means the rule sets differ. The same fingerprint does not show that two runs used the same engine code. Where the report gives no rule versions, the meta text says so.
comparison states that axis 1 counts are compared only between runs with the same CORE version and the same
fingerprint. Different CORE releases, and different builds of one release, can run different rules or run them
differently. Their counts on the same data can differ.
3. Axis 2: Referential integrity
Inputs: datasets.
Four relationship classes, reported separately.
| Class | Check |
|---|---|
subject_presence |
Each subject identifier appearing in any domain resolves to a record in the subject-level domain |
visit_reference |
Each visit reference resolves to a visit the schedule (TV) defines, or to an unplanned visit the subject's SV records hold |
parent_child |
Each supplemental qualifier record's parent record exists |
declared_relationship |
Each explicitly declared relationship resolves at both ends |
Computation. Per class: resolved / answerable, where answerable excludes references into
domains that were not supplied (R4). Those are counted not_evaluable with the missing domain
named.
Unplanned visits. TV lists only the planned visits. A visit reference that is not in TV resolves when SV has a
record for the same subject and the same visit (VISIT, ignoring case) whose SVPRESP is not Y. The SV record
itself is counted as resolved. These references are counted as resolved and are also listed in the group's
breakdown as unplanned_visit:<DOMAIN>. A visit reference that is not in TV is unresolved when no SV record holds it
for the subject, when the SV record marks it planned (SVPRESP = Y), or when SV was not supplied. SVPRESP is an
SDTMIG 3.4 variable, so in SDTMIG 3.2 or 3.3 data, where SV has no SVPRESP, every visit SV holds that TV lacks
resolves.
Reported.
per_class[]: { class, answerable, resolved, unresolved, not_evaluable, missing_targets, missing_targets_total }
records_per_domain[]: { domain, records }
population: subjects_in_subject_domain, visits_defined, unplanned_visit_records
Notes. missing_targets names the unresolved values, at most ten, and missing_targets_total gives how many
there were. A reader should be able to tell "one subject is missing everywhere" from "every subject is
missing one visit."
4. Axis 3: Temporal coherence
Inputs: datasets; schedule if supplied.
Four check families, reported separately.
| Family | Check | Answerable when |
|---|---|---|
ordering |
A record's end (--ENDTC) is on or after its start (--STDTC) |
Both dates present and parseable |
study_bounds |
Activity falls within the subject's own participation period: DM.RFICDTC (informed consent) to DM.RFPENDTC (end of participation) | The bound on that side was recorded for the subject |
visit_dates |
A finding's collection date (--DTC) falls on or between its visit's SVSTDTC and SVENDTC (calendar days, inclusive; SVSTDTC alone when there is no end), matched by VISITNUM, else VISIT |
SV is supplied, the visit has an SV record with a start date, and the finding has a date. A finding at a visit with no dated SV record is not evaluable (missing) |
window_adherence |
Visit dates fall within declared windows | A schedule with windows was supplied (SDTM has no numeric window variables, so windows come from a facts file) |
Computation. Per family: violations / answerable. Partial dates are answerable only to the
precision both endpoints share; a comparison that the shared precision cannot decide is
not_evaluable, not a pass.
study_bounds, as decided 2026-10-20 (§10 item 5). The participation period is DM.RFICDTC to
DM.RFPENDTC, the SDTMIG variables for the date of informed consent and the date of end of
participation. The reference period RFSTDTC–RFENDTC is not used as a bound. A missing bound is not
replaced by another date: the side it would have checked is not evaluated for that subject, and the
number of subjects affected is reported. Dates in MHSTDTC, MHENDTC, CMSTDTC, CMENDTC, PRSTDTC, PRENDTC and
GFDTC that fall before consent are counted and reported, not scored, because these variables can record events
or specimens from before the study. A history date after the end of participation is still a violation.
The bounds are inclusive calendar days. A record on the consent day or on the
end-of-participation day is inside the period whatever its time of day: a first-day record can carry
a time before the consent time, and that is not a violation. Where either side is coarser than a day
(a month-only date against a full consent date), the shared-precision rule above still applies and an
undecidable comparison is not_evaluable. Time of day is used at the epoch boundary instead. Dates
inside the period and before first dose (RFSTDTC) are reported as the pre-treatment share and are
never a defect; a record on the first-dose day counts as pre-treatment when both it and RFSTDTC carry
a time and its time is earlier, and is reported as first_dose_day_untimed when either lacks one.
Reported.
per_family[]: { family, answerable, violations, rate, not_evaluable, breakdown? }
population: dates_full, dates_partial, dates_missing
meta: study_bounds_period (the bounds and the history variables used, as text)
breakdown for study_bounds counts each date by kind and place: before_consent:<DOMAIN>.<VAR> and
after_participation_end:<DOMAIN>.<VAR> (the violations), history_before_consent:<DOMAIN>.<VAR>,
before_first_dose:<DOMAIN> and first_dose_day_untimed:<DOMAIN> (reported, not scored).
Notes. window_adherence reports its rate without a target. Real studies have late and out-of-window
visits, so the methodology does not reward 100% adherence (§0b).
5. Axis 4: Clinical plausibility
Inputs: datasets; reference bundle; define if supplied.
Six checks. (physiological_range was superseded by the decision of 2026-08-26. This table matches what the
scorer computes.)
| Check | Mode | Computation |
|---|---|---|
reasonable_range |
A | Per test code: proportion of results inside an explicitly cited plausibility bound (§5.2). Explicitly not the clinically-normal range: abnormal results are expected in a trial and are not defects |
physically_impossible |
A | Proportion of results that cannot be a measurement of the quantity at all (§5.3). Definitional, not clinical |
reference_interval_excursion |
B | How often results fall outside the published reference range. No target: a dataset in which nothing is ever abnormal is as implausible as one in which everything is. A range published for fasting samples is used only for results whose record says they were fasting (§5.4) |
unit_validity |
A | Proportion of results whose unit is admissible for that test. Conventional and SI units are both admissible where the bundle states the conversion between them |
term_validity |
A | Proportion of coded values present in the codelist the supplied define assigns. Where the define gives value-level metadata for the variable, a record is held to the codelist of the entry whose condition (EQ or IN) it meets, and to none if that entry assigns none; otherwise to the variable's codelist. Exact match, since submission values are case-sensitive. A value coded to an external dictionary (MedDRA, WHODrug) is declined, not passed |
precision_consistency |
A | Proportion of results whose precision is consistent within their series: a test code in one dataset, and in PP one analyte (PPCAT). A result is consistent when it has the series' most common number of decimal places or its most common number of significant figures (§5.4) |
5.1 Interval entries and threshold entries
Added 2026-09-14. A reference entry declares what its source actually published:
kind |
Meaning | reasonable_range |
reference_interval_excursion |
|---|---|---|---|
interval (default) |
An interval estimated from a healthy reference population. The central-95% convention applies to its width | declines unless a cited bound is present | scored |
threshold |
A decision threshold or diagnostic category boundary, such as a guideline treatment target, a category cut-point, a regulatory limit | declines unless a cited bound is present | scored |
Superseded 2026-09-15 (see open item 7). kind no longer decides mode-A eligibility; a cited
bound does, for both kinds. kind remains required reading because it records what the source
published, which is what makes the excursion reading interpretable and the classification
contestable.
The distinction exists because widening a threshold as though its width implied a standard deviation produces bounds that are wrong in both directions. Measured on the transcribed vitals and ECG entries: an ACC/AHA treatment target of 90–130 mmHg derives to 69.6–150.4, so a hypertensive subject at 160 scores as a data defect; a WHO saturation threshold of 95–100% derives to a ceiling of 102.6%, so a physically impossible 101% passes. Fever, obesity and QT prolongation fail the same way.
A declined row is not_evaluable with the reason and the test codes named. It is never omitted and
never counted as conforming. Declined entries remain fully scored in the mode-B excursion reading,
which is what such sources are published for.
5.2 Only a cited bound scores in mode A
Decided 2026-09-15, closing open item 7. reasonable_range scores a result only where
its entry carries an explicit plausibleLow/plausibleHigh and a plausibleSourceCitation.
A bound is never derived from a reference interval for scoring purposes.
The reason is that a healthy-population interval does not describe a patient population. Widened by the default multiple, it still marks ordinary trial findings as data defects. An oral-temperature interval of 36.1–37.2 °C widens to 35.5–37.8, so a fever of 38.5 is scored as a defect; a fasting-glucose interval widens in the same way. No multiple can be defended for every test, so none is used.
An explicit bound is therefore a sourced clinical value in its own right, and rule 1 applies to it on its own account: a bound without its own citation does not load. The entry survives for the excursion reading and the uncited bound is reported as dropped.
The derivation of a bound from a reference interval remains in the code, unused by scoring.
This result is intended. Mode-A range coverage starts near zero and grows only as citable bounds are added. Rule 3 says coverage is reported, not maximized, and plausibility is scored only where a bound can be cited.
kind is decided by what the source publishes, not by clinical judgment about values, and each
entry carries a kindBasis note so the call can be contested. Classify conservatively: calling
an interval a threshold costs coverage, which is reported; calling a threshold an interval invents
defects in real data, which the reader cannot detect.
5.3 Definitional limits: physically_impossible
Added 2026-09-15. A separate mode-A group that asks whether a number can be a measurement of its quantity at all, rather than whether it is clinically usual. A negative mass. A saturation above full. Violations are data errors, which can come from a unit confusion, a transposed digit, a sentinel written into a numeric field, so the target is zero and it needs neither a citation nor a clinical opinion.
Limits come from two places:
| Source | Supplies | Requires |
|---|---|---|
The unit (physical.ts, published with the code) |
Non-negativity only, and only for units denoting a non-negative magnitude: length, mass, area, volume, duration, rate, absolute pressure, concentration, catalytic activity, cell count | nothing; it follows from the unit |
The entry (physicalLow / physicalHigh) |
Ceilings and anything test-specific | a physicalBasis stating why the limit holds. Without one the bound does not load |
Two units are deliberately absent. Celsius and Fahrenheit are interval scales on which negative
values are meaningful, so no bound follows from the unit. And % carries no definitional
ceiling: a percent-predicted spirometry value can exceed 100 and a percent change can be
negative, so a blanket rule on the unit would invent defects in both. An oxygen saturation
is capped at 100, but that is a property of the quantity, declared on its entry, not of the
percent sign.
A quantity with no known limit is not_evaluable with that reason. Nothing is guessed.
This is the group that catches what the old derived range let through: a saturation of 101% passes every clinical-range test and is impossible.
5.4 Age, fasting status, units and precision
Age. An entry's ageLow and ageHigh are in years. A subject's age is DM.AGE read with DM.AGEU: YEARS as
given, MONTHS divided by 12, WEEKS times 7 / 365.25, DAYS divided by 365.25, HOURS divided by (24 x 365.25). An age
with no AGEU, or with a unit outside that list, is not known. An entry limited by age applies only to a subject whose
age is known and inside its limits; a subject whose age is not known is never assumed to be an adult. In bundle 0.3
every entry except oxygen saturation (OXYSAT) carries ageLow 18, and the laboratory entries carry an ageBasis
saying why. A result with no applicable entry because the age is not known is not_evaluable, kind missing. A
result whose subject is outside every entry's ages or sex is not_evaluable, kind declined.
Fasting. An entry with fastingOnly describes fasting samples only. In bundle 0.3 that is glucose, whose cited
range is the fasting plasma glucose range. A result is compared with such an entry only when its --FAST (LBFAST) is
Y. With --FAST blank the comparison is not_evaluable, kind missing; with any other value it is
not_evaluable, kind declined. The scorer does not check the specimen type (LBSPEC), so the plasma glucose range is
applied to any GLUC record marked fasting.
Reading the excursion share. meta.reference_interval_reading states how the share is read. The share
describes the data and is not a pass or fail target. The ranges are published adult reference ranges and limits.
Blood pressure (SYSBP, DIABP) is compared with a normal adult resting range of 90/60 to 120/80 mmHg (MedlinePlus).
Glucose is compared with its fasting range only for fasting samples.
Units. Bundle 0.3 adds SI units, with their conversions, for creatinine, uric acid, urea nitrogen, cholesterol,
triglycerides, sodium, potassium, chloride, bicarbonate, calcium, phosphorus, magnesium, iron, total protein, MCHC,
ferritin, platelets, neutrophils, lymphocytes, eosinophils, and the enzymes ALP, GGT, LDH and CK. Each conversion
states its factor and why the factor holds: an identity (mEq/L and mmol/L for a monovalent ion), a decimal prefix, a
definitional SI relation (1 ukat/L = 60 U/L), or a molar mass from standard atomic weights, which the conversion
names. Triglyceride is a mixture, so its factor uses the molar mass of triolein. A result is converted into the
entry's unit before any comparison; a unit with no stated conversion is not_evaluable, kind declined, never
compared across units.
Precision. For each series, the scorer finds the most common number of decimal places and the most common number
of significant figures, the latter taken from results whose count is unambiguous. A whole number ending in zeros
(120) has an ambiguous count, two or three, and matches when the most common count is either. A zero has no
significant figures and is judged by decimal places alone. A result fails only when it matches neither.
Computation. Test codes absent from the reference bundle are not_evaluable and named. Coverage
(what proportion of supplied results the bundle could judge) is reported alongside, because an
apparently perfect score over 5% of the data is a different claim from one over 95%.
Reported.
per_check[]: { check, answerable, violations, rate, not_evaluable }
population: tests_supplied, tests_in_bundle, results_judged, results_total (reference coverage)
per_test[]: { test, unit, n, in_reference_bundle, judged_against_reference, outside_reference,
outside_reference_rate, outside_plausibility_limit, physically_impossible,
unit_not_accepted, precision_mismatches }
meta.reference_bundle_version <- the bundle's version string
meta.reference_interval_reading <- how the excursion share is read (§5.4)
population.reference_bundle_entries <- how many entries it carried
per_test has one row per dataset and test code (LB.ALT). unit is the standard unit most of its results carry.
In population, results_judged is the number of results compared with a reference range.
population is numeric by contract, so the bundle version travels in the axis's meta map
instead. Both are reported: the version identifies which bundle produced the numbers, and the entry
count says how many entries it had. The command-line version accepts another bundle, and this is what
stops two scores computed against different bundles from being mistaken for comparable.
Notes. per_test is reported because an aggregate hides a single badly parameterized analyte.
6. Axis 5: Longitudinal realism
Inputs: datasets with repeated measures per subject.
The primary statistic is the intraclass correlation. For each repeatedly measured quantity, decompose variance into within-subject and between-subject components and report the ICC.
This is the axis's discriminator: a generator that redraws each visit independently from one shared distribution produces an ICC near zero, however plausible each value is. A share of total variance is expected to be between-subject for many clinical measurements, but the expected share varies by analyte and for some it is low. Hence the reporting rule below: per measure, with no target attached.
Three further measures.
| Measure | Computation |
|---|---|
trajectory_shape |
Number of subject series that are monotone, mean-reverting, or unstructured (pooled across measures in the group, per measure in the table) |
missingness_pattern |
Subjects whose visits stop and do not resume, have intermittent gaps, or are complete |
dropout_curve |
Subjects with a record at each scheduled visit, reported as a series |
Reported.
per_measure[]: { measure, subjects, subjects_with_repeats, observations_per_subject_median, icc,
icc_ci95_low, icc_ci95_high, icc_log, icc_log_ci95_low, icc_log_ci95_high,
icc_log_not_run }
trajectory_shape[]: { measure, monotone, reverting, unstructured, too_short }
population: measures_with_icc, measures_with_log_icc, missingness_terminal, missingness_intermittent,
missingness_none
dropout_curve[]: { visit, retained, n }
meta: icc_estimator, icc_log, icc_not_run_by_design, measure_key
Measures. A measure is a dataset and test code (LB.ALT). In PP it is also the category (PP.CMAX/<PPCAT>),
because PP can hold parameters for several analytes under one test code, and pooling an analyte reported in ug/mL
with one reported in ng/mL would put two scales in one series. PK concentrations (PC) get no ICC, because they are
sampled across each dosing interval and change with the time since the dose. The group is not_evaluable, kind
declined, with that reason.
The estimator. ICC(1), the one-way random-effects form, computed from the subjects with at least two results for the measure. Where subjects have unequal numbers of results, k, the number of results per subject, is replaced by k0, the mean number corrected for the inequality: k0 = (N - sum of n_i squared / N) / (s - 1), for s subjects and N results. The 95% interval comes from the F distribution of the ratio of the between-subject to the within-subject mean square (MSB/MSW), as Shrout and Fleiss (1979) give it for ICC(1,1), with k0 in place of k; with unequal numbers of results the interval is approximate. A measure with fewer than two such subjects, or whose results do not vary, has no ICC and no interval, and both are reported as null. A measure whose results vary between subjects but never within one has an ICC of 1 and no interval.
The log-scale ICC. icc_log is the same estimator, with its interval, computed on the natural logarithm of each
value. It is reported beside icc and changes nothing about it. It is computed only when the ICC is computed and
every value it uses is above zero. When it is not computed, icc_log and its interval are null and icc_log_not_run
says why: the measure has no ICC, a value is zero or below, the logarithms do not vary, or the measure is a PK
concentration. On the log scale a very high value is closer to the rest, so a few very high values beside ordinary
results tend to lower icc_log less than they lower icc. A planned change that every subject goes through, such as a treatment nadir, still lowers it.
Notes. ICC is reported per measure and never averaged across measures, because heart rate and hemoglobin are different quantities. No target value is published, because the expected ICC depends on the analyte; a reader who knows the measure judges it.
7. Axis 6: Cross-domain clinical consistency
Inputs: datasets across two or more domains.
Each relationship rule is published individually, in this form:
id, statement, antecedent_domain, consequent_domain, answerable_when, agreement_when
Draft 0.3 rule set (0.3.0), deliberately small: it holds only relationships that can be stated precisely enough for two readers to agree.
| id | Statement |
|---|---|
CD-01 |
A treatment recorded for an indication (CMINDC) has that indication present in the subject's own history or event records (MHTERM, MHDECOD, AETERM, AEDECOD). Rule set 0.3.0: the comparison ignores only case, extra spaces and punctuation (below) |
CD-02 |
A discontinuation attributed to an adverse event (DSDECOD or DSTERM) has an adverse event recorded for the subject, starting on or before it (calendar days) |
CD-03 |
Study treatment withdrawn because of an adverse event (AEACN = DRUG WITHDRAWN, treatment permanently stopped for that event) is not given after that event ends: the subject's last exposure (latest EXENDTC, else EXSTDTC) is on or before AEENDTC. An event with no end date is not evaluable (missing). Rule set 0.2.0 |
CD-04 |
An event term is compatible with the subject's recorded characteristics (a condition of one sex not recorded for another) |
Computation. Per rule: violations / answerable, where answerable requires both sides present.
CD-01 matching. Both sides are upper-cased. An apostrophe (' or ’) is removed, so CROHN'S and CROHNS match.
Every other Unicode punctuation character becomes a space, so GASTRO-OESOPHAGEAL and GASTRO OESOPHAGEAL match.
Symbols such as + < = ^ are kept. Runs of spaces become one space, and leading and trailing spaces are dropped. The two
must then be equal. CD-01 checks that the condition a treatment was given for is recorded, and it is not run for an
indication containing the word PROPHYLAXIS, PROPHYLACTIC, PREVENTION or PREVENTIVE. Such a record is not_evaluable,
kind declined.
CD-01 near matches. An unmatched indication is compared with each of the subject's terms. A term is a near match of one of three kinds:
spacing: the two are equal once all spaces are removed.spelling: with spaces removed, both have at least six characters, and they differ by at most two insertions, deletions or substitutions.wording: every word of one is a word of the other.
Terms are tried in alphabetical order of their match form, and for each term the kinds are tried in the order listed. The first hit is the near match. The scorer does not judge whether the two terms mean the same thing. The record is still a violation, and the near match is listed so a reader can see the likely cause.
Reported.
per_rule[]: { id, answerable, agreements, violations, rate, not_evaluable }
near_matches[]: { indication, nearest_term, how, records }
population: cd01_near_matches
meta: rule_set_version
The CD-01 group's breakdown counts the near matches by kind, as near_match:<how>. A near-match row holds the two
terms and a count, and no subject identifier.
Notes. The rule set is expected to change through comment, and every change is versioned. A rule that commenters dispute stays published, with the dispute noted, so that readers can see it.
8. Axis 7: Deliverable completeness
Inputs: everything supplied, plus the declared scope.
An inventory, not a percentage. Each element is present, absent, or partial, and the
elements are not interchangeable so they are never averaged.
| Element | Present when |
|---|---|
domains_by_class |
Domains supplied, grouped by observation class, against the declared scope |
dataset_metadata |
A define supplied |
define_consistency |
The supplied define and the data agree (below). partial when there is any finding; not evaluable without a define |
value_level_metadata |
Value-level metadata present in the define |
trial_design |
Trial-design information supplied |
Reported.
inventory[]: { element, state, detail }
declared_scope, scope_met
consistency_findings[]: { check, dataset, variable, count, detail }
define_consistency compares the datasets and variables the define declares (each ItemGroupDef and the
ItemDefs it references) with the datasets supplied. Each finding names its check, the dataset, the variable where
there is one, and how many values it covers:
| Check | Finding |
|---|---|
dataset_not_in_define |
A supplied dataset the define does not declare |
variable_not_in_define |
A column in a supplied dataset that the define does not list for it |
variable_not_in_data |
A variable the define lists that the supplied dataset has no column for |
type_mismatch |
Values that are not whole numbers in a variable the define declares integer, or not numbers in one it declares float |
length_exceeded |
Text values longer than the variable's declared Length |
A declared dataset that was not supplied is the declared_scope element's finding, not this one's. A supplied dataset
with no records has no columns to compare. Date and time variables are not checked against their DataType here,
because the Define-XML date types differ by precision and the partial-date comparison belongs to axis 3.
9. The score object
The scorer's JSON output (the command line's --json, and the results file the browser page saves). It is built
where the scoring runs and is not sent anywhere; no service receives score objects. The score object contains no data
values and no record-level content.
method_version, scored_at_utc, scorer_version
tool: { name, version, declared_by }
study: { label, provenance, therapeutic_area, phase, subjects, duration_visits }
inputs: { formats[], domains[], subjects, records_total, define_supplied, schedule_supplied }
axes: { axis_1..axis_7 as specified above, each carrying its mode: A | B | C, and warnings where there are any }
population: { populated_cells_by_domain[] }
content_hash
An axis may carry warnings, a list of plain sentences a reader should see before the axis's numbers, for example,
a sentence giving the number of rules CORE could not run. A warning restates a count the axis already reports and changes no count. The
field is absent when an axis has no warning.
tool is what the person scoring states about the tool that made the data; declared_by is person scoring when
they state it and null when they do not. The scorer cannot check it.
study.provenance is one of authoring_system_import, standards_native_definition,
synthesized_without_source, or other: the path the study design arrived by. It is null when the person scoring
does not state it, and such a score cannot take part in a comparison across provenances, because §0c depends on
it: a spread across provenances is what distinguishes a property of the tool from a property of one study. label
is a name the person scoring the study chooses and identifies no person. therapeutic_area and phase are the TS
values for THERAREA and TPHASE when TS supplies them, subjects counts the subjects in DM, and duration_visits
counts the planned visits in the schedule. populated_cells_by_domain gives, for each dataset, the cells holding a
value and the cells in all.
Each axis carries its mode, so a consumer of the score object can render Mode A results as defects
and Mode B results as comparisons without re-deriving which is which.
content_hash is a hash of the score object, not of the data, and it identifies nobody. It is the SHA-256 of the
object's canonical JSON (every object's keys sorted, no whitespace) with scored_at_utc and content_hash left
out, so the same inputs and versions give the same hash whenever they are scored (R6). Recomputing it shows whether
a saved score was edited after it was computed. It cannot show that the inputs were what the label says.
10. Open items in this specification
The body refers to these by number.
Open:
- Item 1. Bounds for axis 4. The specification is complete, but no entry in reference bundle 0.3 yet cites a
plausibility limit, so
reasonable_rangedoes not run on any test. Results are judged by their physical limits and by the reference-range reading until cited limits are added. - Item 4. Partial dates. The exact comparison for each pair of date precisions should be written out. Settled so
far:
study_boundscompares by calendar day, inclusive; a start and an end on the same calendar day are in order unless their times say otherwise; a start month with an end date inside it stays undecidable. - Item 6. No rule compares age with birth date. AGE, BRTHDTC, and RFICDTC are never compared on any axis. This is a candidate for axis 3 or for a new cross-domain rule.
Closed:
- Item 8 (2026-11-02). Scorer 0.3.0 emits every field in the "Reported" lists, inside the score object (§9).
- Item 3 (2026-11-02). The ICC estimator is ICC(1) with k0, and its 95% interval comes from the F distribution (§6).
- Item 2 (2026-10-26).
interval_plausibilityis replaced byvisit_dates(§4). - Item 5 (2026-10-20).
study_boundsruns from informed consent (RFICDTC) to the end of participation (RFPENDTC); medical history and medication dates before consent are reported, not scored (§4). - Item 7 (2026-09-15). A healthy-population interval is never widened into a plausibility limit: mode A scores a range only where a bound is explicitly cited (§5.2).