Skip to content

0021 — Pinned source import and lineage-first splitting

Date: 2026-10-20. Scope: W14 / issue #15; local engineering under the approved study authority. No historical experiment, fixture label or result is reinterpreted.

Decision

Implement hallmark.data_admission without downloads or optional dependencies. The caller authenticates source approval, licence rights and review evidence, then enrolls the exact DataImportPlan.identity() outside the supplied bundle. An attacker supplying both a plan and its purported trusted pin controls this starting point. Names, hashes and review_kind="human" cannot authenticate a person, establish rights, or validate annotations.

Each approved source revision is the SHA-256 of its encoded SourceManifest, not an upstream Git revision. That document retains the upstream revision, licence identifier, policy, purpose, privacy classification, exact adapter-output files, label mapping and review, annotation/adjudication evidence, transformation recipe and data card. SourceApproval separately pins licence text, restrictions and approval evidence. All referenced evidence bytes must be present; additional, missing or altered documents/files reject.

The adapter output is strict SourceRow JSONL, preserving prompt/response strings, globally unique record IDs, source labels, family, parents, shared history and a difficult-negative flag. An approved adapter must explain any upstream conversion in the pinned recipe and retain its original-file provenance. This importer does not implement Hub/Parquet adapters or establish that those declarations match an upstream dataset. Unknown labels, missing parents, parent cycles and positive difficult-negative labels reject.

Each normalized Record/1 has a lineage sidecar naming its source manifest/file, one-based row number, exact original line hash including its terminator, parents, history and transformation/label/annotation review pointers. Policy and rubric are bound through the source manifest and plan. No text is trimmed, normalized or discarded.

Group and split recipe

Before splitting, take connected components of parent edges, shared history IDs, identical prompts (even with different responses), and reviewed near candidates. The candidate finder has one fixed, bounded rule: ASCII case folding and ASCII whitespace folding on prompt bytes. Non-ASCII bytes remain unchanged. Candidate clusters contain more than one distinct original prompt; exact copies are already grouped. The reviewed digest binds the complete sorted candidate inventory, encoded as compact sorted-key UTF-8 JSON with a final newline. Every candidate is retained and grouped, including conflicting labels.

This is a conservative lexical check, not semantic duplicate detection. Reviewed paraphrase/translation/semantic links must appear as parents or shared history before import. Missing or untruthful lineage cannot be discovered by a hash check. Broader contamination and label audits remain required before real evaluation.

The frozen study design supplies the split and representative recipe. A group ID is g- plus SHA-256 of its sorted member-ID JSON array with a final newline. Within source/family strata, groups sort by SHA-256 of inj-direct-design/1|split| plus group ID, then group ID. Groups spanning sources or families use one mixed stratum. Integer boundaries are floor(3*n/5) and floor(4*n/5) for train/calibration/test. All three primary partitions must exist; small inventories reject rather than rebalance. Representatives minimize SHA-256 of inj-direct-design/1|representative| plus record ID, with ID as tie break. Labels and model scores do not enter either assignment.

A declared held-out source/family moves its entire connected component to the separate held-out test cohort. Cross-source members travel together; this does not imply every member of that component is from an unseen source. The split manifest exposes all sources, families and members so evaluation can report that limitation. Held-out cohorts must not enter primary estimates.

Artifact and storage boundary

build_dataset returns a typed corpus DatasetManifest and exact bytes for records.jsonl, lineage.jsonl and splits.json. Enroll the resulting dataset identity separately. verify_dataset checks that pin and every file, then rebuilds from the pinned source plan: rehashing a cross-split descendant or changed sidecar still fails. This is same-code reproduction, not independent verification.

write_dataset validates the output inventory and creates a fresh 0700 directory with exclusive 0600 files. It never overwrites an existing directory. Each file is synced; the manifest is linked into its final name only after all writes succeed. Partial failures remain for diagnosis and require a new directory. The caller must provide a trusted parent directory and use verify_dataset before consumption. This writer is not an OS immutability mechanism, a defense against concurrent parent-path replacement, or qualified power-loss recovery.

Limits: 64 sources, 64 files per source, 100,000 total rows, 128 MiB total raw adapter bytes, 256 MiB total output, 256 evidence files of at most 1 MiB each and 8 MiB combined. Prompt/response fields each allow at most 32,768 UTF-8 bytes; the study may reduce the prompt bound. Limit failures never truncate records. The existing baseline loader keeps its separate 8 MiB bound; larger imports require downstream training integration before use.

Fixture and remaining data-card gaps

The committed fixture contains 26 hand-authored marker rows, invented approval/ review documents and the repository's LicenseRef-Proprietary notice marker. Its purpose is local software testing, its privacy class is synthetic, and fixture_only=true prevents its review declarations entering a real import. It has no real human label audit, consent claim, task-quality measurement, representative sampling, semantic contamination study or external source rights. Public/human mode tests also use invented documents and establish binding only.

No real dataset, checkpoint or private review is committed. Actual sources still need reviewed rights, restrictions, privacy/consent, original labels and policy mapping, adjudication and difficult-negative provenance, transformation and generator records, source/temporal/family coverage, contamination limits and independent group counts. A completed import does not satisfy final measurement admission or authorize a quality claim.

Validation

tests/test_data_admission.py checks exact pins/inventories, malformed input, lineage and review mutations, independent graph reachability and split arithmetic, label/order independence, held-out closure, output tampering and write failures. Seven schema snapshots and the harmless fixture reproduce locally. Both package gates run scripts/check_data_import.py against the installed wheel.