Skip to content

v1.0 Release Checklist

Every item names the implemented command that produces its evidence. Release cannot proceed from documentation alone: a v1.0 tag requires all items checked with linked artifacts.

Gates

  • [ ] Local tests pass. Commands: uv run pytest, uv run ruff check ., uv run mypy sommelier tests. Evidence: green CI run on the release commit.

  • [ ] Formatting and parser golden fixtures pass. Commands: uv run pytest tests/formatting tests/evaluation. Evidence: same CI run; any golden regeneration reviewed as a prompt-contract change.

  • [ ] Remote smoke run passes. Command: uv run modal run remote_pipeline.py --config examples/config.smoke.yaml --mode smoke --max-rows 2500 (on a machine with its own GPU: uv run sommelier pipeline run --config examples/config.smoke.yaml --mode smoke --input <raw_rows.jsonl>). Evidence: artifacts/runs/smoke-<run_id>/report/comparison_report.json with all stage manifests succeeded.

  • [ ] Full reference run writes all required artifacts. English-only command: SOMMELIER_GPU=L40S SOMMELIER_TIMEOUT_SECONDS=86400 uv run modal run --detach remote_pipeline.py --config examples/config.full.yaml --mode full --max-rows 60000 (local GPU variant: uv run sommelier pipeline run --config examples/config.full.yaml --mode full --input <raw_rows.jsonl>). examples/config.full.yaml contains only the pinned English root source. Any paired release instead requires a separate config whose paired dataset is an immutable, provenance-complete publication; the historical French v2 commit does not satisfy that gate. Evidence: the run directory matches the required artifact layout, including comparison_report.json and comparison_report.md.

  • [ ] Comparison report states limitations and reproduction commands. Evidence: the Limitations and Reproduction sections of the full run's comparison_report.md; base and adapter share the same test split, prompt set, parser, and decoding digests (enforced by sommelier report compare).

  • [ ] Any Hebrew paired dataset is published from the audited bundle. Commands: run sommelier release publish-dataset once without --execute, review the JSON plan, then repeat with the exact repository confirmation, receipt path outside the bundle, and --execute (plus --create-repo for the absent dedicated repository). Evidence: a verified publication receipt, immutable Hub commit, and matching round-trip SHA-256 for all six allowlisted files; the raw provider journal is absent. Replace the provisional Hebrew revision in examples/config.v3-he-full.yaml with this exact commit before any full evidence run.

  • [ ] The configured Hebrew v3 L40S shape has a diagnostic resource-fit run. Command: uv run modal run --detach remote_qlora_preflight.py --config examples/config.v3-he-full.yaml --run-id <new-id>. Evidence: preflight_report.json records one optimizer step from four batch-4 microbatches, one batch-4 evaluation forward, near-4096-token English and Hebrew inputs paired exactly one-to-one by source, full QLoRA wiring, exactly one visible L40S with CUDA-0-only placement, and peak memory. This diagnostic is not dataset, accuracy, full-training, or release evidence.

  • [ ] Both Hebrew v3 full evaluation arms succeed from the same clean commit. Commands: run remote_pipeline.py --mode full once with the pinned v1 --adapter-id and --adapter-revision as he-v3-v1-baseline, then once without an external adapter as he-v3-full, both with examples/config.v3-he-full.yaml, SOMMELIER_GPU=L40S, and the admitted 86,400-second outer timeout. Evidence: immutable published Hebrew input, succeeded manifests, identical producer/runtime/evaluation contracts, base plus v1 adapter evaluations, and base plus v3 QLoRA evaluations.

  • [ ] The three-arm Hebrew experiment is finalized. Command: uv run sommelier report experiment --base artifacts/runs/he-v3-full/eval/base --v1-en artifacts/runs/he-v3-v1-baseline/eval/adapter --v3-en-he artifacts/runs/he-v3-full/eval/adapter --english-non-inferiority-margin 0.01 --seed 42 --resamples 2000 --out artifacts/experiments/he-v3. Evidence: experiment_report.json independently binds the three arms, measured tokenizer/training/inference evidence, Hebrew uplift decisions, and the preregistered English non-inferiority gate. Failed claims stay failed.

  • [ ] The final curated adapter bundle passes identity-bound preflight. Commands: from the producer repository root at the exact clean run commit, assemble every allowlisted adapter file except release_preflight.json; then run SOMMELIER_ACK_BASE_MODEL_LICENSE=<base_model_id> uv run sommelier release preflight --config <adapter-bundle>/config.resolved.yaml --artifact-root <adapter-bundle>. Evidence: a sommelier.release_preflight.v2 report with every gate pass whose identity records immutable model/tokenizer/dataset revisions, a clean source commit, tracked dependency-lock digest, and final bundle tree. The standalone command records those identity decisions; publish-adapter strictly requires them. The report itself is the sole excluded path. Do not mutate the bundle afterward.

  • [ ] The preflight secret scan of the pre-report adapter bundle is clean. Command: the artifact_secret_scan gate of that same preflight run against the curated adapter bundle before release_preflight.json is written. Evidence: gate status pass (not skip) for every pre-existing scannable bundle file. The generated report is deliberately excluded from its own tree identity. publish-adapter then rescans the complete unchanged bundle, including release_preflight.json, before any Hub access.

  • [ ] Any Hebrew v3 adapter is published from final claim-gated evidence. Commands: run sommelier release publish-adapter on the unchanged bundle in validation-only mode, review its plan, then explicitly execute it with a fresh receipt outside the bundle. Evidence: a verified receipt whose immutable Hub commit contains the exact staged LoRA tree, reviewed NVIDIA and Llama agreement copies, NOTICE, third-party ledger, manifests, config, preflight, and experiment report. The card has no unresolved marker and binds the clean producer plus immutable Hebrew dataset revisions.

  • [ ] Reference-run dependencies are pinned. Evidence: the dependency_lock_sha256 in the full run's stage manifests, plus the GPU-stack versions reported in the run summary. The full remote pipeline image already installs the exact TRAIN_PACKAGES pins and refuses a full run unless the observed runtime matches PIPELINE_RUNTIME_VERSIONS; close this item with that final run evidence.

  • [ ] Changelog is current. Evidence: the changelog Unreleased entries moved under the release version, with migration notes for every behavior change and no undocumented public schema or metric changes.

Blocker mapping

Release blocker Gate evidence
Missing license acknowledgement base_model_license_ack in release_preflight.json
Reference-run dependency drift dependency_lock gate + stage manifest dependency_lock_sha256 + pinned runtime-version admission evidence
Missing comparison report full-run comparison_report.json / .md
Mismatched base/adapter split digests sommelier report compare fails closed (INV-DATA-006)
Secrets in artifacts artifact_secret_scan gate
Test suite failure CI run on the release commit
Undocumented public schema or metric change changelog review against SUPPORTED_SCHEMAS and metric names