M3/E5/#15 — Ingest contract conformance

.github/workflows/constitution.yml

@@ -97,13 +97,17 @@ jobs:
 
       - run: pip install pytest
 
-      # `domain` added by M2/E4/#12, which places the pure domain model's unit tests beside
-      # the code they test. Without it those 244 tests would exist and never run — the exact
-      # defect recorded as EPD-004 when the kernel landed, repeating one issue later.
-      # Reported as EPD-006: the Engineering Plan assigns unit tests to #12 and no issue owns
-      # extending the runner that executes them.
+      # `domain` added by M2/E4/#12 and `contexts` by M3/E5/#15, each placing its unit tests
+      # beside the code they test. Without the path here those tests would exist and never
+      # run — the exact defect recorded as EPD-004 when the kernel landed. Reported as
+      # EPD-006 (domain) and EPD-007 (contexts): the Engineering Plan assigns unit tests to
+      # these issues and no issue owns extending the runner that executes them.
+      #
+      # `contexts` will keep gaining packages through E5–E11. It is named as a root rather
+      # than per-context so that a new context's tests run the day they are written, instead
+      # of on the day someone remembers this file.
       - name: Unit tests beside their code
-        run: pytest kernel domain -q -rs
+        run: pytest kernel domain contexts -q -rs
 
   # Integration tests cross a real boundary, and some of them assert on repository refs
   # rather than on files. Without this job they would never execute — the same gap that left

contexts/ingest/__init__.py

@@ -0,0 +1,89 @@
+"""Ingest — acquire from a Source, and canonicalise via an Instrument parser.
+
+M3/E5/#15 delivers the **contract** half of this context: what an acquisition channel must
+provide, what it may never let out, and where the one sanctioned clock read lives. It delivers
+no channel. The ISSDC-PRADAN adapter is #16; the SoLEXS and HEL1OS parsers are #17 and #18;
+the bitemporal write path is #19.
+
+Public interface (E5 §4):
+
+    SourceAdapter            the protocol a channel satisfies
+    verify_conformance       what an adapter must pass before its data enters the platform
+    SourceDescriptor         source_id · authority · latency_class · granularity · retrieval
+    RawArtifact              a Tier 0 artifact, identified rather than held
+    AcquisitionProvenance    who fetched what, from where, and when the system learned it
+    Acquisition              the two together, which is what `acquire` returns
+    boundary.stamp           the single sanctioned clock read (ADR-0004, TIS §0.4)
+    Credential               a secret that cannot be printed or serialised (STD-19)
+    assert_credential_free   refuse to let one cross the boundary
+
+THE THREE RULES THIS CONTEXT EXISTS TO KEEP
+--------------------------------------------
+**Credentials never leave.** E5 §13 makes this the highest-sensitivity context — it holds the
+only secrets in the system. `Credential` redacts itself in `repr`, `str` and `format`, so the
+common leak routes are closed by the type; `assert_credential_free` closes the rest by walking
+anything about to cross outward.
+
+**Tier 0 bytes are never redistributed.** `RawArtifact` has a digest, a size, a way to
+re-acquire, and no field that could hold the bytes. ADR-0023 and E5 §12 both land here.
+
+**Both times, and the second one never fabricated.** `ingest_time` is stamped at this
+boundary and nowhere else (ADR-0004). Nothing here supplies one for a historical row —
+ADR-0022 gives `null` exactly one meaning, and E5 §11(ii) forbids any code path from writing
+a non-null value for data that predates bitemporal capture.
+
+IMPORTS. `contracts`, `domain`, `kernel` and the standard library — exactly what
+`contexts.ingest`'s policy grants (ADR-0026, enforced by M2/E4/#13). No other context.
+"""
+
+from contexts.ingest.acquisition import (
+    Acquisition,
+    AcquisitionProvenance,
+    RawArtifact,
+)
+from contexts.ingest.contract import (
+    CONFORMANCE_GATE,
+    SourceAdapter,
+    verify_acquisition,
+    verify_conformance,
+    verify_descriptor,
+)
+from contexts.ingest.credentials import (
+    CREDENTIAL_BOUNDARY,
+    Credential,
+    assert_credential_free,
+)
+from contexts.ingest.descriptor import (
+    LatencyClass,
+    RetrievalDescriptor,
+    SourceDescriptor,
+)
+from contexts.ingest.errors import (
+    ContractViolation,
+    IngestError,
+    IntegrityFailure,
+    PolicyRejection,
+    UnavailableResource,
+)
+
+__all__ = [
+    "Acquisition",
+    "AcquisitionProvenance",
+    "CONFORMANCE_GATE",
+    "CREDENTIAL_BOUNDARY",
+    "ContractViolation",
+    "Credential",
+    "IngestError",
+    "IntegrityFailure",
+    "LatencyClass",
+    "PolicyRejection",
+    "RawArtifact",
+    "RetrievalDescriptor",
+    "SourceAdapter",
+    "SourceDescriptor",
+    "UnavailableResource",
+    "assert_credential_free",
+    "verify_acquisition",
+    "verify_conformance",
+    "verify_descriptor",
+]

contexts/ingest/acquisition.py

@@ -0,0 +1,190 @@
+"""What an acquisition produces: a referenced artifact and the provenance of getting it.
+
+E5 §4: `acquire(selector) -> RawArtifact + AcquisitionProvenance`.
+
+THE ARTIFACT HOLDS NO BYTES, AND THAT IS THE POINT
+---------------------------------------------------
+`RawArtifact` has a digest, a size, a retrieval descriptor and an optional cache path. It has
+no `bytes` field and no `content` field, and `test_a_raw_artifact_cannot_carry_bytes` asserts
+that it never gains one. Two separate rules land on the same design:
+
+  ADR-0023 / STD-23 / E5 §11(iv)  Tier 0 bytes are referenced, never redistributed. Local
+                                  copies are evictable caches, never the system of record.
+  E5 §12                          Streaming parse; a whole-day archive must not be fully
+                                  materialised.
+
+A type that could hold the bytes would make both violations one attribute access away, and
+neither would be visible in review — the object would simply be larger.
+
+`cache_path` is the evictable copy. It is optional because a cache that has been evicted is
+the normal case, not an error: what makes the artifact usable again is `retrieval`, which is
+why that field is required and the path is not.
+
+THE CLOCK READ
+--------------
+`AcquisitionProvenance.ingest_time` is stamped by `contexts.ingest.boundary.stamp`, the single
+sanctioned clock read in the system (ADR-0004, TIS §0.4). It is required and non-null here,
+because this type describes an acquisition that *just happened*. Historical rows that predate
+bitemporal capture carry `ingest_time = None` on the Observation (ADR-0022) and have no
+AcquisitionProvenance at all — there is no acquisition to describe, and E5 §11(ii) forbids any
+code path from writing a non-null ingest_time for them.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from contexts.ingest.descriptor import RetrievalDescriptor
+from domain.errors import ContractViolation
+from domain.values import Digest, Identifier, RunId, Timestamp
+
+
+@dataclass(frozen=True)
+class RawArtifact:
+    """A Tier 0 artifact, identified rather than held (ADR-0023)."""
+
+    digest: Digest
+    size_bytes: int
+    retrieval: RetrievalDescriptor
+    #: Where an evictable local copy currently sits, if one does. Never the system of record.
+    cache_path: Path | None = None
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.digest, Digest):
+            raise ContractViolation(
+                "/digest",
+                f"digest must be a Digest, got {type(self.digest).__name__}. Only the "
+                f"provenance kernel mints one (ADR-0005); this records the one it minted.",
+            )
+        if isinstance(self.size_bytes, bool) or not isinstance(self.size_bytes, int):
+            raise ContractViolation(
+                "/size_bytes",
+                f"size_bytes must be an integer, got {type(self.size_bytes).__name__}",
+            )
+        if self.size_bytes < 0:
+            raise ContractViolation(
+                "/size_bytes", f"size_bytes must be non-negative, got {self.size_bytes}"
+            )
+        if not isinstance(self.retrieval, RetrievalDescriptor):
+            raise ContractViolation(
+                "/retrieval",
+                f"retrieval must be a RetrievalDescriptor, got "
+                f"{type(self.retrieval).__name__}. Without one these bytes cannot be "
+                f"re-acquired, and ADR-0023 requires that any stage can.",
+            )
+        if self.cache_path is not None and not isinstance(self.cache_path, Path):
+            raise ContractViolation(
+                "/cache_path",
+                f"cache_path must be a Path or None, got {type(self.cache_path).__name__}",
+            )
+
+    @property
+    def is_cached(self) -> bool:
+        """Whether a local copy is currently recorded. An evicted cache is not an error."""
+        return self.cache_path is not None
+
+
+@dataclass(frozen=True)
+class AcquisitionProvenance:
+    """The record of one acquisition, for the provenance kernel to store (E5 §7).
+
+    Carries the run that performed it, the source it came from, the digest it produced and
+    the instant the system learned of it. Nothing else — and specifically no credential, no
+    URL that might embed a token, and no response body (E5 §13, §14).
+    """
+
+    run_id: RunId
+    source_id: Identifier
+    artifact_digest: Digest
+    #: The sanctioned clock read (ADR-0004). Required and non-null: this describes an
+    #: acquisition that happened, so the moment is known by construction.
+    ingest_time: Timestamp
+    #: Instruments whose products this artifact contains, if the source states them. Empty
+    #: where the channel is instrument-agnostic; a daily archive may carry several.
+    instruments: tuple[Identifier, ...] = field(default=())
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.run_id, RunId):
+            raise ContractViolation(
+                "/run_id", f"run_id must be a RunId, got {type(self.run_id).__name__}"
+            )
+        if not isinstance(self.source_id, Identifier):
+            raise ContractViolation(
+                "/source_id",
+                f"source_id must be an Identifier, got {type(self.source_id).__name__}",
+            )
+        if not isinstance(self.artifact_digest, Digest):
+            raise ContractViolation(
+                "/artifact_digest",
+                f"artifact_digest must be a Digest, got "
+                f"{type(self.artifact_digest).__name__}",
+            )
+        if not isinstance(self.ingest_time, Timestamp):
+            raise ContractViolation(
+                "/ingest_time",
+                f"ingest_time must be a Timestamp, got {type(self.ingest_time).__name__}. "
+                f"An acquisition that happened has a known moment; ADR-0022's null is for "
+                f"historical rows, which have no acquisition to describe.",
+            )
+        if not isinstance(self.instruments, tuple):
+            raise ContractViolation(
+                "/instruments",
+                f"instruments must be a tuple, got {type(self.instruments).__name__}",
+            )
+        if any(not isinstance(i, Identifier) for i in self.instruments):
+            raise ContractViolation(
+                "/instruments", "every instrument must be an Identifier"
+            )
+        if len(set(self.instruments)) != len(self.instruments):
+            raise ContractViolation("/instruments", "instruments must be unique")
+
+    def to_dict(self) -> dict[str, object]:
+        """A structured record, suitable for logging (TIS §0.3) and for the kernel.
+
+        Contains `source_id`, `instrument_id`s and the artifact digest — exactly the fields
+        E5 §14 says to log — and none of the fields it says never to log.
+        """
+        return {
+            "run_id": str(self.run_id),
+            "source_id": str(self.source_id),
+            "artifact_digest": str(self.artifact_digest),
+            "ingest_time": str(self.ingest_time),
+            "instruments": [str(i) for i in self.instruments],
+        }
+
+
+@dataclass(frozen=True)
+class Acquisition:
+    """The pair `acquire` returns (E5 §4), kept together because neither is usable alone.
+
+    An artifact without its provenance is bytes nobody can attribute; provenance without its
+    artifact is a claim about nothing.
+    """
+
+    artifact: RawArtifact
+    provenance: AcquisitionProvenance
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.artifact, RawArtifact):
+            raise ContractViolation(
+                "/artifact",
+                f"artifact must be a RawArtifact, got {type(self.artifact).__name__}",
+            )
+        if not isinstance(self.provenance, AcquisitionProvenance):
+            raise ContractViolation(
+                "/provenance",
+                f"provenance must be an AcquisitionProvenance, got "
+                f"{type(self.provenance).__name__}",
+            )
+        if self.artifact.digest != self.provenance.artifact_digest:
+            raise ContractViolation(
+                "",
+                f"the provenance records digest {self.provenance.artifact_digest.short} "
+                f"but the artifact is {self.artifact.digest.short}. A record that points at "
+                f"different bytes than the artifact it accompanies is worse than no record: "
+                f"it looks like attribution.",
+            )
+
+
+__all__ = ["Acquisition", "AcquisitionProvenance", "RawArtifact"]

contexts/ingest/boundary.py

@@ -0,0 +1,78 @@
+"""The acquisition boundary — the single sanctioned clock read in the system.
+
+ADR-0004: *"`ingest_time` is stamped at the acquisition boundary. This is the single
+sanctioned clock read in the system; determinism elsewhere is governed by ADR-0021."*
+TIS §0.4 states the same rule from the other side: no context may read wall-clock time,
+*"Exception: `ingest_time` capture in E5, which is definitionally a clock read at the
+acquisition boundary."*
+
+WHY THE EXCEPTION LIVES IN ITS OWN MODULE
+------------------------------------------
+`domain/values/Timestamp` deliberately has no `now()`, and `test_timestamp_has_no_clock_reader`
+(M2/E4/#12) pins that. The domain is pure, so the moment the system learned something cannot
+originate there. It has to originate somewhere, and putting it in one named module with one
+exported function means the exception is enumerable: `grep -rn 'ingest.boundary' .` finds
+every place in the repository that reads a clock.
+
+An exception spread across several call sites is not an exception, it is a convention.
+
+WHAT THIS DOES NOT DO
+---------------------
+It does not stamp historical data. E5 §11(ii) is unambiguous: *"No code path writes a non-null
+`ingest_time` for historical data."* There is no function here that takes a row and supplies a
+time for it, and no default. A row that predates bitemporal capture carries `None`, which
+ADR-0022 gives exactly one meaning, and the only way to obtain a stamp is to call `stamp()` at
+the moment of an actual acquisition. Backfilling would require calling it and then attaching
+the result to old rows — which `domain.invariants.ingest_time_is_not_backfilled` detects, and
+which no code here makes convenient.
+"""
+
+from __future__ import annotations
+
+import time
+from datetime import datetime, timezone
+
+from domain.values import Timestamp
+
+
+def stamp() -> Timestamp:
+    """Read the clock at the acquisition boundary and return the instant as a Timestamp.
+
+    Always UTC with an explicit offset, because a local-time stamp is ambiguous twice a year
+    and a naive one is ambiguous always. Microsecond precision is kept: two acquisitions in
+    the same second are two acquisitions, and collapsing them to one instant would make an
+    ordering question unanswerable later.
+    """
+    now = datetime.now(timezone.utc)
+    return Timestamp(now.isoformat().replace("+00:00", "Z"))
+
+
+def stamp_from_epoch(seconds: float) -> Timestamp:
+    """The same stamp, from a caller-supplied epoch instant.
+
+    Exists for the acquisition path that already has the instant a transfer completed — an
+    adapter that records the completion time of a download should record *that* moment, not
+    the slightly later moment it got around to calling `stamp()`.
+
+    It is not a way to fabricate a time. It takes a number a caller must already possess, and
+    supplying an arbitrary one is indistinguishable from lying about when data arrived —
+    which is what the no-backfill invariant checks for, not what a type can prevent.
+    """
+    if isinstance(seconds, bool) or not isinstance(seconds, (int, float)):
+        raise TypeError(f"epoch seconds must be a number, got {type(seconds).__name__}")
+    moment = datetime.fromtimestamp(float(seconds), tz=timezone.utc)
+    return Timestamp(moment.isoformat().replace("+00:00", "Z"))
+
+
+def monotonic() -> float:
+    """A monotonic reading for measuring elapsed time within one acquisition.
+
+    Not a clock in the ADR-0004 sense: it has no epoch, cannot be converted to a Timestamp,
+    and cannot be recorded as when anything happened. It exists so an adapter can implement
+    the retry and timeout policy E5 §15 makes adapter-local without reaching for
+    `time.time()`, which would be a second, unsanctioned wall-clock read.
+    """
+    return time.monotonic()
+
+
+__all__ = ["monotonic", "stamp", "stamp_from_epoch"]