[WIP] V2: Integration of Nemotron suite, Zetaris & H-MEM into Blue Intelligence

.env.example

@@ -0,0 +1,26 @@
+# Copy to .env (gitignored) and fill in. Placeholders only: never commit real values.
+# With no .env at all, backend/run.py runs in SAMPLE_MODE on bundled data.
+
+# Zetaris (data layer)
+ZETARIS_REST_URL=http://localhost:8888/api/v1.0
+ZETARIS_ORG_ID=your-org-id
+ZETARIS_API_KEY=your-zetaris-api-key
+
+# Meterless Nightshift (OpenAI-compatible runtime)
+METERLESS_BASE_URL=http://127.0.0.1:8088/v1
+METERLESS_RUNTIME_TOKEN=your-nightshift-runtime-token
+METERLESS_MODEL=meterless/auto
+
+# Optional: direct NVIDIA Nemotron fallback when Nightshift is unreachable
+NVIDIA_API_KEY=
+
+# Optional: self-hosted SearXNG for the PoE search step (infra/searxng/).
+# 8890 is also the default; 8888-8889 belong to the local Zetaris API.
+SEARXNG_URL=http://127.0.0.1:8890
+
+# true forces bundled sample data and no model calls; leave empty to auto-detect
+SAMPLE_MODE=
+
+# python backend/run.py --serve
+PORT=8000
+LOG_LEVEL=INFO

.gitignore

@@ -43,3 +43,10 @@ credentials.json
 backend/data/climatology/**/*.partial.npz
 backend/data/climatology/**/*.npz.tmp
 logs/*.jsonl
+data/hmem/
+data/.lock
+# Independent port registries downloaded by backend/scripts/zetaris_setup.py --part registries
+backend/data/registries/
+
+# Fixtures volumineuses — ne jamais suivre par Git (limite GitHub 100 Mo)
+fixtures/

README.md

@@ -3,15 +3,14 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE-MIT)
 [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE-APACHE)
 [![NVIDIA Nemotron](https://img.shields.io/badge/NVIDIA-Nemotron-76B900.svg)](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/)
-[![Nebius Token Factory](https://img.shields.io/badge/Nebius-Token%20Factory-0B1F3A.svg)](https://tokenfactory.nebius.com/)
 
 A [Berry-Mappemonde](https://berrymappemonde.org) project. Code is **MIT or Apache-2.0**.
 
 > [!WARNING]
-> **Ne convient pas à la navigation / Not for navigation.**
-> Les cartes et les chiffres viennent de sources publiques et de calculs.
-> Ce n’est pas une carte marine officielle. Vérifiez toujours les publications
-> gouvernementales avant une décision en mer.
+> **Not for navigation.**
+> Maps and figures come from public sources and computations.
+> This is not an official nautical chart. Always check government
+> publications before making any decision at sea.
 
 **Blue Intelligence** ([blueintelligence.online](https://blueintelligence.online))
 is the memory of maritime sources: Ports of Entry per EEZ, marinas,
@@ -24,6 +23,172 @@ The live voyage application **NAVIGUIDE** is a separate repository:
 ([naviguide.fr](https://naviguide.fr)). It reads Blue Intelligence over its
 public API, `https://blueintelligence.online/api`.
 
+## PoE Reconciler (Open Agent Hackathon 2026)
+
+**What it does.** Give it one maritime zone (an EEZ, such as Bermuda or the Azores)
+and it returns that zone's Ports of Entry, the places where a sailor may legally
+clear customs and immigration on arrival. Each port comes with a verdict, a 0–100
+confidence score with reasons, its sources, and an **evidence tier** that says what
+backs it: an official government page that designates it, an independent port
+registry (UN/LOCODE, NGA World Port Index), both, or neither. A short brief for
+sailors closes each run. The answer is built from scattered public sources, and a
+registry listing alone is never presented as port-of-entry status.
+
+- **Tier = legal-status evidence:** what says this port is a port of entry (an official page, a registry, both or neither).
+- **Verdict = data-quality review:** whether the record is reliable enough to use as is (`pass`) or needs a human look (`flag`), from its sources, reading and coordinates.
+
+### Architecture
+
+```text
+RunPipeline ─► ZetarisAdapter  candidates from Zetaris (blue_intel.poe_ports)
+            ─► ZetarisAdapter  registry check (UN/LOCODE + WPI views in Zetaris)
+                               + stored official evidence → evidence tier
+            ─► HMemAgent       recall earlier verdicts (Meterless H-MEM engine)
+            ─► EvaluatorAgent  score each port: pass, flag or retry
+            ─► NemotronJudge   advisory critique of the official text (Nemotron)
+            ─► EvaluatorAgent ⇄ ResearchAgent   retry with memory (max 2)
+            ─► EvaluatorAgent  finalise: pass or flag, never dropped
+            ─► BriefingAgent   brief via Meterless Nightshift → NVIDIA Nemotron
+            ─► HMemAgent       remember verdicts (created / reinforced / superseded)
+```
+
+- **Agents:** seven named agents share one `run_id` and one JSONL trace, with
+  handoffs, critique and retry events: `RunPipeline` (the orchestrator),
+  `ZetarisAdapter`, `HMemAgent`, `EvaluatorAgent`, `NemotronJudge`,
+  `ResearchAgent` and `BriefingAgent`. Each model call is also traced as
+  `MeterlessGenerator`, recording the requested and the served model.
+- **NemotronJudge (advisory critic):** one Nemotron call per zone, through
+  Nightshift, on the official evidence text the run already holds. It records a
+  verdict (`poe`, `not_poe`, `unclear`), a reason and a quoted excerpt per port,
+  and whether that excerpt is really in the evidence text. It never overrides
+  evidence: no tier, validation, official evidence or score comes from it. When it
+  calls a flagged port a port of entry, it triggers one more research round. It is
+  skipped, and the trace says why, without credentials (`SAMPLE_MODE`) or without
+  an answer. Code: [`poe_judge.py`](backend/app/agents/poe_judge.py); tests:
+  [`test_poe_judge.py`](backend/tests/test_poe_judge.py); live recording:
+  [`01_bermuda.live_nemotron.json`](output_examples/01_bermuda.live_nemotron.json)
+  and [its trace](output_examples/01_bermuda.live_nemotron.trace.jsonl).
+- **Zetaris (data layer):** ports, and the two independent registries, are read
+  through Zetaris SQL with a read-only guard, a timeout and a fallback to bundled
+  real fixtures.
+- **Evidence tiers:**
+
+  | Tier | Meaning |
+  | --- | --- |
+  | `tier1` Official Commercial Port of Entry | official page designates it, and a registry lists it |
+  | `tier2` Official Yachting Port of Entry | official page designates it, and no registry lists it (typical for marinas) |
+  | `tier_registered` | registry match only; entry status unverified |
+  | `tier3` | neither: review needed |
+
+  An official page counts only if its URL is a government domain and its text
+  names the port near an entry term. A strong term ("port of entry", "clear
+  customs", "designated marina") is enough on its own. A weak term ("customs",
+  "immigration") also needs a port word nearby and the port's full name. Pages
+  fetched during generation are stored as verbatim snippets, never model output.
+  Runs then look the snippets up by zone.
+- **Meterless:** the H-MEM engine (a Python port that passes the 13 official
+  conformance vectors) makes memory load-bearing. Nightshift is the primary model
+  provider and serves Nemotron.
+
+### Quickstart: fresh clone, no credentials, no database
+
+```bash
+git clone https://github.com/Berry-Mappemonde/Blue-Intelligence.git
+cd Blue-Intelligence
+python3 -m venv backend/.venv   # Python 3.12+ (macOS's /usr/bin/python3 is 3.9: too old)
+backend/.venv/bin/pip install fastapi==0.110.1 httpx==0.28.1 pydantic==2.13.4 \
+    python-dotenv==1.2.3 uvicorn==0.25.0 filelock==3.32.3 pycountry==26.2.16
+backend/.venv/bin/python backend/run.py --zone Bermuda
+backend/.venv/bin/python backend/run.py --input input_examples/01_bermuda.json --output bermuda.json
+```
+
+With no `.env`, `run.py` turns on `SAMPLE_MODE` and logs it on stderr. In this
+mode the agents run for real on bundled data, and only the Zetaris and model
+calls are replaced. Expected output:
+
+- **stderr:** a summary block (abridged):
+
+  ```text
+  WARNING run: SAMPLE_MODE enabled automatically; missing: ZETARIS_API_KEY, ZETARIS_ORG_ID, METERLESS_RUNTIME_TOKEN or MODEL_API_KEY
+  Pipeline Result
+  Zone:     Bermuda
+  Mode:     sample
+  Ports:    1
+  Summary:  {'pass': 1}
+  LLM:      sample / caller-sample-fn
+  Memory:   0 recalled, remembered
+  ```
+
+- **stdout:** each trace event as one JSON line, then the result JSON, with keys
+  `run_id`, `zone`, `mode`, `ports`, `summary`, `brief`, `llm`, `components`,
+  `registry`, `judge`, `retries`, `memory`, `trace_file`, `timings_ms` and `error`.
+  `--output` also writes the result to a file.
+- **Trace file:** `logs/trace-<run_id>.jsonl`. For the input example that's
+  `logs/trace-example-bermuda.jsonl`.
+- **Memory:** run the same zone again and `memory.recalled` is no longer 0, because
+  H-MEM stored the first run in `data/hmem/`.
+
+The 20 sample zones include Bermuda, Angola, Azores, Alaska, Bahamas and Antigua
+and Barbuda. `input_examples/04_benin.json` shows a retry round: EvaluatorAgent
+critiques the port, ResearchAgent looks for a remembered official source, twice. Across them the sample run gives 2 `tier1`, 2 `tier2`, 11
+`tier_registered` and 5 `tier3`. Official evidence comes from small verbatim
+snapshots of real government pages in
+[`backend/data/sample/official_pages/`](backend/data/sample/official_pages/).
+
+The same pipeline runs over HTTP:
+
+```bash
+backend/.venv/bin/python backend/run.py --serve   # http://127.0.0.1:8000; PORT=… if taken
+curl -s -X POST http://127.0.0.1:8000/run -H 'Content-Type: application/json' \
+     -d '{"zone_name": "Angola"}'
+```
+
+It also serves `GET /healthz` and a demo page at `/`.
+
+### Live mode
+
+```bash
+cp .env.example .env     # fill in the values below
+backend/.venv/bin/python backend/run.py --input input_examples/02_angola.json
+```
+
+- **Zetaris:** `ZETARIS_REST_URL`, `ZETARIS_ORG_ID` and `ZETARIS_API_KEY`. The
+  PoE view and the registry views are registered once by
+  `backend/scripts/zetaris_setup.py`.
+- **Meterless Nightshift:** `METERLESS_BASE_URL` (default
+  `http://127.0.0.1:8088/v1`), `METERLESS_RUNTIME_TOKEN` and `METERLESS_MODEL`
+  (default `meterless/auto`).
+- **Optional:** `NVIDIA_API_KEY`, for a direct Nemotron fallback when Nightshift
+  is down.
+
+Every external call has a timeout and a fallback, in this order: Nightshift, then
+NVIDIA, then the sample brief. A Zetaris failure falls back to the bundled
+fixtures. `llm.provider` and `llm.model` report the model that actually wrote the
+brief; Nightshift may route to a different Nemotron than the one requested.
+
+### Sponsor technology and evidence
+
+[`docs/sponsors/SPONSOR_TECH.md`](docs/sponsors/SPONSOR_TECH.md) (merges with
+PR #31) links every claim about Zetaris, Meterless H-MEM, Nightshift and NVIDIA
+Nemotron to the code and tests that back it. Setup details, bounds and tests are
+in [`docs/RUNNING.md`](docs/RUNNING.md).
+
+### Limits, stated plainly
+
+- **Sample data is small:** 20 real ports, one per zone, and small real
+  subsets of the registries. Official evidence exists only for zones whose
+  official pages have been fetched: 4 zones in the sample.
+- **Tier and verdict can differ** (definitions above). Only a port with official
+  evidence can have its position corroborated by a registry (≤ 2 km), which `pass`
+  needs: St. George's passes (`tier1`, 67), Road Bay stays `flag` (`tier_registered`).
+- **Grounding is a heuristic:** it checks the official domain, the entry term
+  and the distance. A port name with "Port of" stripped can match the town.
+  The committed snapshots were reviewed by hand.
+- **Recorded output:** the recorded live run in `output_examples/` is for
+  reference. Nothing reads it, and judges cannot reproduce it without
+  credentials.
+- **Not for navigation:** always check official publications before arriving.
+
 ## Dependencies
 
 | Part | Runtime | Dependencies |
@@ -42,7 +207,7 @@ cd Blue-Intelligence
 # API
 cd backend
 python3.12 -m venv .venv
-.venv/bin/pip install -r requirements.txt
+.venv/bin/pip install -r requirements.txt   # pins a Linux-only torch +cpu wheel: on macOS use the Quickstart pins above, or install requirements.txt without the torch line
 cat > .env <<'EOF'
 MONGO_URL=mongodb://127.0.0.1:27017
 DB_NAME=blueintel_db

backend/app/adapters/__init__.py

@@ -0,0 +1 @@
+"""Adapters for external data sources."""

backend/app/adapters/evidence_store.py

@@ -0,0 +1,152 @@
+"""Official-source evidence records: stored snippets of fetched official text.
+
+A record is built deterministically from text fetched at generation (or from the
+committed official-page snapshots in sample mode), never from model output:
+
+    {mrgid, port_key, url, domain, matched_name, snippet, content_sha256, fetched_at,
+     extract_level}
+
+``snippet`` is a verbatim slice (at most 400 chars) around a mention of the port that
+passes registry_check.entry_mention: an official URL, the port's name near an entry term,
+not inside an address. At most MAX_PER_PORT records per port (best match per URL).
+
+Records live in the Mongo collection ``poe_evidence`` (unique on mrgid + port_key + url),
+not on the port document, because the run pipeline and /api/poe/verify read ports from
+Zetaris. ``lookup`` reads Mongo, or the snapshots in backend/data/sample/official_pages/
+in SAMPLE_MODE or when Mongo is unreachable; any other failure means no evidence, so a
+lookup can never make a port official on its own.
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+
+from app.adapters.zetaris import ZetarisClient
+from app.agents import registry_check
+from app.agents.evaluator import is_official_source
+from app.core import paths
+
+COLLECTION = "poe_evidence"
+MAX_PER_PORT = 3
+MONGO_TIMEOUT_MS = 2000
+SAMPLE_PAGES_DIR = paths.SAMPLE_DATA_DIR / "official_pages"
+RECORD_FIELDS = ("mrgid", "port_key", "url", "domain", "matched_name", "snippet",
+                 "content_sha256", "fetched_at", "extract_level")
+
+
+def sha256_text(text: str) -> str:
+    return hashlib.sha256((text or "").encode("utf-8")).hexdigest()
+
+
+def build_records(mrgid: int | None, port_names: list[str], documents: list[dict]) -> list[dict]:
+    """Evidence records for each port name from official documents.
+
+    documents: [{url, text, sha256, fetched_at, extract_level}], where sha256 is the hash of
+    the page's full fetched text (text may be a section of it). Only official URLs count;
+    each port keeps its closest match per URL, MAX_PER_PORT URLs at most.
+    """
+    if mrgid is None:
+        return []
+    official = [d for d in documents if d.get("text") and is_official_source([d.get("url") or ""])]
+    records = []
+    for name in dict.fromkeys(n for n in port_names if n):
+        variants = registry_check.grounding_variants(name)
+        best: dict[str, tuple[int, dict]] = {}
+        for doc in official:
+            mention = registry_check.entry_mention(doc["text"], variants)
+            if not mention:
+                continue
+            record = dict(mrgid=int(mrgid), port_key=registry_check.port_key(name), url=doc["url"],
+                          domain=registry_check._domain(doc["url"]), matched_name=mention["matched_name"],
+                          snippet=mention["snippet"], content_sha256=doc.get("sha256") or sha256_text(doc["text"]),
+                          fetched_at=doc.get("fetched_at"), extract_level=doc.get("extract_level"))
+            if doc["url"] not in best or mention["gap"] < best[doc["url"]][0]:
+                best[doc["url"]] = (mention["gap"], record)
+        records += [r for _, r in sorted(best.values(), key=lambda item: item[0])][:MAX_PER_PORT]
+    return records
+
+
+def documents_from_context(text: str | None, used_sources: list[dict]) -> list[dict]:
+    """Generation documents: the "[SOURCE: url]" blocks of the collected text with their
+    fetch metadata (sha256 of the full fetched text, collected_at, extraction level)."""
+    meta = {s.get("url"): s for s in used_sources or []}
+    return [dict(url=url, text=body, sha256=(meta.get(url) or {}).get("sha256"),
+                 fetched_at=(meta.get(url) or {}).get("collected_at"),
+                 extract_level=(meta.get(url) or {}).get("level"))
+            for url, body in registry_check.source_texts_from_context(text).items()]
+
+
+async def store(db, records: list[dict]) -> int:
+    """Upsert records (one per mrgid + port_key + url) and keep the newest MAX_PER_PORT per port."""
+    if not records:
+        return 0
+    coll = db[COLLECTION]
+    await coll.create_index([("mrgid", 1), ("port_key", 1), ("url", 1)], unique=True,
+                            name="mrgid_port_key_url")
+    for record in records:
+        doc = {k: record.get(k) for k in RECORD_FIELDS}
+        await coll.update_one({k: doc[k] for k in ("mrgid", "port_key", "url")}, {"$set": doc}, upsert=True)
+    for mrgid, key in {(r["mrgid"], r["port_key"]) for r in records}:
+        stale = await coll.find({"mrgid": mrgid, "port_key": key}, {"_id": 1}) \
+            .sort("fetched_at", -1).skip(MAX_PER_PORT).to_list(100)
+        if stale:
+            await coll.delete_many({"_id": {"$in": [d["_id"] for d in stale]}})
+    return len(records)
+
+
+def sample_documents(mrgid: int | None) -> list[dict]:
+    """Documents from the committed official-page snapshots for a zone (one per section)."""
+    if mrgid is None or not SAMPLE_PAGES_DIR.is_dir():
+        return []
+    docs = []
+    for path in sorted(SAMPLE_PAGES_DIR.glob("*.json")):
+        page = json.loads(path.read_text(encoding="utf-8"))
+        if int(mrgid) not in {int(z["mrgid"]) for z in page.get("zones") or []}:
+            continue
+        docs += [dict(url=page["url"], text=section["text"], sha256=page["sha256"],
+                      fetched_at=page["fetched_at"], extract_level=page.get("extract_level"))
+                 for section in page.get("sections") or []]
+    return docs
+
+
+def _mongo_records(mrgid: int) -> list[dict]:
+    from pymongo import MongoClient
+    client = MongoClient(os.environ.get("MONGO_URL") or "mongodb://127.0.0.1:27017",
+                         serverSelectionTimeoutMS=MONGO_TIMEOUT_MS, connectTimeoutMS=MONGO_TIMEOUT_MS,
+                         socketTimeoutMS=MONGO_TIMEOUT_MS)
+    try:
+        coll = client[os.environ.get("DB_NAME") or "bi_local"][COLLECTION]
+        return list(coll.find({"mrgid": int(mrgid)}, {"_id": 0}).limit(500))
+    finally:
+        client.close()
+
+
+def lookup(mrgid: int | None, port_names: list[str], run_id: str) -> list[dict]:
+    """Stored evidence records for a zone's ports; [] on any failure (never official).
+
+    SAMPLE_MODE=true reads the official-page snapshots and builds records with
+    build_records, as generation does. Otherwise Mongo (poe_evidence); if Mongo is
+    unreachable, the snapshots. Traced as a ZetarisAdapter ``evidence_lookup`` event.
+    """
+    source, status, records, error = "none", "ok", [], None
+    try:
+        if mrgid is None:
+            source = "no-mrgid"
+        elif (os.environ.get("SAMPLE_MODE") or "").strip().lower() == "true":
+            source, records = "sample_pages", build_records(mrgid, port_names, sample_documents(mrgid))
+        else:
+            try:
+                source, records = "mongo", _mongo_records(mrgid)
+            except Exception as exc:  # noqa: BLE001 - Mongo down: use the snapshots
+                error, status = f"{type(exc).__name__}: {str(exc)[:120]}", "fallback"
+                source, records = "sample_pages", build_records(mrgid, port_names, sample_documents(mrgid))
+    except Exception as exc:  # noqa: BLE001 - no evidence is the safe answer
+        source, status, records, error = "error", "error", [], f"{type(exc).__name__}: {str(exc)[:120]}"
+    by_port = {}
+    for r in records:
+        by_port[r.get("port_key")] = by_port.get(r.get("port_key"), 0) + 1
+    ZetarisClient(run_id=run_id).trace_event(
+        "evidence_lookup", status=status, source=source, records=len(records), records_by_port=by_port,
+        input_summary={"mrgid": mrgid, "ports": len(port_names)}, **({"error": error} if error else {}))
+    return records