feat: claim confidence scoring, extraction hardening, Claude Code setup - #42
Open
arc-butler wants to merge 15 commits into
Open
feat: claim confidence scoring, extraction hardening, Claude Code setup#42arc-butler wants to merge 15 commits into
arc-butler wants to merge 15 commits into
Conversation
Approved design for rCiv civilization Constitution: - Per-institution Graphiti nodes tagged constitution:<institution> - JSON format, fixed schema, Harness-parseable - Graphiti temporal edges as version control (no git, no extra table) - Amendment flow via constitutional researcher agents + verification pipeline
- Task 8: verification pipeline detects [constitutional-amendment] tag after
scoring; auto-tier calls apply_amendment, human-tier calls flag_for_human_review
- Task 9: GET /constitution, GET /constitution/{institution}, GET /constitution/amendments/pending,
POST /constitution/amendments/{id}/approve — Graphiti reads stubbed, PostgreSQL writes real
- Wires constitution router into main.py
Add GET /webhooks (list all, optional ?event_type filter) and
DELETE /webhooks/{event_type}/{url} (unregister) endpoints to
complement the existing POST /webhooks registration.
- events.list_webhooks() returns a copy of the registry
- events.unregister_webhook() removes a specific URL and cleans
up empty event_type keys
- Tests cover empty, populated, filtered list, 404 on unknown
webhook, and idempotent unregister
Issue #7 - Claim confidence scoring: - Replace NotImplementedError stubs with real implementations - update_claim_confidence: weighted running average based on stance polarity - derive_and_persist_paper_confidence: type-weighted mean of claim scores - derive_paper_confidence from claims auto-triggers assign_tags - Full test suite for all confidence functions Issue #6 - Harden claim extraction: - Minimum body length guard (50 chars minimum) - Maximum body truncation (32K chars to avoid LLM context overflow) - Retry LLM calls with linear backoff (2 attempts) - Global extraction timeout (120s) via asyncio.wait_for - Graceful degradation: persist to PostgreSQL even if Graphiti unavailable - Graceful degradation: skip embedding dedup if embedder fails - Failed extraction leaves paper in queue for retry - All external calls wrapped in try/except with structured logging
- Creates .claude/settings.json with project permissions and hooks - Adds CLAUDE.local.md with PR workflow guidelines - Updates CLAUDE.md with git workflow section - Adds .claude personal overrides to .gitignore
- graphiti_client.py: move all heavy imports (GroqClient, GeminiClient, google.genai, Graphiti) inside init_graphiti() — prevents module-level ImportError cascade when dependencies are missing in test env - postgres.py: move psycopg_pool import inside init_pool() — libpq not needed until actual DB operations - confidence.py: replace module-level from backend.db.postgres import get_pool with local imports inside each async function — pure function tests now load without libpq - conftest.py: wrap DB setup in try/except, skipping gracefully when dependencies missing instead of crashing test collection - test_confidence.py: fix mock patch targets (backend.db.postgres.get_pool) to match lazy import pattern; fix _make_pool_mock to handle arbitrary execute counts without StopAsyncIteration
Implements Issue #34 — Court ruling confidence score and tag update: - Add POST /claims/{claim_id}/evaluate endpoint that accepts stance (supports|contradicts|qualifies), weight (0-1), and source_id - Internally calls update_claim_confidence to adjust claim score, then derive_and_persist_paper_confidence to recompute parent paper - Add update_claim_status to confidence.py — derives claim status from score using same thresholds as papers (verified/disputed/preliminary) - Wire update_claim_status into update_claim_confidence so status updates automatically on each evaluation - 7 new unit tests + 1 integration test for the evaluate endpoint - All 47 tests pass (claims_api + confidence + verification)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement claim confidence scoring with weighted running average, harden paper extraction pipeline, and add Claude Code project configuration.
Changes
Confidence Scoring (new):
backend/scoring/confidence.py— weighted running average scoring for claimsExtraction Hardenings:
Claude Code Setup:
.claude/settings.json— project permissions and lint hooksCLAUDE.mdupdated with git workflow sectionFiles Changed
backend/scoring/confidence.py— full scoring implementation (112 lines)backend/tests/test_confidence.py— comprehensive test suite (290 lines)backend/extraction.py,backend/events.py,backend/verification.py— hardening.claude/settings.json,CLAUDE.md,.gitignore— tooling setupTest Plan
pixi run test— all existing tests passbackend/tests/test_confidence.pycovers scoring edge casesReviewer: @ARC345