refactor: modular restructure across memory/runtime/proactive/web#3
Merged
Conversation
pytest_collection_modifyitems receives the global items list, not just items under the conftest's directory. The previous implementation applied the eval-corpus skip marker to every collected item when the corpus file was missing — silently skipping the entire 1432-test suite on hosts without the gitignored develop-docs/memory/05-eval-corpus file. Filter items by nodeid prefix before applying the marker so the skip is contained to tests/eval/. Add tests/test_eval_conftest_scope.py that runs pytest on a known non-eval test and verifies it actually runs (not silently skipped) when the corpus is absent.
Both functions are imported by callers via the deep module path (echovessel.memory.retrieve.retrieve, echovessel.memory.consolidate.consolidate_session). Add them to the package-level re-export so they are reachable from echovessel.memory directly and declared in __all__ alongside the other public symbols. Pure declaration cleanup — no behavior change.
The contents of voice/models.py are dataclass value objects
(VoiceResult), not SQLModel ORM tables — unlike memory/models.py which
holds the actual database schema. Sharing the file name across
subsystems with different intent invites confusion.
Rename to types.py to match what's inside, and update all 8 import
sites under src/ and tests/ plus the two doc references in
docs/{en,zh}/voice.md.
Move retrieve.py into retrieve/core.py and add retrieve/__init__.py that re-exports the full public surface (scoring constants, ScoredMemory / RetrievalResult / ConceptSearchHit, loaders, entry points, event-time helpers). Behavior unchanged. The sub-package layout makes room for the content split in commit D1 (core / scoring / search) without changing any caller's import path — `from echovessel.memory.retrieve import retrieve` continues to resolve.
Move consolidate.py into consolidate/core.py and consolidate_tracer.py into consolidate/tracer.py. Add consolidate/__init__.py that re-exports the entry point (consolidate_session), Phase A gates (is_trivial + trivial thresholds), reflection thresholds, extraction DTOs, callable shapes (ExtractFn / ReflectFn / EmbedFn), and the trace recorder constructors. Update internal imports in core.py / slow_cycle.py / consolidate_worker.py / tests to point at the new tracer location. Behavior unchanged. The sub-package layout sets up D2's split of core into core / phase_a / phase_bce.
Empty __init__.py shells with destination docstrings. Files move into them in commits C1 / C2 / C3. Splitting the scaffold into its own commit keeps the subsequent git mv diffs focused on file motion rather than directory creation.
Empty __init__.py shells with destination docstrings. Files move into them in commit C4. Same scaffold-then-move split as B3 — keeps the subsequent git mv diffs focused on file motion.
Three files move into the new sub-package:
- interaction.py -> turn/coordinator.py (also renamed; the file
orchestrates a turn and 'coordinator' is more precise than
'interaction', which read as a synonym for turn anyway)
- turn_dispatcher.py -> turn/dispatcher.py (drop redundant prefix)
- turn_tracer.py -> turn/tracer.py (drop redundant prefix)
Update 17 import sites under src/, tests/, and docs/{en,zh} to point
at the new paths. interaction.py used to re-export IncomingMessage /
IncomingTurn for legacy callers; the canonical home in
echovessel.channels.base is unchanged and the comment is updated to
match the new module location. Behavior unchanged.
consolidate_worker.py and idle_scanner.py move under loops/. Both are asyncio tick loops (wake on timer, scan for work, sleep) — not job queue consumers — so 'loops' is more precise than 'workers'. Update 13 import sites under src/ and tests/. Behavior unchanged.
Five files move under wiring/, four with name simplification (the
sub-package's name now carries the role context):
- memory_facade.py -> wiring/memory.py
- importer_facade.py -> wiring/importer.py
- prompts_wiring.py -> wiring/prompts.py
- memory_observers.py -> wiring/memory_observer.py (singular: file
contains exactly one class, RuntimeMemoryObserver)
- persona_extraction.py -> wiring/persona_extraction.py (kept as-is;
name describes a distinct operation)
"adapters" was the original candidate but didn't fit: hexagonal
adapters mean external-world boundary, while this cluster is the
runtime composition root — DI factories, mediators, registries. wiring
matches what Python DI projects (pinject, dependency-injector) call
this.
Update 18 import sites and 6 docstring references across src/ and
tests/. Behavior unchanged.
Nine flat files re-grouped into three sub-packages by role:
- core/ : base.py / config.py / errors.py — Protocols + value types,
ProactiveConfig, exception hierarchy
- engines/: policy.py / generator.py — decision logic (when to speak,
what to say)
- execution/: scheduler.py / queue.py / delivery.py / audit.py —
tick loop, event queue, channel routing, decision audit
Each sub-package's __init__.py re-exports its public surface, and
proactive/__init__.py continues to re-export everything callers
already get via 'from echovessel.proactive import X'. Deep imports
(from echovessel.proactive.base import …) update to the new
sub-package paths.
Update 30+ import sites across src/, tests/, and docs/{en,zh}
proactive.md. Behavior unchanged.
retrieve/core.py was 1123 lines with three concerns fused: the main retrieve() pipeline, the rerank scoring weights + helpers, and the admin-side concept-node search/listing. Split per the audit: - scoring.py (141 lines) — WEIGHT_*, ScoredMemory, _recency_score / _relevance_score / _impact_score / _relational_bonus, _score_node. Tuning rerank now happens in one focused file with the rationale comment for DEFAULT_MIN_RELEVANCE adjacent to the constant. - search.py (330 lines) — search_concept_nodes (FTS5 + LIKE fallback), _build_like_snippet, list_concept_nodes, plus the ConceptSearchHit dataclass. - core.py (696 lines) — retrieve() entry, L1 load_core_blocks, L2 list_recall_messages, entity anchor helpers, force-load thoughts, event-time delta rendering, and _expand_session_context. retrieve/__init__.py keeps the same public surface — every name that was reachable via 'from echovessel.memory.retrieve import X' before still resolves. Behavior unchanged.
consolidate/core.py was 1069 lines mixing the consolidate_session orchestration with phase-specific helpers. Split per the audit: - phase_a.py (90 lines) — TRIVIAL_MESSAGE_COUNT / TRIVIAL_TOKEN_COUNT, STRONG_EMOTION_KEYWORDS, _has_strong_emotion, is_trivial. The trivial-skip gate now reads as one focused module. - phase_bce.py (282 lines) — SHOCK_IMPACT_THRESHOLD, TIMER_REFLECTION_HOURS, REFLECTION_HARD_LIMIT_24H constants; _count_reflections_24h, _is_timer_due, _fallback_source_turn_id, _load_reflection_inputs, _consolidate_entities (entity resolution + L3↔L5 junction wiring with the surface-form filter). - core.py (759 lines) — orchestration only: consolidate_session entry point, ExtractedEvent / ExtractedEntity / etc DTOs, ExtractFn / ReflectFn / EmbedFn callable shapes, ConsolidateResult. consolidate/__init__.py keeps the public surface stable. phase_bce imports ExtractionResult / EmbedFn from core under TYPE_CHECKING to avoid the circular dependency that would arise from a direct import. Behavior unchanged.
coordinator.py was 1808 lines mixing turn orchestration with the pure prompt-rendering helpers. Split per the audit: - prompt_assembly.py (~700 lines) — STYLE_INSTRUCTIONS, PersonaFactsView, DAY_BUCKET_ORDER, _DAY_NAMES_EN, day_bucket_of, _format_now_section, _format_episodic_state_section, _load_anchored_entity_descriptions, _render_entity_disambiguation_hint, _node_description, _load_active_intentions, _load_pending_expectations, _load_recent_session_summaries, build_system_prompt, build_turn_user_prompt, build_user_prompt. Pure functions — given the same inputs they produce the same prompt text. - coordinator.py (~900 lines) — orchestration only: assemble_turn (the 12-stage pipeline), TurnContext, AssembledTurn, OnTokenCb / OnTurnDoneCb, _pending_id_for_turn, _invoke_on_turn_done, maybe_decay_episodic_state (runtime mutation, not prompt assembly), EXPECTATION_MATCH_COSINE_THRESHOLD, _cosine, check_pending_expectations. Update 9 test imports to point at prompt_assembly for the prompt-side symbols. coordinator's __all__ slimmed to its actual surface (prompts no longer re-exported). Behavior unchanged.
… + helpers admin.py was 3733 lines: 17 Pydantic models + 20 helpers + 49 inline route handlers all in one file. First step of the split: turn admin.py into admin/__init__.py and pull the schemas + helpers into focused sibling files. - admin/models.py (390 lines) — every wire-format Pydantic class (PersonaFactsPayload + 16 others) plus _enum_or_none, the soft-normalisation helper that backs PersonaFactsPayload's enum field validators. - admin/helpers.py (702 lines) — block-label mappings, persona block / fact helpers, ConceptNode serializer, avatar storage, channel config + status loaders, voice clone sample store, and the display-name persistence helper. - admin/__init__.py (2774 lines) — orchestration only: the build_admin_router factory and every route handler. Imports the schemas + helpers it actually uses; the long block of inline class / function definitions is gone. Behavior unchanged. The route-by-domain split into persona.py / memory.py / voice.py / diagnostics.py / config.py / core.py is the follow-on commit (E1b).
…ring guards Two new contracts cover invariants that emerged from the restructure: - proactive sub-package layering · execution -> engines -> core. core is foundational types/config/errors, engines decide before execution delivers. A reverse import (engines->execution or core->either) would be a regression. - runtime.wiring is the composition root. It builds adapters and factory closures that runtime.app hands to channels / loops / turn at startup. A turn-or-loops import inside wiring/ would couple the wiring graph to per-turn or background-loop code and break the build-once-hand-out-everywhere model. The original layered contract (runtime -> channels|proactive -> memory|voice -> core) and the proactive-no-runtime-no-prompts forbidden contract are unchanged. Total contracts: 2 -> 4, all green against the current tree.
Sweep docs/en/*.md for source-code path references that broke when files moved during the restructure. 9 references updated across 5 files: - memory.md (4) · consolidate_session entry path, retrieve entry path, three SHOCK/TIMER threshold pointers, runtime memory observer location - runtime.md (1) · runtime memory observer location - import.md (2) · importer facade location (mediator + authoritative source) - channels.md (1) · turn dispatcher + turn coordinator paths - proactive.md (1) · AuditSink Protocol module path No content changes — only src path strings updated to match the new sub-package layout. The bilingual sync to docs/zh lands in F3.
Mirror F2's path updates into docs/zh/*.md so the bilingual docs stay in lockstep (CLAUDE.md mandates docs/en and docs/zh land together). Same 9 references, identical translation context.
Update the tree to surface the new sub-packages (memory/retrieve,
memory/consolidate, proactive/{core,engines,execution},
runtime/{turn,loops,wiring}, channels/web/routes/admin/) so the
on-ramp doc matches what's actually under src/echovessel/.
Refresh the layered-architecture summary to reference all four
import-linter contracts: the original layered + proactive-no-runtime
plus the two new ones (proactive sub-package layering and
runtime.wiring isolation, both added in F1).
_ExplodingBackend.insert_vector now accepts conn= and passes it to super(), matching SQLiteBackend.insert_vector's current signature. The test exercises consolidate's events loop rolling back when a vector write raises mid-loop, and the prior xfail marker is dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Single-track restructure plus follow-ups. 30 commits, every one green at landing.
Sub-package introductions (skeleton → file moves → content splits):
memory/retrieve/(core / scoring / search) andmemory/consolidate/(core / phase_a / phase_bce / tracer)runtime/turn/(coordinator / prompt_assembly / dispatcher / tracer),runtime/loops/(consolidate_worker / idle_scanner),runtime/wiring/(memory / importer / prompts / memory_observer / persona_extraction)proactive/{core, engines, execution}/Admin routes split (
channels/web/routes/admin/):admin/__init__.pybecomes a 79-LOC composition entry; 49 route handlers split acrosscore.py/config.py/voice.py/diagnostics.py/persona.py/memory.pyviaregister_*_routes(router, *, ...)functions._persona_id/_open_dbmove from closures to module-level helpers inhelpers.py.Layered architecture (import-linter) — 1 contract becomes 4:
runtime → channels|proactive → memory|voice → coreproactivemust not importruntimeorpromptsproactive.execution → engines → coreruntime.wiringmust not importruntime.turnorruntime.loopsOther:
voice/models.py→voice/types.py(it holds dataclass value objects, not SQLModel ORM)tests/eval/conftest.pyskip marker now scoped totests/eval/items only (was silently skipping the whole suite when the eval YAML was absent)tests/voice/test_fishaudio.pymocks rewired to fish_audio_sdk's current flatcreate_model/list_modelsAPItests/perf/test_llm_latency.pyremoved (never run, never updated since initial snapshot)docs/{en,zh}/paths updated to reflect newsrc/echovessel/layoutCLAUDE.mdProject structure section refreshedTest plan
uv run pytest→ 1413 passed, 17 skipped, 7 deselected, 0 failed, 0 xfaileduv run ruff check src/ tests/→ cleanuv run ruff format --check src/ tests/→ cleanuv run lint-imports→ 4 contracts kept, 0 brokenbuild_admin_router(...)smoke-instantiates with all 49 routes registered, no path-method shadowing