[FEATURE]: Plugin state transfer API for TenantPluginManager rebuild

[jonpspri]

Summary

When a TenantPluginManager instance is rebuilt — either because its TTL expired or because reload_tenant()/invalidate_all() was called — there is no mechanism for a plugin to preserve its runtime state across the transition. The old instance is shut down and the new instance starts cold, losing any in-memory state the plugin had accumulated.

Context

This is filed on behalf of ContextForge issue IBM/mcp-context-forge#5141, where the original request is described as:

Plugins should have a transferable state object which can be set by the plugin creator. This plugin state needs to be transported to the new instance of the plugin on rebuild to maintain state consistency.

ContextForge's TenantPluginManagerFactory._build_manager() already holds a reference to the outgoing manager before it calls shutdown() on it (gateway_plugin_manager.py, lines 183–191), so the gateway side is ready to extract and inject state — it just needs TenantPluginManager to expose the API.

Requested API (sketch)

A minimal addition to TenantPluginManager (and optionally PluginManager) that allows:

1. Exporting state from the outgoing instance before shutdown:

   state: dict[str, Any] = await old_manager.export_plugin_states()
   # Returns a mapping of plugin_name -> opaque state blob

2. Importing state into the freshly built instance before (or during) initialize():

   await new_manager.initialize(initial_states=state)

3. Plugin-side hooks — each plugin can opt in by implementing:

   async def get_state(self) -> Any: ...       # called during export
   async def set_state(self, state: Any): ...  # called during import, before first hook

The state blob should be opaque to the framework — the plugin is responsible for serialisability and version compatibility.

Why this matters

• Rate-limiter plugins lose their per-token counters on every TTL rebuild (default 30 s in ContextForge).
• Retry-with-backoff plugins lose their per-client backoff tables.
• Custom stateful enrichment plugins lose cached lookup results or connection handles that are expensive to rebuild.

Without this, plugin authors are forced to use an external store (Redis, DB) for any state that must survive a manager rebuild, even when the state is node-local and ephemeral.

Alternatives considered

• External state store: works but adds latency and operational overhead for state that is inherently process-local.
• Increase TTL to avoid rebuilds: masks the problem; invalidation on config change still triggers a cold rebuild.
• Bypass via PluginContextTable: the per-request context table is request-scoped, not persistent across rebuilds.

Acceptance criteria

• TenantPluginManager exposes export_plugin_states() and accepts initial_states on initialize() (or equivalent API).
• Plugin base class / protocol defines optional get_state() / set_state() hooks.
• State transfer is opt-in; plugins that do not implement the hooks are unaffected.
• Unit tests covering state round-trip through a simulated rebuild.
• Documentation updated for plugin authors.