[Feature Request] Poison-message / repeated-failure strategy for grain activations

[thnak]

Problem

Epic #128 (pillar 4) lists "poison message / repeated failure strategy" as a required failure-semantics element, and it is currently absent by design: a behavior exception is contained to the failing call (GrainActivation.RunLoopAsync logs and continues; LocalGrainCallInvoker rethrows to the caller), the activation survives, and the mailbox keeps processing.

That containment is the right default, but it means a systematically failing grain — corrupt persistent state, a bad message it re-reads every call, a dependency that always throws for its key — fails forever at full throughput. There is no dead-lettering, no failure-rate tracking, no quarantine, and no automatic recycle. Callers just see every call fail. (Akka.NET answers this with supervision directives; Orleans with activation-failure handling + deactivation.)

Proposal (design needed)

Add an opt-in repeated-failure policy at the activation level, configurable per grain type:

• Failure tracking: consecutive-failure count and/or failure rate per activation (window), maintained by the shell.
• Recycle (default action when threshold crossed): deactivate the activation so the next call rebuilds it — persistent state re-reads, IManagedActivationMemory resources rebuild. Equivalent to Akka Restart, and cheap in the engine model since behaviors are already per-call.
• Quarantine (escalation): after N recycles within a window, mark the grain identity as quarantined for a backoff period — calls fail fast with a distinct exception (GrainQuarantinedException) instead of hammering storage/dependencies.
• Dead-letter hook: IQuarkDiagnosticListener.OnActivationQuarantined(...) (+ counter quark.grain.quarantined) so operators can observe and act; optionally a pluggable sink receiving the failing envelope for offline inspection.
• All thresholds in SiloRuntimeOptions (or per-type override, cf. [Feature Request] Per-grain-type idle-collection age override #115's per-type pattern); disabled by default to preserve current semantics.

Non-goals

• Automatic retry of the failed call (delivery guarantees are [B1] Delivery-guarantee semantics + request-response dedup + one-way acks #59's domain).
• Cross-silo blacklisting (single-silo quarantine first).

Related: #128 (pillar 4), #59 (delivery guarantees), #112 (circuit breaker for storage/transport — same spirit, different layer), #123 (per-type concurrency limits). Documented as a known gap in wiki/Lifecycle-and-Failure-Semantics.md.