Continuous queries: handle late / out-of-order data (lookback window + idempotent upsert)

[xe-nvdk]

Summary

Continuous queries compute each window exactly once over [last_processed_time, now) and then advance the watermark past it, with no lookback/grace window. Data that arrives after its window has already been processed (late ticks, corrections, backfills, out-of-order events) is silently dropped from that window's aggregate. There is no way to recover it short of manually re-running.

This is acceptable for clean, in-order infra metrics, but it is a blocker for financial roll-ups (OHLC bars, VWAP, PnL/position aggregates) where late corrections and out-of-order events are normal and materially change results. Raised by a prospect evaluating Arc for financial data.

Current behavior (code)

• internal/api/continuous_query.go:464-470 — window start = previous last_processed_time; end = time.Now() (wall-clock).
• No lookback/grace: startTime is exactly the prior end (:468/:586 only set a default on first run).
• Watermark advances to endTime after each run (:504), so earlier windows are never revisited.

Proposed feature

1. Configurable lookback / grace window per CQ: each run reprocesses the last N intervals (or a duration) so recent late data is folded in. Example knob: reprocess_window / late_data_grace.
2. Idempotent output (depends on Continuous queries can produce duplicate aggregate rows (non-idempotent output) #521): reprocessing a window must REPLACE that window's prior aggregate rows, not append. This requires the idempotency fix in Continuous queries can produce duplicate aggregate rows (non-idempotent output) #521 (replace/upsert by window+tags). Without it, lookback just multiplies duplicates.
3. (Optional, later) Event-time boundary instead of wall-clock now() — see separate event-time issue.

Acceptance criteria

• A CQ with a lookback of K intervals, given a row that lands one interval late, produces an aggregate for the correct window on the next run (not dropped).
• Re-running an already-computed window overwrites its output (no duplicates), enabled by Continuous queries can produce duplicate aggregate rows (non-idempotent output) #521.
• Default behavior (no lookback configured) is unchanged for existing users.

Notes

• Strong candidate for an Enterprise feature — a paying financial customer needs it.
• Blocked on / coupled with Continuous queries can produce duplicate aggregate rows (non-idempotent output) #521 (idempotent CQ output).

[xe-nvdk]

Depends on #521 (idempotent CQ output). Longer-term: #523 (event-time watermarking).