Add option to parse protobuf oneof as Variant#3
Open
filimonov wants to merge 1 commit into
Open
Conversation
filimonov
pushed a commit
that referenced
this pull request
Sep 10, 2025
filimonov
pushed a commit
that referenced
this pull request
Sep 10, 2025
Backport ClickHouse#82422 to 25.6: Read AWS ECS token from file, attempt #3
filimonov
pushed a commit
that referenced
this pull request
Jan 20, 2026
…and more predictable timings, #3
filimonov
added a commit
that referenced
this pull request
Jun 5, 2026
… + Tier 2 G1/G3) Resolves the B69 attended-review gate. Framing correction (verified): the sweep's current safety net is a generation-BLIND full reachability re-validate scan, so #1/#2 are leak/log-drift today, NOT data-loss — they become data-loss only when Tier 2 (#4) removes that scan. Hence Tier 1 (generation accounting #1/ClickHouse#6, fail-closed sticky session #2, race ClickHouse#5, contracts ClickHouse#7) MUST precede Tier 2 (lock-free GcLogWriter I/O #3 + sealed-tombstone index #4). New lockless-path oracles are the gate. #2 retain-session = sticky-exempt-from-reaping + bounded re-log-retry (not reconciliation). ClickHouse#6 = settled generation in the .meta bundle sidecar. Rest -> backlog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 5, 2026
… 1-6) 18 tasks across 7 phases, grounded in verbatim current code. Tier 1 first (#1 splitDeltaByShard generations, ClickHouse#6 sidecar drop-keying, #2 fail-closed sticky session + bounded re-log, ClickHouse#5 pin-snapshot race, ClickHouse#7 *Locked rename) then Tier 2 (#3 lock-free GcLogWriter I/O + fold-ins, #4 gc/sealed index). Scan B (markReachableBlobs delete gate) deliberately untouched — its replacement is B78. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 5, 2026
…on the warm path) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 5, 2026
Take mtx only to move fragments in/out of the buffer (drainBufferLocked); do the epoch read and the object PUT lock-free (writePending). Concurrent commits no longer serialize on the per-pool writer across multiple ~300ms round-trips. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 5, 2026
…tSession; oracle 3 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 5, 2026
…); B78 still open Tier 1 (#1 splitDeltaByShard generations, ClickHouse#6 sidecar drop-keying, #2 fail-closed sticky session, ClickHouse#5 pin-snapshot race, ClickHouse#7 *Locked rename) + Tier 2 (#3 lock-free GcLogWriter I/O + fold-ins, #4 gc/sealed index) landed with oracles 1-6 green, 156 ContentAddressed gtests + CA stateless smoke + non-CA regression passing. Scan B (the markReachableBlobs delete gate) untouched — B78 (replace it with the sources+compaction authoritative gate) remains the deferred, data-loss-critical follow-up with its own attended-review gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 12, 2026
Merged the branch onto current upstream/master (e0bc679), built with ASan, and ran the CA-S3 suite with the mitigation reverted (async upload live). ASan fired repeatedly — the merge did NOT fix it. Definitive attribution: heap-buffer-overflow READ of a MemoryTracker* in MemoryTracker::setParent (parent-chain walk) <- ThreadStatus:: attachToGroupImpl <- ThreadGroupSwitcher <- a threadPoolCallbackRunnerUnsafe detached task on a ThreadPool worker. The attached ThreadGroup's MemoryTracker parent chain references a freed query-lifetime tracker (nearest freed chunk: a QueryStatusInfo released in QueryMetricLog:: collectMetric). err.log corroborates the detached task is the S3 upload (21x WriteBufferFromS3::finalizeImpl). Same bug as the original 0x1413 crash, via the adjacent performance_counters.setParent instead of memory_tracker.setParent. Confirms hypothesis #3: a generic upstream S3-async-upload UAF (threadPoolCallbackRunnerUnsafe is Unsafe by contract), exposed (not caused) by CA's S3-write-heavy workload. Fix is high-blast-radius generic IO and is left as a decision (generic fix vs retain CA mitigation). Reports saved: tmp/asan_report_b90.txt, tmp/asan_errlog_b90_full.txt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
filimonov
added a commit
that referenced
this pull request
Jun 12, 2026
…econcile rebuild; drop Keeper epoch cache) External review (Codex) found real edge-accounting holes. Fixes: #1/#2 Stale root positives. The §4.1 orderings (+-before-setRef, removeRef-before--) already bias a crash to over-count (leak), never under-count (loss). The real bug was that they could leak FOREVER because reconcile (zero-weight markers) could not subtract a stale +. Fix: reconcile is now an AUTHORITATIVE REBUILD (§4.5) — it recomputes in-degree from real refs/ reachability + the physical LIST, with a high-watermark (snap/<E> authoritative-through-E; discard logs ≤ E), so a stale + recomputes to its true value and dies. Keep logging root edges (so the routine fold needs no refs/ LIST — answers the "full traversal each round" concern); reconcile is the periodic authoritative truth-maker. #3 Generation ABA when reclaim lags. Added a durable per-hash floor (floors/<H> = 1+max-condemned-gen); reuse iff g ≥ floor(H) else resurrect to floor. Replaces the bounded recent-condemned window as the reuse authority. #4 Closed-epoch reappend. Concrete protocol: leader writes a durable seal (gc/sealed/<e>) at close; a writer whose append target is sealed re-syncs and reappends to the open epoch. The fold processes only sealed epochs. ClickHouse#5 gc/condemned is now a FULL reclaim record (hash, gen, kind, child-edges, fold-epoch); R4 cascade reads children from it (crash-safe successor). #5b Bounds: per-writer caps ≤3 (tree + 2 children) so multi-child commit is reachable in the model. Epoch cache: DROPPED from Keeper (v1) per three concurring reviews. The epoch lives only in S3 gc/epoch; writers read it with a short process-memory TTL (lag-only = safe; the seal is the event-invalidation). Removes the fragile "Keeper never ahead of S3" invariant and the ghost-epoch recovery hazard entirely. Keeper now holds ONLY leader election + per-writer leases. Threaded through layout, writer/GC/recovery/reconcile protocols, invariants, hinges, failure table, decisions (now D1–D6), verification scope, §11 open items, and the formal appendix (floors/sealed variables, floor-based reuse, seal+reappend, authoritative-rebuild Reconcile, updated bounds + scenarios). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
input_format_protobuf_oneof_as_variantsettingallow_suspicious_variant_typescheck if this setting is enabledTesting
clickhouse-test --help(fails: command not found)https://chatgpt.com/codex/tasks/task_b_68540a36ca4883238e9c5c453679ba2b