[ANSIENG-5869] Add KRaft dynamic quorum (KIP-853) greenfield support - #2686
Open
Rohan Kunwar (rkunwar-28) wants to merge 8 commits into
Open
Rohan Kunwar (rkunwar-28) wants to merge 8 commits into
Rohan Kunwar (rkunwar-28) wants to merge 8 commits into
Conversation
Add opt-in support for KRaft dynamic controller quorum on CP 8.4+ for greenfield deployments (single-region and MRC): - kraft_dynamic_quorum_enabled flag, explicit kafka_controller_bootstrap_node, and a controller.quorum.bootstrap.servers template. - Gated kraft_static_quorum / kraft_dynamic_quorum property sections for controller and broker (bootstrap.servers + auto.join instead of voters). - Format fork: bootstrap formats --standalone, others --no-initial-controllers, with a disk-state split-brain guard and an all-controllers-reported assert. - Post-start health check that fails if any controller is stuck as Observer. - Version/prereq validation (kraft_enabled, CP >= 8.4.0, valid bootstrap node). - Sample inventories for 1DC and MRC 3-3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ller_initial_voter Avoids naming collision with controller.quorum.bootstrap.servers. The variable selects the single controller that formats --standalone (the sole initial voter); it does not define the bootstrap.servers discovery list, which lists all controllers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lth check Assert FinalizedVersionLevel for kraft.version is 1 via kafka-features describe, directly confirming dynamic quorum is in use rather than only inferring it from the absence of Observer controllers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… separately) Docs, including sample inventories, are tracked in a separate workstream/PR; drop docs changes from the feature PR. Role/config behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| ### (it becomes the sole initial voter). All other controllers join as observers and are promoted | ||
| ### automatically via auto-join. Must be set to one of the kafka_controller inventory hosts when | ||
| ### kraft_dynamic_quorum_enabled is true; the run is halted by validation if it is left unset. | ||
| kafka_controller_initial_voter: "" |
There was a problem hiding this comment.
what should be the value here? is it the ip address or kraft node or something else?
… single-line var docs Remove the kraft.version and voter-join checks from the controller health_check; with auto-join these cannot fail at runtime if the code is correct, so they belong in molecule verify (kept there), not the deploy health check. Collapse the three new variable doc comments to a single ### line each so docs/doc.py (VARIABLES.md generation) parses them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y hostname Spell out that the value is the Ansible inventory hostname of a controller (one of the kafka_controller group entries), not an IP, node id or advertised hostname, and that the inventory name is used even when hostname aliasing is enabled. The initial voter is matched on inventory_hostname, so aliasing does not change what to set here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fault initial voter Replace the no --limit assert with an active probe: the bootstrap node reads every controller's data directory state via delegate_to, which is not filtered by --limit. --standalone runs only if no controller is already formatted, so a wiped bootstrap node rejoins instead of founding a second cluster. If any controller cannot be reached the run halts rather than guessing. This lets --limit be used safely and removes the need for the operator to pick the bootstrap node, so kafka_controller_initial_voter now defaults to the first kafka_controller and stays overridable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…al voter, auto-join toggle Parenthesize the probe reachability assert and the use_standalone expression for readability. Split the single format task into three explicit tasks: static, dynamic initial voter (--standalone) and dynamic observer (--no-initial-controllers). Remove the default from kafka_controller_initial_voter so it must be set when dynamic quorum is enabled (existing validation enforces it) and note it should not change after the initial deployment. Add an undocumented kafka_controller_kraft_auto_join_enabled toggle (default true) for controller.quorum.auto.join.enable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds optional support for KRaft dynamic controller quorum (KIP-853) on Confluent Platform 8.4 and later. It covers new (greenfield) clusters, both single region and multi region (MRC). Static quorum stays the default and does not change.
JIRA: ANSIENG-5869
What dynamic quorum changes
With dynamic quorum, controllers can be added or removed at runtime. Controllers use
controller.quorum.bootstrap.serversinstead of the staticcontroller.quorum.voters. One controller formats storage as the first voter, and the others join as observers that get promoted to voters automatically on CP 8.2 and later.What this PR does
kraft_dynamic_quorum_enabled(default false) andkafka_controller_initial_voterto pick the first voter.kafka_controller_quorum_bootstrap_servers, which lists all controllers.controller.quorum.bootstrap.servers(plus auto join) instead ofcontroller.quorum.voterswhen the flag is on, for both controllers and brokers.--standaloneand every other controller with--no-initial-controllers.--standaloneonly runs on a brand new cluster. If any other controller already has formatted data, the run does not create a second cluster. A check also stops partial runs (for example runs with--limit) that cannot see all controllers.kraft.versionis 1 and that no controller is stuck as an observer.kraft_enabled, CP 8.4 and later, and a validkafka_controller_initial_voter.Scope
In scope: new clusters, single region and MRC, opt-in flag, config switch, format flow, verification, validation.
Out of scope: static to dynamic migration (separate PR), runtime add or remove controller, disaster recovery.
Docs and tests
Docs and sample inventories are handled in a separate PR. Molecule tests are a follow-up PR.
🤖 Generated with Claude Code