[ANSIENG-5923] Cherry-pick rootless support (#2602, #2638) to master - #2671
Open
Ishika Paliwal (ishikaa-p) wants to merge 154 commits into
Open
Ishika Paliwal (ishikaa-p) wants to merge 154 commits into
Ishika Paliwal (ishikaa-p) wants to merge 154 commits into
Conversation
… Next Gen Add control_center_next_gen_bundled_monitoring_enabled (default true). When false, the bundled Prometheus and Alertmanager for Control Center Next Gen are not installed or started, for use with an external Prometheus. Pure bundled tasks (config copy, prometheus overlay, web-configs, dependency file permissions, prometheus/alertmanager keystores) are gated on the toggle. Tasks that configure C3 and the bundled parts together (systemd service copy, override dir/write, service start) keep the C3 parts and skip only the bundled parts. Default true preserves existing behavior.
Rename control_center_next_gen_bundled_monitoring_enabled to control_center_next_gen_external_prometheus_enabled (default false, true = BYOP). The bundled-task gates invert accordingly; default behavior is unchanged. Skip the Alertmanager health check when the external Prometheus is enabled, and keep the Prometheus health check (now pointing at the external endpoint). Brownfield cleanup of the old bundle is a documented manual step for this release, not automated in the playbook.
…BYOP Split the C3 property groups so the bundled Alertmanager/Prometheus management properties (alertmanager url, prometheus config.file, rules.file, alertmanager config.file) render only when the bundle is deployed. When external_prometheus_enabled is true, emit confluent.controlcenter.alerts.enable=false and disable the alertmanager client. In BYOP the customer configures promotion/out-of-order/recording-rules on their own external Prometheus, so those bundled-only properties are not set. Bundled render is unchanged: combine_properties flattens enabled groups by key and the template sorts by key, so moving properties between enabled groups is byte-identical.
…ode) New validate_byop.yml, imported at the top of the C3 role's main.yml and gated on control_center_next_gen_external_prometheus_enabled, so a misconfigured external Prometheus fails fast before any install work. Enforces the supported contract: the endpoint is set, TLS is enabled, and exactly one auth mode (Basic over TLS or mutual TLS) is chosen. There is no Alertmanager-with-external check because the single cp-ansible toggle disables the bundled Prometheus and Alertmanager together.
…heus inventory Document control_center_next_gen_external_prometheus_enabled: turn its defaults comment into a ### doc comment (VARIABLES.md is generated from those) and add the entry to docs/VARIABLES.md. Add docs/sample_inventories/byop_external_prometheus.yml showing the toggle plus the control_center_next_gen_dependency_prometheus_* settings (host/port/ssl/CA), a Basic-auth-over-TLS option and a commented mTLS option, and the external-Prometheus prerequisites as header comments.
The dependency Prometheus keystore/truststore task was gated to bundled-only, so in BYOP mode C3 had no truststore for the external Prometheus and failed to start with a FileNotFoundException on the truststore. C3 needs this truststore to trust the external Prometheus over TLS, so run the task whenever Prometheus TLS is enabled, external or not.
…n BYOP C3 uses this flag to relax the bundled prometheus rules / alertmanager config-file requirement independently of alerts.enable, so it can boot against an external Prometheus. Pairs with the control-center-backend change that gates prometheusBackendEnabled() on this flag. Harmless on builds that predate the flag (unknown config is ignored).
…temp health-check debug
…t from dest paths)
…TLS queries; revert debug
When Control Center reads from an external Prometheus, the Kafka brokers and controllers push their telemetry to that same endpoint. The push uses the node's own truststore, which does not contain the external Prometheus CA, so over TLS the exporter cannot verify the endpoint (and under mTLS the handshake never completes) and no broker metrics reach it. Import the external Prometheus CA into the broker and controller truststores (JKS and, under FIPS, BCFKS) when external Prometheus over TLS is enabled, reusing the shared idp_certs import task and restarting the node so the exporter picks it up. The CA source is the existing control_center_next_gen_dependency_prometheus_provided_ca_cert_path. Test wiring: every external molecule scenario now sets the provided CA source, and each mTLS scenario rebuilds the mock Prometheus client CA as a bundle of the mock CA and the per-run cluster CA (byop_mock_trust_cluster_ca) so the mock accepts the nodes' cluster-signed client certificates on push.
Two problems prevented Control Center from reading an external Prometheus over Basic auth on a TLS endpoint (no mTLS): - The custom-certs switch keyed off the on-host client cert path, which has a non-empty default, so it was always on. Without a provided client cert the copy of the (absent) signed cert failed and Control Center configuration was left with the default localhost Prometheus URL. Gate the switch on the PROVIDED client cert path instead, which is set only for mTLS. - With custom certs off, the ssl role self-signs the Prometheus client truststore with a generated CA, so it does not trust the external endpoint. Import the external Prometheus CA into that truststore (JKS and, under FIPS, BCFKS) when reading an external Prometheus over TLS without mTLS, so Control Center trusts the endpoint's server cert on read. mTLS is unaffected: the provided-cert path still builds the client keystore and imports the provided CA into the truststore.
Three fixes for the remaining split BYOP scenarios: - playbooks/all.yml: the certificate-authority generation was gated only on each component's own listener TLS. When the cluster runs without TLS (for example SASL_PLAINTEXT) but Control Center still reads an external Prometheus over TLS, the shared self-signed CA was never generated and the Control Center Prometheus client keystore step failed. Include the external Prometheus TLS flag in the condition. - byop_verify: discover the Control Center properties file (package/rpm under /etc, archive under the confluent home) instead of hard-coding the package path, so the archive scenario reads the right file. - byop_mock_trust_cluster_ca: normalise the combined client CA bundle so the two PEM blocks never merge onto one line, which made the bundle unparseable and broke the mock Prometheus TLS handshake.
- certificate_authority.yml: the inner CA-generation and copy-back tasks carried the same component-TLS condition as the play-level include, so the CA was still skipped on a non-TLS cluster that only needs TLS for the external Prometheus. Add the external Prometheus TLS flag to both. - byop_verify: skip the broker/controller push assertions under mTLS. The mock Prometheus requires a client cert signed by its own CA, so it rejects the node's cluster-signed client cert in this harness; the node-push path is covered by the Basic-auth-over-TLS scenarios with the same product code. - Drop the mock cluster-CA trust step from the mTLS scenarios: modifying the running mock's client CA broke its TLS handshake. mTLS coverage validates the Control Center read path; node push is covered over Basic auth.
The sample set the on-host dependency Prometheus cert paths, which have defaults and do not trigger the CA import into the Control Center and node truststores. Point the sample at the provided_* variables instead (the external Prometheus CA, and the mTLS client cert/key, staged on the Ansible control node), matching how the role consumes them.
…he mock host from upgrade - The Control Center Prometheus client truststore is a single PKCS12 store with no BCFKS variant, even under FIPS, so the BYOP CA import must not attempt a BCFKS import. This unblocks the FIPS Basic-auth-over-TLS read. - byop-upgrade-migrate: exclude the mock Prometheus host from the all-hosts version-clear step, which cannot run against the local mock container.
The teardown removed /usr/lib/systemd/system/{prometheus,alertmanager}.service,
which the Control Center Next Gen package owns and the role reinstalls the
bundled services from. Removing them broke any later bundled run (and the
round-trip scenario). Stopping and disabling the services and removing the
overrides, config, and data is enough to take bundled monitoring out of
service, so leave the package unit files in place.
…brownfield scenarios
…ring and harden verify
…seline The molecule tree carried over stale versions of six non-BYOP scenario files (archive-plain-debian, oauth-plain-*). Restore them to the 8.4.x baseline so this branch's delta contains only BYOP changes.
Replace the inconsistent 'length > 1' / '<= 1' (keystore) and 'length > 0' (CA import) idioms with a single canonical 'is defined and (| trim | length) > 0' across the Control Center Next Gen Prometheus keystore branches and the CA-import gates in the control_center_next_gen, kafka_broker, and kafka_controller roles. Behavior is unchanged for all real inputs (unset, empty, and normal cert/CA paths); the idiom only differs for degenerate whitespace-only or single-character values, which no scenario or sample inventory uses.
# Conflicts: # molecule/oauth-rbac-mtls-provided-ubuntu/verify.yml
"Install packaging module" (python3-packaging, RHEL10-only) came in via upstream PR confluentinc#2395 (RHEL 10 support) after this cherry-pick effort's rootless sweep had already passed through this branch's ancestry - same class of gap as the pyyaml/python3-debian fixes: become: true unconditionally, no rootless guard, fails "sudo: a password is required" on RHEL10 rootless hosts.
…ain-debian Upstream reintroduced usm-agent1/ccloud-mock-service testing for this scenario starting at 8.1.x (same as it did for kerberos-rhel, which already got this treatment) - archive-plain-debian's molecule.yml host list was already correctly commented out, but its verify.yml still imported verify_usm_agent.yml unconditionally, which fatals checking confluent.telemetry.exporter._usm.* properties that no longer render without a usm_agent host in groups.
Copilot started reviewing on behalf of
Ishika Paliwal (ishikaa-p)
September 9, 2026 16:26
View session
There was a problem hiding this comment.
🟡 Changes recommended
Several rootless changes introduce concrete functional breakages (unsafe /etc ownership changes in Kerberos tasks, incorrect EnvironmentFile emission for values with spaces, and missing ansible_user_uid gathering in plays with gather_facts: false).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Cherry-picks the “rootless” (non-root) deployment support onto master, adding a systemd --user lifecycle, a one-time privileged bootstrap playbook, and extensive role/molecule/CI updates to ensure Confluent Platform can be installed/configured/run entirely as an unprivileged deploy user.
Changes:
- Adds rootless deployment documentation, sample inventories, and a privileged
rootless_bootstrapplaybook to set up the deploy user, linger, and optional prerequisites. - Updates core roles to support rootless paths/users, generate
systemd --userunits/env files, and start/restart services via user-scoped systemd while skipping root-only tasks. - Extends molecule coverage and adds Semaphore sanity checks to prevent rootless regressions (privileged-tag conflicts, unguarded
become: true).
File summaries
| File | Description |
|---|---|
| ROOTLESS_DEPLOYMENT_STEPS.md | New end-to-end rootless deployment guide and operational checks |
| roles/variables/vars/main.yml | Adds rootless path normalization + rootless-aware default users/dirs |
| roles/schema_registry/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/schema_registry/tasks/main.yml | Skip root-only steps under rootless; add rootless lifecycle/start |
| roles/schema_registry/defaults/main.yml | Adds JAVA_HOME env override for rootless/systemd lifecycle |
| roles/ksql/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/ksql/tasks/main.yml | Skip root-only steps under rootless; add rootless lifecycle/start |
| roles/ksql/defaults/main.yml | Adds JAVA_HOME env override for rootless/systemd lifecycle |
| roles/kerberos/tasks/main.yml | Adjusts ownership handling for Kerberos client config/keytabs |
| roles/kafka_rest/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/kafka_rest/tasks/main.yml | Skip root-only steps under rootless; add rootless lifecycle/start |
| roles/kafka_rest/defaults/main.yml | Adds JAVA_HOME env override for rootless/systemd lifecycle |
| roles/kafka_controller/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/kafka_controller/tasks/rbac.yml | Skips root-only RBAC SSL directory creation under rootless |
| roles/kafka_controller/tasks/main.yml | Rootless guards + rootless lifecycle/start wiring |
| roles/kafka_controller/tasks/get_meta_properties.yml | Passes rootless component name for master-key recovery |
| roles/kafka_controller/defaults/main.yml | Adds JAVA_HOME env override for rootless/systemd lifecycle |
| roles/kafka_connect/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/kafka_connect/tasks/main.yml | Rootless guards + rootless lifecycle/start wiring |
| roles/kafka_connect/tasks/connect_plugins.yml | Ensures plugin dirs include confluent-hub dirs under rootless |
| roles/kafka_connect/defaults/main.yml | Adds JAVA_HOME env override for rootless/systemd lifecycle |
| roles/kafka_connect_replicator/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/kafka_connect_replicator/tasks/rbac_replicator.yml | Rootless-friendly SSL dir path/ownership updates |
| roles/kafka_connect_replicator/tasks/rbac_replicator_producer.yml | Skip root-only SSL dir under rootless |
| roles/kafka_connect_replicator/tasks/rbac_replicator_monitoring.yml | Skip root-only SSL dir under rootless |
| roles/kafka_connect_replicator/tasks/rbac_replicator_consumer.yml | Rootless-friendly SSL dir path/ownership updates |
| roles/kafka_connect_replicator/tasks/main.yml | Rootless guards + rootless lifecycle/start wiring |
| roles/kafka_broker/tasks/restart_and_wait.yml | Rootless-aware restart via systemd --user |
| roles/kafka_broker/tasks/rbac.yml | Skips root-only RBAC SSL directory creation under rootless |
| roles/kafka_broker/tasks/main.yml | Rootless guards + rootless lifecycle/start wiring |
| roles/kafka_broker/tasks/health_check.yml | Passes rootless component name for master-key recovery |
| roles/kafka_broker/tasks/get_meta_properties.yml | Passes rootless component name for master-key recovery |
| roles/kafka_broker/defaults/main.yml | Adds JAVA_HOME env override for rootless/systemd lifecycle |
| roles/control_center_next_gen/tasks/restart_and_wait.yml | Rootless-aware restarts for 3 user units (c3ng + deps) |
| roles/control_center_next_gen/tasks/main.yml | Rootless guards + lifecycle/start for c3ng + deps |
| roles/control_center_next_gen/tasks/health_check.yml | Skips logrotate validations under rootless |
| roles/common/templates/rootless.service.j2 | New template for user-scoped rootless systemd unit |
| roles/common/templates/rootless_component.env.j2 | New template for rootless EnvironmentFile |
| roles/common/tasks/validate_package_availability.yml | Skip package availability checks under rootless |
| roles/common/tasks/ubuntu.yml | Adds rootless guards for privileged/package operations |
| roles/common/tasks/secrets_protection.yml | Ensures SSL dir exists even when rootless + secrets protection |
| roles/common/tasks/rootless_start_service.yml | New helper to enable/start user unit + wait on port |
| roles/common/tasks/rootless_service_state.yml | New helper to query systemd --user service state |
| roles/common/tasks/rootless_prereqs.yml | Rootless PyYAML bootstrap via user-site pip |
| roles/common/tasks/rootless_lifecycle.yml | New helper to generate env/unit + daemon-reload in user scope |
| roles/common/tasks/redhat.yml | Adds rootless guards for privileged/package operations |
| roles/common/tasks/rbac_setup.yml | Ensures SSL dir exists even for RBAC without SSL under rootless |
| roles/common/tasks/main.yml | Adds rootless assertions, prereqs, writability checks, sudo probe |
| roles/common/tasks/idp_certs.yml | Skips privileged IdP truststore steps under rootless |
| roles/common/tasks/get_masterkey.yml | Reads master key from rootless env file vs root override.conf |
| roles/common/tasks/fips-redhat.yml | Skips FIPS OS-level steps under rootless + adds tags/guards |
| roles/common/tasks/debian.yml | Adds rootless guards for privileged/package operations |
| roles/common/tasks/custom_java_install.yml | Skips privileged alternatives updates under rootless |
| roles/common/tasks/config_validations.yml | Fails fast for FIPS+rootless incompatibility |
| roles/common/tasks/collect_support_bundle.yml | Makes become conditional under rootless when collecting files |
| playbooks/tasks/certificate_authority.yml | Skips package installs under rootless; adjusts tagging |
| playbooks/rootless_bootstrap.yml | New privileged bootstrap playbook for rootless installs |
| playbooks/ksql.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| playbooks/kafka_rest.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| playbooks/kafka_controller.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| playbooks/kafka_connect.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| playbooks/kafka_connect_replicator.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| playbooks/kafka_broker.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| playbooks/control_center_next_gen.yml | Uses rootless service-state helper when rootless lifecycle enabled |
| molecule/rootless_lifecycle_verify.yml | New shared molecule verification for rootless lifecycle proof |
| molecule/oauth-rbac-mtls-provided-ubuntu/verify.yml | Rootless lifecycle + secrets protection verification updates |
| molecule/oauth-rbac-mtls-provided-ubuntu/prepare.yml | Ensures early prepare steps run as root before deploy user exists |
| molecule/oauth-rbac-mtls-provided-ubuntu/molecule.yml | Enables rootless; disables USM agent; secrets-protection lists |
| molecule/kerberos-rhel/verify.yml | Rootless lifecycle verification + rootless path fixes for checks |
| molecule/kerberos-rhel/prepare.yml | Ensures early prepare steps run as root before deploy user exists |
| molecule/kerberos-rhel/molecule.yml | Enables rootless; updates kerberos paths/plugin.path; disables USM |
| molecule/collections_converge.yml | Two-phase molecule converge: bootstrap as root, deploy as user |
| molecule/archive-plain-debian/verify.yml | Rootless lifecycle verification + restart-propagation regression test |
| molecule/archive-plain-debian/side_effect.yml | Side-effect play to force unit override change and validate restart |
| molecule/archive-plain-debian/prepare.yml | Ensures early prepare steps run as root before deploy user exists |
| molecule/archive-plain-debian/molecule.yml | Enables rootless; adds side_effect play; disables USM agent |
| docs/VARIABLES.md | Documents rootless variables and rootless-aware defaults |
| docs/sample_inventories/non_root_deployment.yml | Updates sample inventory to use rootless “3 knobs” model |
| docs/sample_inventories/non_root_deployment_rbac_oauth.yml | New sample inventory: rootless + RBAC LDAP + OIDC SSO |
| docs/sample_inventories/non_root_deployment_rbac_ldap.yml | New sample inventory: rootless + RBAC LDAP + mTLS |
| .semaphore/tests/test_rootless_privileged_tag_check.py | Unit tests for privileged-tag conflict scanner |
| .semaphore/tests/test_rootless_become_check.py | Unit tests for unguarded-become scanner |
| .semaphore/sanity_tests.sh | Wires new rootless sanity checks into CI |
| .semaphore/rootless_privileged_tag_check.py | New scanner to prevent rootless-skipped tags on config/rootless tasks |
| .semaphore/rootless_become_check.py | New scanner to prevent literal become: true without rootless guard |
| .gitignore | Ignores .ansible/ |
Review details
- Files reviewed: 86/87 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
3
to
+8
| ansible.builtin.file: | ||
| path: "{{ kerberos_client_config_file_dest | dirname }}" | ||
| state: directory | ||
| mode: '755' | ||
| owner: "{{ kerberos_user }}" | ||
| group: "{{ kerberos_group }}" |
Comment on lines
+41
to
+47
| - name: "Reload the per-user systemd manager for {{ rootless_component_name }}" | ||
| ansible.builtin.systemd: | ||
| daemon_reload: true | ||
| scope: user | ||
| environment: | ||
| XDG_RUNTIME_DIR: "/run/user/{{ ansible_user_uid }}" | ||
| tags: [filesystem] |
Comment on lines
+5
to
+13
| - name: "Enable + start cp-{{ rootless_component_name }} (rootless systemd --user)" | ||
| ansible.builtin.systemd: | ||
| name: "cp-{{ rootless_component_name }}.service" | ||
| scope: user | ||
| enabled: true | ||
| state: started | ||
| environment: | ||
| XDG_RUNTIME_DIR: "/run/user/{{ ansible_user_uid }}" | ||
|
|
Comment on lines
+4
to
+8
| {% for key, value in (rootless_service_environment_overrides | default({}, true)).items() %} | ||
| {% if value %} | ||
| {{ key }}={{ value }} | ||
| {% endif %} | ||
| {% endfor %} |
Comment on lines
+11
to
+14
| # Confirm no trailing / on rootless_deployment_path - used (instead of the raw rootless_deployment_path) by every | ||
| # rootless_deployment_path-derived path below and in defaults/main.yml, so a trailing slash in the | ||
| # inventory (e.g. rootless_deployment_path: /cp-data/) can't produce double-slash paths. | ||
| rootless_deployment_path_final: "{{ rootless_deployment_path | regex_replace('\\/$', '') }}" |
Add io.confluent.system.(?!.*delta).* to the _c3.metrics.include allowlist for both the Kafka broker and KRaft controller metrics_reporter_for_control_center_next_gen config so that system storage/CPU/memory metrics are reported to Control Center Next Gen in ansible-based deployments, matching confluent-operator PR #5298. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-skip-bundled-monitoring-84x [SETU-3482] BYOP: skip bundled monitoring, point C3 at an external Prometheus (8.4.x)
…em-metrics-ansible MMA-19368: Add io.confluent.system metrics to C3 NG telemetry allowlist
…m BYOP/SETU-3482 conflicts) # Conflicts: # molecule/kerberos-rhel/molecule.yml # molecule/oauth-rbac-mtls-provided-ubuntu/molecule.yml # molecule/oauth-rbac-mtls-provided-ubuntu/verify.yml # roles/control_center_next_gen/tasks/health_check.yml # roles/control_center_next_gen/tasks/main.yml # roles/control_center_next_gen/tasks/restart_and_wait.yml
# Conflicts: # docs/VARIABLES.md
Rohan Kunwar (rkunwar-28)
approved these changes
Sep 10, 2026
Rohan Kunwar (rkunwar-28)
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merge-forward of the rootless support carried by #2670 (
8.4.x) ontomaster, the final branch in this chain, plus one additional conflict from upstream's own subsequent merge of8.4.xintomaster:docs/VARIABLES.mdhad a version-bump conflict (confluent_usm_agent_package_versiondefault1.0.0vs1.2.1) - unrelated to rootless, took upstream's newer value.control_center_next_gen_dependency_prometheus_ca_cert_pathrootless fix - see [ANSIENG-5923] Cherry-pick rootless support (#2602, #2638) to 8.4.x #2670) carried through cleanly from the8.4.xmerge.This is the last of the chain:
7.7.x(#2662) →7.8.x(#2664) →7.9.x(#2665) →8.0.x(#2666) →8.1.x(#2667) →8.2.x(#2668) →8.3.x(#2669) →8.4.x(#2670) →master.Test plan
Verified via Semaphore CI (
cp-ansible-toolson-demand task),SCENARIO=ALL, full molecule pytest matrix (KRaft-only), against the final commit:cb5b9306— 1 failure (rbac-mds-mtls-custom-rhel-fips), matches the pre-existingrbac-mds-*/myid.j2group_by-accumulation flake class already documented on every earlier branch in this chain.An earlier run against an intermediate commit surfaced
kerberos-rhelandarchive-plain-debianas real failures (root-caused via live Semaphore panes) — both confirmed resolved in this run.🤖 Generated with Claude Code