ClickHouseOperatorConfiguration watch.namespaces.include is overwritten by env WATCH_NAMESPACE in OLM installation

[echoOfNull]

Description

When ClickHouse Operator is installed via OLM, the generated CSV contains the following environment variable:

- name: WATCH_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace

As a result, the operator pod starts with:

WATCH_NAMESPACE=openshift-operators

I created a ClickHouseOperatorConfiguration resource to enable watch-all mode:

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseOperatorConfiguration
metadata:
  name: watch-all-namespaces
  namespace: openshift-operators
spec:
  watch:
    namespaces:
      include:
        - ".*"

According to the default operator configuration template, [".*"] should force watch-all mode:

https://github.com/Altinity/clickhouse-operator/blob/master/deploy/builder/templates-config/config.yaml

Observed behavior

The operator successfully discovers and loads the ClickHouseOperatorConfiguration resource:

I0615 12:13:42.047018       1 config_manager.go:150] getAllCRBasedConfigs():Looking for ClickHouseOperatorConfigurations in namespace 'openshift-operators'.
I0615 12:13:43.086423       1 config_manager.go:183] getAllCRBasedConfigs():Append ClickHouseOperatorConfigurations 'openshift-operators/watch-all-namespaces'.

The CR-based configuration contains the expected value:

watch:
  namespaces:
    include:
      - .*

The merged configuration also contains the expected value:

Unified CHOP config - with secret data fetched (but not post-processed yet)

runtime:
    configFilePath: /etc/clickhouse-operator/config.yaml
    configFolderPath: /etc/clickhouse-operator
    configCRNamespace: openshift-operators
    configCRName: watch-all-namespaces
    configCRSources:
        - namespace: openshift-operators
          name: watch-all-namespaces
    namespace: ""
watch:
    namespaces:
        include:
            - .*

However, after post-processing, the final configuration changes to:

Final CHOP config

runtime:
    configFilePath: /etc/clickhouse-operator/config.yaml
    configFolderPath: /etc/clickhouse-operator
    configCRNamespace: openshift-operators
    configCRName: watch-all-namespaces
    configCRSources:
        - namespace: openshift-operators
          name: watch-all-namespaces
    namespace: openshift-operators

watch:
    namespaces:
        include:
            - openshift-operators

As a result, ClickHouseInstallation resources created in other namespaces are not reconciled.

Investigation

The operator calls Postprocess() between the "Unified CHOP config" and "Final CHOP config" stages:

https://github.com/Altinity/clickhouse-operator/blob/release-0.27.1/pkg/chop/config_manager.go#L129

log.V(1).Info("Unified CHOP config - with secret data fetched (but not post-processed yet):")
log.V(1).Info("\n" + cm.config.String(true))

cm.Postprocess()

log.V(1).Info("Final CHOP config:")
log.V(1).Info("\n" + cm.config.String(true))

Postprocess() invokes applyEnvVarParams():

https://github.com/Altinity/clickhouse-operator/blob/release-0.27.1/pkg/apis/clickhouse.altinity.com/v1/type_configuration_chop.go#L1486

func (c *OperatorConfig) applyEnvVarParams() {
	if ns := os.Getenv(deployment.WATCH_NAMESPACE); len(ns) > 0 {
		// We have WATCH_NAMESPACE explicitly specified
		c.Watch.Namespaces.Include = types.NewStrings([]string{ns})
	}

	if nss := os.Getenv(deployment.WATCH_NAMESPACES); len(nss) > 0 {
		// We have WATCH_NAMESPACES explicitly specified
		if namespaces := c.splitNamespaces(nss); len(namespaces) > 0 {
			c.Watch.Namespaces.Include = types.NewStrings(namespaces)
		}
	}

	if nss := os.Getenv(deployment.WATCH_NAMESPACES_EXCLUDE); len(nss) > 0 {
		// We have WATCH_NAMESPACES_EXCLUDE explicitly specified
		if namespaces := c.splitNamespaces(nss); len(namespaces) > 0 {
			c.Watch.Namespaces.Exclude = types.NewStrings(namespaces)
		}
	}
}

This appears to overwrite the value previously loaded from ClickHouseOperatorConfiguration.

Expected behavior

If spec.watch.namespaces.include is explicitly configured in ClickHouseOperatorConfiguration, it should remain unchanged during post-processing.

Alternatively, if WATCH_NAMESPACE is expected to have higher priority in OLM deployments, this behavior should be documented explicitly.

Question

Is this expected behavior for OLM installations, or is this a configuration precedence issue?

Environment

• RedHat OpenShift cluster version 4.20.19
• Operator installed from OLM Community Operators
• Altinity® Kubernetes Operator for ClickHouse version 0.27.1

[echoOfNull]

Hi everyone!

Just a gentle follow-up on this issue

Could you please clarify whether it is intentional that the OLM injected WATCH_NAMESPACE takes precedence over the explicitly configured watch.namespaces.include from ClickHouseOperatorConfiguration?

From my investigation, WATCH_NAMESPACE is automatically injected by OLM from the operator's namespace and then unconditionally overrides the value configured in the CR during Postprocess(). Im not sure whether this is the intended design or an unintended interaction between OLM and the operator configuration.

If this is considered a bug rather than the expected behavior, I'd be happy to prepare a PR.

Thanks!

[alex-zaitsev]

Hi @echoOfNull, this is fixed in 0.27.2. Please expect a release in a few days