vmcluster: support `extraStorageNodes` for vmselect and vminsert

[zekker6]

Currently, using multi-level deployments require to define additional storage nodes via {vmselect/vminsert}.extraArgs.storageNodes.
That is error prone as changes in number of storage nodes needs to be reflected for these flags as well.

VLCluster and VTCluster resources support extraStorageNodes which allows to add an additional list of storage nodes automatically, so that it is easier change number of storage nodes without changing other parameters. It would be great for VMCluster to support the same parameter.

[AndrewChubatiuk]

VTCluster and VLCluster have this property on select level, it was done to simplify migration from single to cluster. there's no such option for data ingestion. wdyt about having targetRefs or clusterSelector instead as cluster nodes scaling require also to change listed storage nodes at extraStorageNodes

[zekker6]

Using targetRef would be even better in terms of simplicity of use. But this might be limited to having VMClusters located within a single k8s cluster which is not always the case.

[AndrewChubatiuk]

also this option wasn't added to vmcluster since vmsingle cannot at the moment be joined as a storage

[f41gh7]

As far as I understand, multi-level config must be:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: top-level
spec:
  retentionPeriod: "4"
  vmselect:
    replicaCount: 2
    extraArgs:
      storageNode: vmselect-zone-0:8421,vmselect-zone1:8421
  vminsert:
    replicaCount: 2
    extraArgs:
      storageNode: vminsert-zone-0:8420,vminsert-zone-1:8420
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: zone-0
spec:
  retentionPeriod: "4"
  replicationFactor: 2
  vmstorage:
    replicaCount: 2
  vmselect:
    clusterNativeListenPort: "8421"
    replicaCount: 2
  vminsert:
    clusterNativeListenPort: "8420"
    replicaCount: 2
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: zone-1
spec:
  retentionPeriod: "4"
  replicationFactor: 2
  vmstorage:
    replicaCount: 2
  vmselect:
    clusterNativeListenPort: "8421"
    replicaCount: 2
  vminsert:
    clusterNativeListenPort: "8420"
    replicaCount: 2

Since, top-level doesn't have any storage nodes, it's enough to use extraArgs.

[zekker6]

The full use-case I'm working on is not really multi-level.
There is a metrics source which will produce very high cardinality data and it is needed with really low retention (cilium hubble metrics), in order to avoid bloating indexes of the main installation I've set up a separate cluster which stores these metrics and data is joined via multi-level queries in order to keep a single entry point for alerting/Grafana datasources/VMUI.