unable to get K8s metrics after enabling OC rules

[trs91km]

in Dashboard, I enabled SW_OTEL_RECEIVER_ENABLED_OC_RULES rules as below, but from UI I can only see K8s node metrics. k8s-cluster, k8s-service and vm metrics are not coming.

I cleared cache in browser, and also on DB I am seeing metrics only for k8s_node.

export SW_OTEL_RECEIVER_ENABLED_OC_RULES="vm,k8s-cluster,k8s-service,k8s-node"

tried updating it in application.yml as below but not working

enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"vm,k8s-cluster,k8s-service,k8s-node"}

Please advise

[wu-sheng]

Do you have VM agent installed? K8s metrics are for k8s dashboard only. Nothing more.

[trs91km]

when I enable rules for only VM, I can see VM metrics in dashboard, after configured K8s, I tried to enable both VM and K8s in OC rule and started VM metrics gone and in K8s, out of cluster, node and service, only node metrics I am getting in dashboard as well as DB. Cluster and Service are not working.

[wu-sheng]

Check your setup. There is no connection between these two.

[trs91km]

Sure I will review it.

enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"vm,k8s-cluster,k8s-service,k8s-node"}

The above format is correct? when I give like this and start OAP, its stopped with logs as below

2021-11-10 23:00:24,384 - org.apache.skywalking.oap.server.starter.OAPServerBootstrap - 57 [main] ERROR [] - null
java.lang.NullPointerException: null
at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:280) ~[client-java-13.0.0.jar:?]
at io.kubernetes.client.util.ClientBuilder.getClientBuilder(ClientBuilder.java:129) ~[client-java-13.0.0.jar:?]
at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:108) ~[client-java-13.0.0.jar:?]
at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:100) ~[client-java-13.0.0.jar:?]
at io.kubernetes.client.util.Config.defaultClient(Config.java:114) ~[client-java-13.0.0.jar:?]
at org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry.start(K8sInfoRegistry.java:75) ~[meter-analyzer-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.meter.analyzer.Analyzer.init(Analyzer.java:226) ~[meter-analyzer-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.meter.analyzer.Analyzer.build(Analyzer.java:82) ~[meter-analyzer-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.meter.analyzer.MetricConvert.lambda$new$2(MetricConvert.java:52) ~[meter-analyzer-8.8.1.jar:8.8.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
at org.apache.skywalking.oap.meter.analyzer.MetricConvert.(MetricConvert.java:58) ~[meter-analyzer-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.meter.analyzer.prometheus.PrometheusMetricConverter.(PrometheusMetricConverter.java:73) ~[meter-analyzer-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.server.receiver.otel.oc.OCMetricHandler.lambda$active$1(OCMetricHandler.java:177) ~[otel-receiver-plugin-8.8.1.jar:8.8.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
at org.apache.skywalking.oap.server.receiver.otel.oc.OCMetricHandler.active(OCMetricHandler.java:178) ~[otel-receiver-plugin-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.server.receiver.otel.OtelMetricReceiverProvider.lambda$start$1(OtelMetricReceiverProvider.java:66) ~[otel-receiver-plugin-8.8.1.jar:8.8.1]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:1.8.0_252]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_252]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_252]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:1.8.0_252]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) ~[?:1.8.0_252]
at org.apache.skywalking.oap.server.receiver.otel.OtelMetricReceiverProvider.start(OtelMetricReceiverProvider.java:66) ~[otel-receiver-plugin-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.server.library.module.BootstrapFlow.start(BootstrapFlow.java:49) ~[library-module-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:60) ~[library-module-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.server.starter.OAPServerBootstrap.start(OAPServerBootstrap.java:43) [server-starter-8.8.1.jar:8.8.1]
at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:23) [server-starter-8.8.1.jar:8.8.1]

[wankai123]

where did you deploy the OAP? OAP need access to k8s API server, once enabled k8s-service
see: #7540

[trs91km]

for OAP deploy, I got the source from git and added storage mysql and started it(not as docker and K8s), otel I deployed as K8s in my K8s cluster, I will go through this #7540, and try it.
OAP need access to k8s API server - settings will be changed for my scenario?

[wu-sheng]

OAP must be in the same k8s cluster, like @wankai123 mentioned and logs reported, OAP cluster need the API server access for metadata.

[trs91km]

I have OAP, Storage and UI installed in a host
My target K8 cluster in another hosts, I have configured OTEL collector in the same cluster to get metrics from K8s state metrics and send it to my OAP server
just to be clear OAP needs to be installed in every K8s target I want to monitor?

[wu-sheng]

I think you are asking more than what we provided. Multiple clusters monitoring is actually not the case we provided.
Usually multiple k8s clusters are commercial products' typical case. We don't have people contributed this.

[trs91km]

Thanks, I will try this setup in single cluster

[wu-sheng]

If you are just trying first time deployment, follow this, https://skywalking.apache.org/docs/skywalking-showcase/latest/readme/.

It works on local KinD or private k8s.