Skip to content

kind: Loki as the log store, with the local-logs query and per-replica aggregation in the docs - #2645

Open
Davanum Srinivas (dims) wants to merge 2 commits into
agent-substrate:mainfrom
dims:kind-loki
Open

Davanum Srinivas (dims) wants to merge 2 commits into
agent-substrate:mainfrom
dims:kind-loki

Conversation

@dims

@dims Davanum Srinivas (dims) commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Adds Loki to kind clusters so actor lifecycle logs remain queryable after kubelet rotates container logs. Includes instructions for querying an actor's logs and aggregating worker counts across API server replicas without double-counting. Loki storage is temporary: logs are lost when its pod restarts.

  • Tests pass: the kind overlay renders with kustomize; no Go changes.
  • Appropriate changes to documentation are included in the PR: docs/observability.md and docs/dev/best-practices/otel-collector.md.

@dims Davanum Srinivas (dims) changed the title [WIP] kind: Loki as the log store, with the local-logs query and per-replica aggregation in the docs kind: Loki as the log store, with the local-logs query and per-replica aggregation in the docs Oct 14, 2026
@dims
Davanum Srinivas (dims) marked this pull request as ready for review October 14, 2026 14:13
@dims
Davanum Srinivas (dims) force-pushed the kind-loki branch 4 times, most recently from 6ace6f5 to 877ab32 Compare October 18, 2026 21:11
@bowei Bowei Du (bowei) added area/dev-infra area/observability kind/docs Improvements or additions to documentation kind/feature An enhancement / feature request or implementation labels Oct 19, 2026
@dims
Davanum Srinivas (dims) force-pushed the kind-loki branch 2 times, most recently from 4ef37cf to 2e71057 Compare October 20, 2026 15:28
The logs pipeline ended at the debug exporter, so lifecycle events died
with kubelet's 10 MiB stdout rotation. A single-binary Loki in otel-system
now receives them through an otlphttp exporter next to debug.
@dims
Davanum Srinivas (dims) force-pushed the kind-loki branch 2 times, most recently from 4ffa79f to 4bd95eb Compare October 21, 2026 11:31
Local-logs section with the port-forward and a query by actor name.
ate.workerpool.workers is reported by every ateapi replica, so use max by
the pool labels, not sum.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/dev-infra area/observability kind/docs Improvements or additions to documentation kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants