feat(telemetry): core-node observability foundation — OTLP logs + traces + metrics#1317
Merged
Conversation
Adds vendor-neutral OpenTelemetry (OTLP/HTTP) log forwarding for V10 nodes,
built on the existing Logger sink seam rather than rebuilding V8's Graylog.
- core: extract canonical `LogRecord` type; add `log-redaction.ts`
(`createLogRedactor`) — at-source, key-name-driven scrubbing of wallet
keys / mnemonics / tokens / JWTs, conservatively leaving public 0x hashes
and Merkle roots intact.
- node-ui: `OtlpLogWorker` — hand-rolled OTLP/HTTP logs exporter (the JS Logs
SDK is still "Development"); bounded drop-oldest buffer, batched async flush,
exponential backoff honoring Retry-After, outbound-only, never blocks the node.
- cli: fan out the daemon `Logger.setSink` to local SQLite (full fidelity) +
a single redacted copy to syslog and/or OTLP; add `config.telemetry.logs`
({exporter,endpoint,token,level,redact,bufferMaxEntries}) + per-network
`otlpLogs` endpoint; boot + runtime exporter dispatch. Forwarding stays OFF
by default; local logging is always on.
- tools/log-collection-poc: reference OTel Collector -> Loki -> Grafana stack
+ sample sender driving the real redactor->exporter pipeline.
Verified: 10 redaction tests + 7 OTLP-worker tests (real local HTTP server,
incl. 503-retry and 400-drop) + full core suite (1114) green; live end-to-end
through the stack confirmed logs land in Loki, dkg.* correlation metadata is
queryable, and a wallet private key + mnemonic arrive [REDACTED].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the hosted-fleet "pick a node, see its logs" workflow first-class: - node identity is now emitted as OTel `service.instance.id` (= node name) and `deployment.environment` (= network), which Loki promotes to index labels (`service_instance_id`, `deployment_environment`) with no Loki/ collector config change — so they can drive Grafana dashboard variables. dkg.* stay as structured metadata for correlation. - add tools/log-collection-poc/grafana-dashboard-dkg-node-logs.json: a Node + Environment selector, regex filter, log-volume-by-level panel, and the built-in time range — importable into any Grafana with a Loki source. - sample sender takes [nodeName] [network] so multiple named nodes can be simulated. Verified live: two named nodes (testnet-core-01 / mainnet-core-01) ingested; Loki label_values(service_instance_id) returns both; the dashboard's Node dropdown filters to a single node's logs over a chosen time range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: retry_exhausted
Wires V10 OTLP node logs into a pre-existing Loki 2.5.0 (which lacks native OTLP), verified end-to-end locally against Loki 2.5.0: - tools/log-collection-poc/production/config.alloy — Grafana Alloy: OTLP/HTTP receiver → loki.write to the existing local Loki, promoting node identity (service.name, service.instance.id, deployment.environment, dkg.node_role) to index labels for per-node Grafana selection. - docker-compose.alloy.yml — run Alloy on the Loki host (network_mode host). - docker-compose.sim.yml — local Loki-2.5.0 + Alloy rig that validated the whole path (per-node labels + redaction survive the bridge). - grafana-dashboard-dkg-node-logs.json — Node + Level selectors, regex filter, json|line_format clean display, volume-by-level; queries validated on 2.5.0. - node-config.example.json + RUNBOOK.md — exact node config + deploy steps (Alloy on host, Cloudflare-tunnel ingest hostname + bearer token, point nodes). - sample sender gains OTLP_TOKEN env for authed pushes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: retry_exhausted
Turnkey, copy-paste artifacts so the access-gated deploy is zero-guesswork: - cloudflared-config.example.yml — ingest hostname → Alloy (no Access policy). - alloy.systemd.service — non-Docker alternative for running Alloy on the host. - smoke-test.sh — post-deploy check (push one OTLP log + token → confirm it reaches Loki; verify tokenless push is rejected). - MANAGER-HANDOFF.md — DONE / VERIFIED (incl. real-daemon proof) / BLOCKED-on- access summary with the exact 3 remaining steps. Verified the REAL dkg daemon (not a harness) ships logs end-to-end: booted an edge node with telemetry.logs.exporter=otlp → 30 live operational log lines landed in a real Loki 2.5.0 via Alloy, queryable by service_instance_id. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ication note Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":10046239}
- grafana-dashboard-dkg-fleet-logs.json — fleet overview (active-node count,
log volume per node, errors per node, recent fleet errors). Imported into
polaris and verified ('Active nodes = 2' against test data).
- OPERATOR-GUIDE.md — self-serve enablement for any node operator (their own
backend), complementing the OriginTrail-fleet RUNBOOK.
- example-alerts.md — node-quiet, error-spike, and redaction-leak alert rules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These evm-module/deployments/localhost/* files were untracked build artifacts in the worktree and got swept into an earlier 'git add -A'. They are unrelated to log collection and not tracked on main. Removing from the PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ocalhost/ dir should have been dropped) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: retry_exhausted
…correlation, config - core/src/telemetry-api.ts: call-site facade (getTracer, withSpan, linkedSpan, currentTraceIds, activeSpanContext) + low-cardinality metric instruments (getMetrics/rebuildMetrics). Global-API model → no-op when no provider. - node-ui/src/telemetry.ts: real initTelemetry (NodeTracerProvider + MeterProvider + OTLP/proto exporters + shared Resource + W3C/AsyncLocalStorage via register()); shutdownTelemetry; idempotent; no-op when disabled/no endpoint. (Logs stay on OtlpLogWorker — OTel Logs SDK still Development.) - core/src/logger.ts: attach active trace_id/span_id to every LogRecord; OtlpLogWorker emits them as top-level OTLP fields. Redaction/buffer untouched. - cli config: telemetry.traces/metrics blocks; lifecycle registers the SDK at boot (env OTEL_EXPORTER_OTLP_* > config; no TBD prod defaults) + flushes on shutdown. Deps: @opentelemetry/api (core); sdk-trace-node, sdk-metrics, exporter-*-otlp-proto, resources, semantic-conventions (node-ui). Full graph builds; 21 core tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":10094535}
…ain + tests Spans (explicit boundaries, duration/status/exception): agent.publish, agent.publish_from_swm, sync.request, sync.response (inbound root), publisher.ack_collect, publisher.ack_peer_request, publisher.ack_verify_identity, publisher.storage_ack_handler (inbound root), chain.eth_call (×3 seams), chain.eth_getLogs, chain.tx_send/tx_submit/tx_wait. Metrics (low-cardinality, bounded attrs only): dkg.publish.total/.duration, dkg.ack.quorum/peer/verify/handler.total, dkg.chain.rpc.total/.duration/.failover.total, dkg.sync.request/response.total. No peer_id/cg_id/tx_hash/op_id as labels. Tests (node-ui/test/telemetry.test.ts, 9): disabled=no-op, span ERROR status + exception recording, metric label-cardinality allowlist, log↔trace correlation. Full graph builds; publisher ACK/host-mode suite (52) green; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":10163598}
…ector for verification - otel-collector-config.yaml: add traces + metrics pipelines (debug exporter) alongside logs→Loki, so a node exports all three signals to one collector. - mock-otlp-collector.mjs: zero-dep local OTLP receiver for verifying export without Docker. Verified live: a real dkg daemon (telemetry on, endpoints → :4318) exported logs+traces+metrics; captured payloads contained all 11 span names (agent.publish, publisher.ack_collect/peer/verify/handler, chain.eth_call/ getLogs/tx_send, sync.request/response), all 8 metric instruments, the resource identity (service.instance.id/dkg.network/dkg.node.role/dkg.peer_id), and dkg.operation_id on logs. enabled=false → no-op (unit-tested). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
Review addressed — moved from "OTLP log PoC" to an observability foundation (logs + traces + metrics)Commits: Architecture: call sites use the ambient 1. Logs ✅
2. Traces ✅ (explicit spans, not blanket wrapping)
3. Metrics ✅ (low-cardinality)
4. Transport/config ✅
5. Backward compatibility ✅
Acceptance criteria ✅
Open item (ops, not code): real collector hostnames for testnet/mainnet — until provided, traces/metrics stay inert on those networks by design (no guessed prod URL). |
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":10165655}
Bojan131
changed the title
…tric Closes the last review 'minimum spans' / metrics item: ProtocolRouter.send is now wrapped (span 'protocol_router.send' + dkg.protocol_router.send.total/ .duration, protocol_id label). send body moved to private sendInner; the public send is a thin span+metric wrapper (no-op when telemetry off). protocol-router suite (94) + telemetry suite (9) green; full graph builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
Follow-up (): closed the last item from the review's Minimum spans + Metrics lists — |
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":10168322}
- 🔴 redaction: auth-scheme credentials ('authorization: Bearer <token>') now
redact the token too, not just the scheme word (log-redaction.ts) + test.
- 🟡 daemon telemetry routing now unit-tested: extracted resolveOtelSignals /
resolveLogExporterMode (cli/src/telemetry-config.ts) used by lifecycle.ts;
test covers env precedence, no-TBD-default, per-signal gating, exporter mode.
- 🟡 OtlpLogWorker overflow test now proves drop-OLDEST (survivors = newest 3),
not merely capped.
- 🟡 RUNBOOK: corrected Alloy bind address to 0.0.0.0:4318 (matches config.alloy)
+ firewall note.
- 🔵 OtlpLogWorker.minLevel narrowed to a LogLevel union.
- 🔵 droppedNonRetryable now surfaced in the diagnostic message (no dead state).
Full graph builds; redaction (11), node-ui worker+telemetry (16), cli routing (7) green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
otReviewAgent
left a comment
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: Error: turn/start: turn/start failed: Input exceeds the maximum length of 1048576 characters. (code -32602), data: {"input_error_code":"input_too_large","max_chars":1048576,"actual_chars":10174996}
…llection # Conflicts: # packages/agent/src/dkg-agent-publish.ts
…llection # Conflicts: # packages/chain/src/evm-adapter-base.ts
…strumentation in tests (#1317 review) Addresses the red otReviewAgent findings on PR #1317: 🔴 Disable/enable telemetry now controls ALL signals, not just logs. Previously `telemetry.enabled` and the runtime settings toggle only started/stopped the log exporters; the OTel trace/metric SDK was initialized once at boot and torn down only at shutdown. So disabling telemetry kept exporting traces/metrics, enabling from a boot-disabled state never started them, and a log-exporter start failure flipped the master `enabled=false` (killing metrics/traces too). - lifecycle.ts: extract startOtelSdk(); startTelemetry() now starts OTel SDK + log exporter; stopTelemetry() is async and awaits shutdownTelemetry(); boot + settings-toggle both go through the same path; a failed log shipper no longer disables the master gate or the other signals. - telemetry.ts: shutdownTelemetry() clears the OTel API globals (trace.disable()/metrics.disable() + rebuildMetrics()) so a live re-enable can register fresh providers instead of silently no-opping. 🔴 Metric test now exercises a REAL instrumented call site. - chain/test/chain-rpc-telemetry.unit.test.ts (new): drives the actual contractReadWithFailover seam under an in-memory meter and asserts dkg.chain.rpc.total carries ONLY {rpc_method,outcome,retryable,chain_id} — mutation-verified to fail if a high-cardinality label (e.g. rpc_url) is added. Adds @opentelemetry/api + sdk-metrics as chain devDeps. - node-ui/test/telemetry.test.ts: new initTelemetry() bootstrap test that exports a span + metric to a local OTLP collector (covers the real OTLP/proto exporter + register()/rebuildMetrics() path). 🔴 OTLP retry test no longer counts the failed 503 as delivery evidence. - otlp-log-worker.test.ts: the server now records response status; the 503→200 case asserts the record arrived on the SUCCESSFUL request. 🔴 redaction of `authorization: Bearer <token>` — verified already fixed (auth-scheme alternation) and guarded by an existing regression test; no code change needed. All affected suites green: node-ui 1475, chain 711, core redaction 11, cli telemetry-config 7. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ coverage)
Blocking (P1):
- log-redaction: an UNQUOTED multi-word mnemonic/seed phrase leaked all but its
first word (bare-value matcher stopped at whitespace). Added a dedicated
phrase-key matcher (mnemonic/seedPhrase/seed_phrase) that consumes up to 24
alpha words; phrase keys are excluded from the single-token matcher so a value
is never processed twice. +regression tests (unquoted phrase, no over-redaction
past a delimiter, no false-positive on a bare "mnemonic" word).
- lifecycle: runtime telemetry-enable now ROLLS BACK. If the log exporter fails
after the OTel SDK started, setTelemetryEnabled(true) awaits stopTelemetry()
before returning the error and persists nothing — never leaves traces/metrics
exporting while config/UI report disabled.
- node-ui telemetry API back-compat: restored recordGauge/setOperationSpan/
TelemetryConfig exports; initTelemetry now also accepts the legacy flat
{ metricsEndpoint, serviceName } shape (mapped to metrics.endpoint).
Important (P2):
- Browser safety: the heavy Node OTel SDK + core are now loaded via DYNAMIC
import() inside initTelemetry/shutdownTelemetry; only @opentelemetry/api is a
static import. The package root re-exporting telemetry.ts no longer statically
pulls server-only code into a browser bundle graph. (initTelemetry is now async.)
- OTLP logs honor env precedence (OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, then
OTEL_EXPORTER_OTLP_ENDPOINT + /v1/logs, then config) like traces/metrics, and
the hardcoded TBD per-network otlpLogs fallback is removed (no shipping to a
placeholder URL).
- OtlpLogWorker gains async shutdown() that AWAITS the final flush; the daemon
disable/shutdown path awaits it (stopOtlpExporter/stopTelemetry are async).
- Metrics semantics: publish failures that throw now record
publishTotal{outcome:'error'} (try/catch around both publish bodies); ACK
result:'ack' is recorded only AFTER signature/merkle/identity validation
(rejections counted result:'rejected', not 'ack'); the sync missing-
contextGraphId early return records syncResponseTotal{outcome:'invalid'}; the
chain populate+sign exhaustion counter is relabeled eth_estimateGas (it is the
prepare phase, not eth_sendRawTransaction).
- Ops docs: node-config.example.json sets traces/metrics enabled:false (polaris
backend is logs-only); config.alloy comment no longer says "localhost" while
binding 0.0.0.0:4318.
Coverage: new tests for the real instrumented paths — ACK after-validation
metric (mutation-verified), sync early-return metric, telemetry enable→disable→
re-enable lifecycle, real OTLP-collector bootstrap, legacy-config compat, plus
the redaction cases above. Added @opentelemetry/api/sdk-metrics as test-only
devDeps to chain/publisher/agent.
All affected suites green: core redaction 15, publisher 1270, node-ui 1477,
agent telemetry/sync/publish subset 49.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…llection # Conflicts: # packages/chain/src/evm-adapter-base.ts
… fan-out (#1317 review) - agent: extract DKGAgent.recordPublishOutcome(outcome, source, startedAt, chainId) and call it from both publish entry points (_publish + _publishFrom SharedMemory) instead of pasting the metric sequence into each — addresses the "publish telemetry duplicated across both paths" review note. - cli: extract the daemon log-sink fan-out into createDaemonLogSink (daemon/log-sink.ts) — the privacy trust boundary that writes the FULL record locally but ships only a REDACTED copy to active remote shippers. lifecycle.ts delegates to it. New log-sink.test.ts proves: local DB gets the original, every active shipper gets the redacted copy (incl. operator extra-redact keys), no remote push when no shipper is active, null shippers skipped, and a thrown DB write never breaks the node — closing the "remote log redaction wiring is not verified" gap. cli + agent build green; log-sink 4, telemetry-config 7, agent publish/sync 35. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…adow) The publish-dedup helper was added as a PublishMethods method, but the agent unit tests invoke `PublishMethods.prototype._publish.call(stub, …)` with a hand-built `this`, so `this.recordPublishOutcome` was undefined — its TypeError masked the real OVERSIZED_RDF_LITERAL error (publish-literal-size + encrypt-inline-policy, 5 tests). It needs no `this` (only getMetrics()), so it's now a module-level function called directly. Tests green (23/23). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ler metric coverage (#1317 review) 🔴 P1 redaction: an unquoted `seed=<phrase>` leaked all-but-first-word — `seed` was a single-token key, not a phrase key. Added `seed` (and `passphrase`) to PHRASE_KEYS and generalized the phrase value matcher to consume a single token OR a bare token + up to 23 trailing alpha words, so it redacts BOTH `seed=12345`/`seed=0xabc` AND `seed=legal winner thank …` while stopping at a delimiter and not false-positiving on "seeded". +regression tests. 🟡 coverage — chain write/receipt failover metrics: new rpc-failover-telemetry.unit.test.ts drives the real RpcFailoverClient.broadcast + getReceipt (success + all-endpoints-exhausted) and asserts dkg.chain.rpc.total / dkg.chain.rpc.failover.total with bounded labels and the exhaustion counter — read-path coverage didn't touch these branches. 🟡 coverage — inbound storage-ACK metrics: storage-ack-handler.test.ts now drives the real handler ack + decline paths and asserts dkg.ack.handler.total{outcome:'ack'} and {outcome:'decline',decline_code} with no high-cardinality labels. core redaction 19, chain rpc-failover-telemetry 4, publisher storage-ack 21; full build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gs metric test (#1317 review) 🔴 P1 metric-cardinality: ackPeerTotal recorded `decline_code` straight from an untrusted peer StorageACK (only length-sanitized), so a peer could emit endless unique codes and explode time-series cardinality. Added `boundedDeclineCodeLabel()` in core (maps anything outside the fixed STORAGE_ACK_DECLINE_CODES enum to 'other') and applied it to the metric label in ack-collector (peer-supplied) AND defensively in storage-ack-handler. The full code still rides the span + declines diagnostic map; only the metric dimension is bounded. 🟡 coverage: chain-rpc-telemetry.unit.test.ts now drives the real queryEventLogsPage (eth_getLogs) success + all-backends-error paths and asserts dkg.chain.rpc.total{rpc_method:eth_getLogs, outcome} with bounded labels. core redaction 19, publisher ack-metrics+storage-ack 23, chain telemetry 8; full build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ing once + sync metric test (#1317 review) 🟡 ack-collector: quorumOutcomeFromError classified the WHOLE legacy message with `includes('storage_ack_timeout')`, so a peer could put that substring in a decline message and flip an insufficient/impossible quorum into the 'timeout' bucket. Now classifies on the structured PREFIX (`startsWith`). Exported + tested (insufficient message carrying 'storage_ack_timeout' decline text → 'impossible'). 🟡 rpc-failover-client: extracted one `recordRpcOutcome(method, outcome, {retryable, exhausted})` boundary that owns the dkg.chain.rpc.total (+ failover) recording with a single identical bounded label shape; broadcast/getReceipt/readContract no longer hand-roll the label object per branch (duration stays in each method's uniform `finally`). Behaviour unchanged — the broadcast/receipt/read telemetry tests still pass. 🟡 coverage: new agent/test/sync-transport-metrics.test.ts drives the real sendSyncRequest (success + error) and asserts dkg.sync.request.total{outcome, protocol_id} — previously only hand-emitted via the facade. publisher ack-metrics 4, chain telemetry 33, agent sync 3; full build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…review P3)
The metric-label docstrings drifted from the call sites:
- ackPeerTotal documented {ack|decline|transport_error} but the collector also
emits result:"rejected" (signature/merkle/identity validation failure).
- syncResponseTotal omitted "invalid" (malformed/missing-contextGraphId early
return in the responder).
- publishTotal omitted "error" (publish flow throwing before a status).
Docs now list the full enum + a one-line meaning for the non-obvious values, so
dashboard/alert authors aren't misled. No code/behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ic tests (#1317 review) 🟡 telemetry.ts: replaced the single `configured` flag with independent tracer/meter provider tracking. A metrics-only init no longer makes a later traces init silently no-op — each signal registers iff it has an endpoint and isn't already registered; isTelemetryConfigured() = either provider set. +regression test (metrics-first then traces-only still registers a real tracer). 🟡 coverage — publish real path: publish-outcome-metrics.test.ts drives the real _publish (source=direct) and publishFromSharedMemory (source=swm) to a throw and asserts dkg.publish.total{outcome:error, source} — a dropped recordPublishOutcome now fails the build. 🟡 coverage — protocol-router real path: protocol-router-metrics.test.ts drives the real ProtocolRouter.send (success + forced failure, overriding only sendInner) and asserts dkg.protocol_router.send.total{outcome, protocol_id}. node-ui telemetry 13 + protocol-router 2, agent publish-metrics 2; full build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… traces/metrics (#1317 review) 🟡 The log exporter (OtlpLogWorker) and the OTel SDK bootstrap (telemetry.ts) each built the node-identity resource attributes independently, so a key/default could drift between Loki labels and trace/metric resources (e.g. logs defaulted service.instance.id to nodeName/peerId while traces/metrics only set it when the caller supplied it). Extracted ONE canonical builder + type, `buildTelemetryResourceAttrs` / `TelemetryResourceInput` (node-ui/src/telemetry-resource.ts, pure/no-SDK); both the log worker and the SDK bootstrap now consume it, and `TelemetryResource` is an alias of the shared type. Emitted attributes are unchanged (otlp-log-worker + telemetry suites green: 20 tests), so Loki labels/dashboards are unaffected — there's now just one place to change service name / instance id / network / role / chain / peer / commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…review 🔴) 🔴 The config is user JSON/YAML, so the TS union doesn't guard the runtime boundary: resolveLogExporterMode did `exporter ?? 'syslog'`, so a typo'd / unsupported value (e.g. "none ", "otpl", "disabled") fell through to the legacy syslog pusher — sending logs off-node even when the operator intended OTLP-only or local-only. Now: UNSET still defaults to 'syslog' (documented), the three known values pass through, and ANY other value FAILS CLOSED to 'none' (local-only, never off-node). startTelemetry logs a warning naming the bad value + valid options. +tests proving unknown→none and the isUnknownLogExporter flag. cli telemetry-config 8; full build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ verify wire trace IDs (#1317 review) 🟡 sync responder: a malformed request (parseSyncRequest throws on non-JSON peer bytes) escaped before any syncResponseTotal.add, so malformed failures were invisible. Wrapped ALL pre-limiter work in a guard that records dkg.sync.response.total{outcome:'invalid'} and re-throws (limiter promise keeps its own ok/error recording — no double count). +handler test for malformed bytes. 🟡 OtlpLogWorker: extracted the pure OTLP/HTTP wire encoder into `encodeOtlpLogPayload(batch, resourceAttrs, scopeVersion)` so payload-shape changes are independent from the worker's buffer/retry/backoff mechanics (buildPayload now delegates). No behaviour change. 🟡 trace correlation: added OtlpLogWorker tests asserting a pushed record's W3C traceId/spanId survive as TOP-LEVEL OTLP fields on the captured wire request (and are absent when no span was active) — a dropped spread in the encoder now fails the build. node-ui otlp-log-worker 9, agent sync-responder-metrics 2; full build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lupuszr
approved these changes
Jun 29, 2026
branarakic
approved these changes
Jun 30, 2026
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Task: enable log collection on core nodes → expanded per review into a proper observability foundation: opt-in OpenTelemetry logs + traces + metrics, off by default, built on V10's existing
Loggerseam. Nodes are independently operated, so the model is local-first + opt-in forwarding + at-source redaction.Architecture
Call sites use the ambient
@opentelemetry/api(withSpan/getMetricsindkg-core); the SDK registers once at daemon boot, only when enabled and an endpoint resolves. Disabled ⇒ true no-op (no provider, no outbound calls, no per-call-site guards).In this PR
OtlpLogWorker(bounded buffer + retry/backoff + redaction); records gaindkg.*resource/record attrs +trace_id/span_id.agent.publish,agent.publish_from_swm,publisher.ack_collect/ack_peer_request/ack_verify_identity/storage_ack_handler,chain.eth_call/eth_getLogs/tx_send/tx_submit/tx_wait,sync.request/sync.response. Duration/status/exception; inbound boundaries are fresh roots.dkg.publish.*,dkg.ack.*,dkg.chain.rpc.*,dkg.sync.*); bounded attribute keys only (no peer/tx/cg/op ids as labels).telemetry.{logs,traces,metrics}, env-first (OTEL_EXPORTER_OTLP_*), no TBD prod defaults; a signal registers only when its endpoint resolves.syslogstill supported. Shutdown flush.tools/log-collection-poc/.Verified
dkgdaemon → local collector on127.0.0.1:4318exported logs+traces+metrics; payloads contained all 11 span names (publish → ACK collect → per-peer → verify → RPC), all 8 metric instruments, resource identity, anddkg.operation_idon logs.Remaining (access-gated, not code)
Deploy on the OriginTrail fleet (Alloy on the Loki host, Cloudflare ingest hostname + token, node configs) — see
tools/log-collection-poc/production/MANAGER-HANDOFF.md. Real collector endpoints for testnet/mainnet traces/metrics to be provided by ops (inert until then, by design).🤖 Generated with Claude Code