Skip to content

protobuf-formats.md

Latest commit

 

History

History
207 lines (165 loc) · 13.1 KB

protobuf-formats.md

File metadata and controls

207 lines (165 loc) · 13.1 KB
Raw

Protobuf formats

xtcp2's configuration and its exported data are defined as Protocol Buffers. There are three schemas — the daemon config, the data export record, and a small ClickHouse test format — and buf generates bindings for Go, C++, Python, Dart, and OpenAPI/Swagger from them. This document describes each schema, links to the source and generated Go, and explains how to regenerate everything.

Table of contents

Layout

The canonical .proto sources live under proto/; each is its own <name>/v1/<name>.proto module. buf (configured by buf.yaml and buf.gen.yaml) compiles them and writes all generated code into a single gen/ tree, one subdirectory per language (gen/go, gen/python, gen/dart, gen/cpp, gen/openapi). Generated files are committed, so a clean checkout builds without running buf.

Schema Source Generated Go
Config proto/xtcp_config/v1/xtcp_config.proto gen/go/xtcp_config/
Data export proto/xtcp_flat_record/v1/xtcp_flat_record.proto gen/go/xtcp_flat_record/
ClickHouse test proto/clickhouse_protolist/v1/clickhouse_protolist.proto gen/go/clickhouse_protolist/

Config: xtcp_config

Source: proto/xtcp_config/v1/xtcp_config.proto · Generated Go: gen/go/xtcp_config/ (xtcp_config.pb.go messages, xtcp_config_grpc.pb.go service stubs, xtcp_config.pb.gw.go REST gateway).

The daemon's entire runtime configuration is the XtcpConfig message — every CLI flag in cmd/xtcp2 maps to a field on it (poll frequency/timeout, netlinkers, marshaller, destination, Kafka/S3/Pyroscope settings, io_uring tuning, …). It also defines a ConfigService for runtime control:

RPC Purpose Disruption
Get(GetRequest) → GetResponse Read the live XtcpConfig (S3 credentials redacted). none
SetPollFrequency(SetPollFrequencyRequest) → SetPollFrequencyResponse Change the poll frequency + timeout live. none (hot)
TriggerPoll(TriggerPollRequest) → TriggerPollResponse Trigger a single poll immediately, without changing the cadence. none (hot)
TriggerPollBurst(TriggerPollBurstRequest) → TriggerPollBurstResponse Schedule count polls interval apart (e.g. a socket snapshot every 10s for a minute). none (hot)
SetS3Upload(SetS3UploadRequest) → SetS3UploadResponse Change the s3parquet flush timer and/or byte cap live. none (hot)
SetEnvelopeFlush(SetEnvelopeFlushRequest) → SetEnvelopeFlushResponse Change the protobufList envelope flush row and/or byte caps live. none (hot)
Set(SetRequest) → SetResponse Validate and apply a full new XtcpConfig via a graceful soft restart — re-exec (syscall.Exec) in place, same container/PID. brief (soft restart)

The "hot" RPCs change a running daemon with no restart; Set re-execs for config baked in at startup (exported field groups, tag/location/hostname, marshaller, destination). Fields carry buf.validate CEL constraints that are enforced at startup and on every RPC — e.g. numeric ranges, marshal_to length 3–40, and a message-level rule that poll_frequency > poll_timeout. Invalid config makes the daemon refuse to start (or the RPC return InvalidArgument) with a precise message. See grpc-api.md for the full runtime operator-control workflow and the xtcp2ctl / grpcurl client usage.

Data export: xtcp_flat_record

Source: proto/xtcp_flat_record/v1/xtcp_flat_record.proto · Generated Go: gen/go/xtcp_flat_record/.

This is the exported TCP data. Two core messages:

  • XtcpFlatRecord — one socket snapshot, deliberately flat (no nesting): timestamp, hostname, network namespace, the inet_diag message fields, the full tcp_info, socket memory, congestion-control state (BBR/DCTCP/Vegas), cgroup/class IDs, and more. The flatness is what makes CSV/TSV and tabular analysis easy. Addresses are raw bytes; the congestion algorithm is the kernel's name string (inet_diag_cong) plus the derived CongestionAlgorithm enum (inet_diag_cong_enum, CONGESTION_ALGORITHM_CUBIC … BBR3).
  • Envelope { repeated XtcpFlatRecord row } — a batch of records. This is the unit the daemon marshals and ships; framed length-delimited it is exactly ClickHouse's ProtobufList input format. See protobuflist-migration.md for the wire-format deep dive and output-and-destinations.md for the marshallers.

Every record carries two provenance fields at the low field numbers:

  • schema_version (field 1) — the record format epoch, stamped unconditionally from the daemon constant XtcpFlatRecordSchemaVersion (currently 2). Bump it whenever a field is renamed or renumbered. 0 is the "legacy" bucket: pre-versioning daemons never set the field, so it decodes to the proto3 zero default. Downstream this drives per-version ClickHouse routing — see record-versioning.md.
  • daemon_version (field 2) — build provenance (git commit / date / version from -ldflags), for debugging which binary produced a row. Informational only; not used for routing.

Field layout policy

The field-number space is allocated in blocks so that related fields stay together, every block has headroom, and nothing is ever reused (the proto's header comment is the authoritative copy of this policy):

Range Contents Notes
1–2 schema_version, daemon_version single-byte tags
3–299 metadata (host, netns, container, labels, bookkeeping, uplink slots 100s/200s) numbers frozen; free sub-ranges listed in the proto
300–399 enrichment (daemon-computed, not from the kernel) 300 socket-side · 310–349 destination-side · 350–389 reserved for future source-side
400–999 spare
1000+ payload, one hundred-block per kernel subsystem inet_diag_msg 1000s · meminfo 1100s (deprecated) · tcp_info 1200s · cong 1300s · tos/tclass 1400s · skmeminfo 1500s · shutdown 1600s · vegas 1700s · dctcp 1800s · bbr 1900s · class_id/sockopt/cgroup_id 2000s; next free block 2100

Every tag ≤ 2047 costs two bytes on the wire (2048+ costs three), so free slots inside existing blocks are filled before a new block is opened above 2047.

Payload names mirror the kernel. A payload field is named after the kernel struct member it copies (tcpi_rttvar → tcp_info_rttvar, SK_MEMINFO_RCVBUF → sk_mem_info_rcvbuf), and struct-less INET_DIAG_* attributes take the lowercased attribute name (INET_DIAG_TOS → inet_diag_tos, INET_DIAG_CGROUP_ID → inet_diag_cgroup_id). The one deliberate exception is the descriptive inet_diag_msg_socket_{source,destination,…} sockid names. Every payload field carries a trailing comment naming its kernel source, e.g. // struct tcp_info.tcpi_rttvar (__u32), // SK_MEMINFO_RCVBUF (__u32, sock_diag.h), // INET_DIAG_TOS (5): inet->tos (__u8, net/ipv4/inet_diag.c), or // derived by xtcp from inet_diag_cong (not a kernel field). go run ./tools/proto-field-audit fails if a field with tag ≥ 1000 lacks such a comment. Members the deserializer does not read yet (the post-6.10 AccECN tcp_info fields) are pre-assigned by comment at 1266–1276.

Epoch 2 (2026-09) applied this policy retroactively: 18 payload fields and one enrichment field were renamed and three were renumbered. The full old → new table lives in record-versioning.md; the old names and numbers are reserved in the proto.

Deprecated: mem_info_* (fields 1101–1104). These four socket-memory fields are a value-subset of the sk_mem_info_* fields (mem_info_rmem=sk_mem_info_rmem_alloc, mem_info_wmem=sk_mem_info_wmem_queued, mem_info_fmem=sk_mem_info_fwd_alloc, mem_info_tmem=sk_mem_info_wmem_alloc) — the kernel derives both from the same sk counters. The meminfo deserializer is off by default, so on current records these columns ship as 0; use sk_mem_info_* instead (see netlink/collection.md). The fields are retained (never renumbered), so the deprecation alone did not bump schema_version: the record is structurally identical and the data is fully recoverable from sk_mem_info_*.

It also defines the streaming XTCPFlatRecordService, which xtcp2client consumes:

RPC Shape Purpose
FlatRecords(FlatRecordsRequest) → stream FlatRecordsResponse server streaming Daemon pushes records as it collects them.
PollFlatRecords(stream PollFlatRecordsRequest) → stream PollFlatRecordsResponse bidirectional Client drives a poll on demand.

Each FlatRecordsResponse/PollFlatRecordsResponse carries a single XtcpFlatRecord (the gRPC path is per-record; the Envelope batch is only used by the destination pipeline).

ClickHouse test format: clickhouse_protolist

Source: proto/clickhouse_protolist/v1/clickhouse_protolist.proto · Generated Go: gen/go/clickhouse_protolist/.

A tiny Record { repeated uint32 my_uint32 } + Envelope { repeated Record rows } used to validate ClickHouse's ProtobufList ingestion path in isolation (the clickhouse_* tools under cmd/). Not part of the live data path.

Generated code

buf.gen.yaml drives generation for every schema into a single committed gen/ tree. Every plugin is a local, nix-pinned binary (see nix/versions.nix), not a buf.build remote plugin — so generation runs fully offline and can't be rate-limited by buf's cloud. The plus-grpc C++/Python stubs come from grpc_cpp_plugin / grpc_python_plugin; the Python/C++ message code and OpenAPI use protoc builtins and grpc-gateway.

Language Plugin(s) (nixpkgs) Output
Go protoc-gen-go, protoc-gen-go-vtproto, protoc-gen-go-grpc, protoc-gen-grpc-gateway gen/go/<schema>/ (*.pb.go, *_grpc.pb.go, *.pb.gw.go, *_vtproto.pb.go)
C++ protoc (--cpp_out), grpc_cpp_plugin gen/cpp/<schema>/v1/
Python protoc (--python_out/--pyi_out), grpc_python_plugin gen/python/<schema>/v1/
Dart protoc-gen-dart (grpc) gen/dart/<schema>/v1/
OpenAPI 2.0 protoc-gen-openapiv2 gen/openapi/<schema>/v1/<schema>.swagger.json

The bufbuild/validate-cpp plugin (protovalidate's C++ codegen) has no nixpkgs equivalent and nothing in-repo consumes the C++ output, so it is intentionally not generated — C++ keeps message and gRPC code. Go validation needs no plugin (protovalidate-go is runtime-reflection based).

Rebuilding

After editing any .proto, regenerate the bindings. From the dev shell (nix develop):

nix run .#regen-protos        # buf dep update → buf lint → buf build → buf generate
# equivalently, the helper available in the dev shell:
regen-protos

This runs nix/protos/buf-generate.nix with local nix-pinned plugins (no buf-cloud round-trip). After regenerating, review and commit the drift across the gen/ tree (gen/go, gen/cpp, gen/python, gen/dart, gen/openapi).

Notes:

  • The generator also re-syncs the ClickHouse format schemas (build/containers/clickhouse/format_schemas/{xtcp_flat_record,clickhouse_protolist}.proto) from the canonical protos, so those mounts never drift. (This absorbed the old check_protos.bash.) The daemon's cmd/xtcp2/xtcp_flat_record.proto is a symlink to the canonical proto, so it stays in sync automatically.
  • Adding a field means regenerating all language bindings; commit them together.
  • buf.validate rules live in the .proto (e.g. marshal_to min length), so loosening or tightening a constraint is a proto edit + regen, not a Go change.

See also