Skip to content

feat(remoteconfig): add Dynamic Instrumentation system probe module with per-node Remote Config support#3200

Open
grantseltzer wants to merge 3 commits into
DataDog:mainfrom
grantseltzer:grantseltzer/di-remote-enable-signal
Open

feat(remoteconfig): add Dynamic Instrumentation system probe module with per-node Remote Config support#3200
grantseltzer wants to merge 3 commits into
DataDog:mainfrom
grantseltzer:grantseltzer/di-remote-enable-signal

Conversation

@grantseltzer

@grantseltzer grantseltzer commented Jun 26, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Adds the Dynamic Instrumentation system probe module to the operator, plus the operator-side plumbing to enable it through Remote Config.

  • Adds a features.dynamicInstrumentation field. Setting it on a DatadogAgent (or a DatadogAgentProfile) now turns the module on.
  • Adds the feature handler that flips it on (env vars, volumes, system-probe setup), the same way USM does.
  • Teaches the Remote Config callback to react to it: a payload with no node selector enables it cluster-wide, a payload with a node selector creates a DatadogAgentProfile so only those nodes get it, and removing or disabling a payload tears the profile back down.

Motivation

This is the operator side of letting users enable the module from the Datadog UI. The backend product and UI that send the Remote Config payload come in follow-up changes, so the full UI flow isn't wired up yet.

Additional Notes

Nothing in the Remote Config path does anything until a payload arrives, so this is safe to merge before the backend and UI land. The config field itself works today.

Minimum Agent Versions

  • Agent: TBD (needs a version with the DI system probe module; confirming with the
    Agent team)
  • Cluster Agent: none

Describe your test plan

Build, vet, lint, and existing tests pass. Two ways to exercise it:

  1. Set features.dynamicInstrumentation.enabled: true on a DatadogAgent and confirm the node agents pick up the module.
  2. Send a hand-made Remote Config payload and confirm: no node selector enables it everywhere, a node selector creates a profile for just those nodes, and removing the config deletes the profile.

Checklist

  • PR has at least one valid label: enhancement
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: [signing commits][1])

[1]: https://docs.github.com/en/authentication/man Jump to bottom (ctrl+End) ↓ cation/signing-commits

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jun 26, 2026

Copy link
Copy Markdown

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

pull request linter | Check Milestone   View in Datadog   GitHub Actions

pull request linter | build   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 39cabd4 | Docs | Datadog PR Page | Give us feedback!

@grantseltzer grantseltzer force-pushed the grantseltzer/di-remote-enable-signal branch from cde1f24 to e4dd37e Compare June 26, 2026 18:19

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cde1f24292

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

},
Config: &v2alpha1.DatadogAgentSpec{
Features: &v2alpha1.DatadogFeatures{
DynamicInstrumentation: &v2alpha1.DynamicInstrumentationFeatureConfig{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow Dynamic Instrumentation profiles through validation

When a node-scoped Remote Config payload arrives, this builds a DatadogAgentProfile whose config sets features.dynamicInstrumentation. The DatadogAgent profile reconciliation path validates every profile before applying it (agentprofile.ValidateProfileAndReturnRequirements -> v1alpha1.ValidateDatadogAgentProfileSpec), and api/datadoghq/v1alpha1/datadogagentprofile_validation.go still allowlists only gpu and apm, so these generated profiles are marked invalid/skipped and the per-node RC enablement never reaches matching nodes. Add dynamicInstrumentation to the DatadogAgentProfile feature allowlist (and its test allowlist) or avoid creating profiles with unsupported feature config.

Useful? React with 👍 / 👎.

Add a DynamicInstrumentationFeatureConfig type with an Enabled flag and wire
it into DatadogFeatures as the dynamicInstrumentation field. Because
DatadogFeatures is embedded in DatadogAgentSpec, this field is also exposed on
the DatadogAgentInternal and DatadogAgentProfile CRDs, which lets a profile
enable the module on a node subset. Regenerated deepcopy, openapi, CRD
manifests, and docs via make generate and make manifests.
Add a dyninst feature that enables the Dynamic Instrumentation system probe
module on the node agent. The handler follows the USM pattern: it requires the
core agent and system-probe containers, sets HostPID, the AppArmor annotation,
system-probe capabilities, and the procdir/cgroups/debugfs/socket volumes, and
sets DD_DYNAMIC_INSTRUMENTATION_ENABLED, DD_SYSTEM_PROBE_ENABLED, and
DD_SYSPROBE_SOCKET on the core agent and system-probe containers. The env var
name matches the agent binding dynamic_instrumentation.enabled. Configure reads
features.dynamicInstrumentation.enabled from the spec and merges the cluster-wide
remote config status into the spec, so the same handler serves both the
cluster-wide and node-scoped (DatadogAgentProfile) enablement paths. Registered
the new DynamicInstrumentationIDType and the blank import in both the
datadogagent and datadogagentinternal controllers.
Wire the agent_config remote config product to enable the Dynamic
Instrumentation system probe module on a node subset. Add a dynamic_instrumentation
field to SystemProbeFeaturesConfig and a node_selector field to
DatadogAgentRemoteConfig, and merge the new field into the cluster-wide status
path alongside CWS/USM/CSPM. Because parseReceivedUpdates merges every config
into a single struct, a node selector carried there would collapse multiple
node-scoped payloads into one; to avoid that, parseReceivedUpdates now
partitions configs, returning the merged cluster-wide config plus the list of
node-scoped configs untouched. Each node-scoped config that enables the module
is reconciled into a DatadogAgentProfile named by a stable hash of its node
selector, with config.features.dynamicInstrumentation.enabled set so the feature
handler turns the module on only for matching nodes. Every callback reconciles
the full set of managed profiles and deletes any whose config was removed or
disabled, so teardown happens without leaking profiles. Verified with go build,
go vet, and the existing remoteconfig and feature tests.
@grantseltzer grantseltzer force-pushed the grantseltzer/di-remote-enable-signal branch from e4dd37e to 39cabd4 Compare June 26, 2026 18:27
@jeff-morgan-dd jeff-morgan-dd self-assigned this Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants