ci(ff): support feature freeze automation by fr4nc1sc0-r4m0n · Pull Request #7339 · elastic/fleet-server

fr4nc1sc0-r4m0n · 2026-07-09T09:30:20Z

What is the problem this PR solves?

Fleet Server release automation still relies on Makefile-based scripts and external CLI tools. This PR migrates the feature-freeze release workflow to pure Go Mage targets so release steps can run without depending on hub, gh, sed, yq, or Python.

This continues the work from #6584, rebased onto current main with merge conflicts resolved.

How does this PR solve the problem?

Adds Mage release commands for major/minor and patch workflows (release:runMajorMinor, release:runPatch, and supporting targets)
Extracts release logic into dev-tools/mage/release/ (release.go, git.go, github.go), aligned with elastic-agent and beats
Keeps thin mage release:* wrappers in magefile.go that delegate to the release package
Makes all release steps idempotent so workflows can be safely re-run after partial failure or CI retry:
- updateVersion no-ops when DefaultVersion already matches
- updateMergify no-ops when the backport rule already exists
- createBranch checks out an existing branch instead of failing
- commitAll skips when there are no changes
- push succeeds when the remote is already up to date
- createPR returns an existing open PR for the same head/base
Adds DRY_RUN mode for safe local testing
Documents the workflow, idempotency guarantees, and package layout in RELEASE.md

How to test this PR locally

export PROJECT_OWNER="your-user"
export CURRENT_RELEASE="9.5.0-test"
export GITHUB_TOKEN=$(gh auth token)
export DRY_RUN=true

# Dry run major/minor workflow
mage release:runMajorMinor

# Review changes
git diff

# Run release unit tests
cd dev-tools && go test ./mage/release/... -count=1

Re-run the same commands to confirm idempotent behavior (no duplicate branches, commits, PRs, or file changes).

Design Checklist

I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool

Validation

Merged latest main and resolved go.mod / go.sum conflicts
cd dev-tools && go test -race -count=1 ./mage/release/... passes
go build -tags mage succeeds with thin magefile wrappers

Related issue: https://github.com/elastic/observability-robots/issues/3404

Resolve go.mod/go.sum conflicts by keeping main dependency versions and retaining go-git/go-github deps required for release automation.

snyk-io · 2026-07-09T09:30:37Z

⚠️ Snyk checks are incomplete.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
⚠️	Open Source Security	0	0	0	0	See details
⚠️	Licenses	0	0	0	0	See details

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

mergify · 2026-07-09T09:30:56Z

This pull request does not have a backport label. Could you fix it @fr4nc1sc0-r4m0n? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

Allow release workflows to be safely retriggered by no-oping when the target version is already applied, reusing existing branches and open PRs, and skipping empty commits.

Move release workflow logic into a dedicated package aligned with elastic-agent and beats, leaving thin mage release:* wrappers in magefile.go.

github-actions · 2026-07-09T11:03:26Z

TL;DR

:gcloud: Cloud e2e FIPS Test failed before running the E2E tests because the deployed Cloud resource did not expose a Fleet endpoint: Terraform’s fleet_url output was empty, and .buildkite/scripts/cloud_e2e_test.sh exits with status 1 in that case.

Remediation

Retry the Cloud e2e FIPS job if this was a transient Cloud/integrations-server provisioning issue.
If it repeats, inspect the Cloud deployment/integrations-server logs for the custom FIPS image docker.elastic.co/beats-ci/elastic-agent-cloud-fips:pr-7339-bf17d6ffe9cf; the next fix should make that deployment expose integrations_server.endpoints.fleet before mage test:cloudE2ERun starts.

Investigation details

Root Cause

The immediate failure is a configuration/infrastructure failure in the Cloud FIPS deployment, not a test assertion failure. The step script reads fleet_url from Terraform and exits when it is empty:

# .buildkite/scripts/cloud_e2e_test.sh:31-45
FLEET_SERVER_URL=$(terraform output --raw --state=dev-tools/cloud/terraform/terraform.tfstate fleet_url)
...
if [[ "${FLEET_SERVER_URL}" == "" ]]; then
    message="FLEET_SERVER_URL is empty, cloud e2e tests cannot be executed"
    ...
    exit 1
fi

That output is derived from the integrations server Fleet endpoint:

# dev-tools/cloud/terraform/outputs.tf:28-30
output "fleet_url" {
  value = ec_deployment.deployment.integrations_server.endpoints != null ? ec_deployment.deployment.integrations_server.endpoints.fleet : ""
}

The captured teardown log shows fleet_url = "" -> null while the integrations server still had a generic HTTPS endpoint, then the job exited 1. That means the job stopped before mage test:cloudE2ERun could execute.

Evidence

Build: https://buildkite.com/elastic/fleet-server/builds/15397
Job/step: :gcloud: Cloud e2e FIPS Test (.buildkite/scripts/cloud_e2e_test.sh)
Key log excerpt:

Changes to Outputs:
  - fleet_url        = "" -> null
  - integrations_url = "(b58eb8a8e21842a1919a99ef...st2.gcp.elasticcloud.com/redacted)" -> null
...
Destroy complete! Resources: 1 destroyed.
^^^ +++
🚨 Error: The command exited with status 1
^^^ +++
user command error: exit status 1

Verification

Not run locally; this requires the Buildkite Cloud/FIPS provisioning environment and the prefetched job log only contains teardown plus the final exit status, not the earlier deployment startup logs.

Follow-up

No existing flaky-test issue was found for this Cloud e2e FIPS/empty fleet_url failure pattern.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Enable gomoddirectives replace-local so the dev-tools submodule replace required by mage release imports passes golangci-lint.

The root go.mod replaces dev-tools with a local path. Docker layer caching must include dev-tools/go.mod and go.sum so go mod download can resolve the replaced module.

mergify · 2026-07-09T12:36:07Z

This pull request is now in conflicts. Could you fix it @fr4nc1sc0-r4m0n? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/support-ff-fleet-server upstream/feature/support-ff-fleet-server
git merge upstream/main
git push upstream feature/support-ff-fleet-server

mallendem · 2026-07-10T10:13:24Z

 	github.com/elastic/elastic-agent-client/v7 v7.18.1
 	github.com/elastic/elastic-agent-libs v0.44.0
 	github.com/elastic/elastic-agent-system-metrics v0.14.4
+	github.com/elastic/fleet-server/dev-tools v0.0.0


Is there a way we can avoid polluting the project's go.mod?

mallendem · 2026-07-10T10:13:59Z

+		if err != nil {
+			return fmt.Errorf("failed to checkout existing branch: %w", err)
+		}
+		fmt.Printf("✓ Checked out existing branch: %s\n", branchName)


We should't checkout branches. Maybe delete them first?

mallendem · 2026-07-10T10:16:49Z

+	if err != nil {
+		return err
+	}
+	return release.CreateReleaseBranch(cfg, ".", os.Getenv("DRY_RUN") == "true")


Suggested change

return release.CreateReleaseBranch(cfg, ".", os.Getenv("DRY_RUN") == "true")

return release.CreateReleaseBranch(cfg, ".", envToBool("DRY_RUN"))

mallendem · 2026-07-10T10:17:15Z

+	if err != nil {
+		return err
+	}
+	return release.CreateReleasePR(cfg, ghClient, os.Getenv("DRY_RUN") == "true")


We do have envToBool as a helper available.

mallendem · 2026-07-10T10:20:00Z

+}
+
+// PreparePatchRelease prepares files for a patch release.
+func PreparePatchRelease(cfg *ReleaseConfig) error {


This function is not doing anything besides setting the version of *ReleaseCofnig

mallendem · 2026-07-10T10:20:35Z

+}
+
+// RunPatchRelease orchestrates the complete patch release workflow.
+func RunPatchRelease(cfg *ReleaseConfig, dryRun bool) error {


Is this a no-op?

mallendem · 2026-07-10T10:22:11Z

+		return nil
+	}
+
+	err := g.repo.Push(&git.PushOptions{


What credentials are we using here?

v1v and others added 8 commits March 12, 2026 17:39

ci(ff): support feature freeze automation phase 1

a1fecf3

ci(ff): support feature freeze automation phase 2

23422b9

update

a666488

fix linting

eff7847

Merge branch 'main' into feature/support-ff-fleet-server

8f0005e

go mod tidy

de623a9

Merge branch 'main' into feature/support-ff-fleet-server

f48d911

Merge upstream/main into feature/support-ff-fleet-server

6d46ae9

Resolve go.mod/go.sum conflicts by keeping main dependency versions and retaining go-git/go-github deps required for release automation.

fr4nc1sc0-r4m0n requested a review from a team as a code owner July 9, 2026 09:30

fr4nc1sc0-r4m0n added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog labels Jul 9, 2026

fr4nc1sc0-r4m0n requested review from blakerouse and macdewee July 9, 2026 09:30

fr4nc1sc0-r4m0n added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog labels Jul 9, 2026

fr4nc1sc0-r4m0n self-assigned this Jul 9, 2026

fr4nc1sc0-r4m0n added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-8.19 Automated backport to the 8.19 branch labels Jul 9, 2026

fr4nc1sc0-r4m0n mentioned this pull request Jul 9, 2026

ci(ff): support feature freeze automation #6584

Closed

8 tasks

This comment has been minimized.

Sign in to view

fr4nc1sc0-r4m0n added 3 commits July 9, 2026 11:53

fix(ci): update magefile_test.go license header to Elastic License 2.0

bf17d6f

Make release automation idempotent on re-run

c67df45

Allow release workflows to be safely retriggered by no-oping when the target version is already applied, reusing existing branches and open PRs, and skipping empty commits.

refactor(release): extract FF automation to dev-tools/mage/release

64eae34

Move release workflow logic into a dedicated package aligned with elastic-agent and beats, leaving thin mage release:* wrappers in magefile.go.

This comment has been minimized.

Sign in to view

fr4nc1sc0-r4m0n added 2 commits July 9, 2026 13:15

fix(ci): allow local go.mod replace for dev-tools module

815c688

Enable gomoddirectives replace-local so the dev-tools submodule replace required by mage release imports passes golangci-lint.

fix(docker): copy dev-tools go.mod before go mod download

457a0a7

The root go.mod replaces dev-tools with a local path. Docker layer caching must include dev-tools/go.mod and go.sum so go mod download can resolve the replaced module.

mallendem reviewed Jul 10, 2026

View reviewed changes

	return release.CreateReleaseBranch(cfg, ".", os.Getenv("DRY_RUN") == "true")
	return release.CreateReleaseBranch(cfg, ".", envToBool("DRY_RUN"))

Uh oh!

Conversation

fr4nc1sc0-r4m0n commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the problem this PR solves?

How does this PR solve the problem?

How to test this PR locally

Design Checklist

Checklist

Validation

Uh oh!

snyk-io Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Snyk checks are incomplete.

Uh oh!

mergify Bot commented Jul 9, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions Bot commented Jul 9, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

mergify Bot commented Jul 9, 2026

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mallendem Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fr4nc1sc0-r4m0n commented Jul 9, 2026 •

edited

Loading

snyk-io Bot commented Jul 9, 2026 •

edited

Loading