AgentOps CI/CD Pipeline for Amazon Bedrock AgentCore

End-to-end CI/CD pipeline for building, deploying, evaluating, and promoting AI agents on Amazon Bedrock AgentCore. Teams create agents via a self-service CLI portal; the pipeline handles the full AgentOps lifecycle from code generation through multi-environment promotion with automated quality gates.

Includes an e-commerce demo with 8 Lambda-based tools, mock data, and pre-built agent personas.

Supports both GitHub Actions and GitLab CI/CD -- same scripts, same stages, dual workflow definitions.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│  SELF-SERVICE PORTAL (CLI)                                              │
│                                                                         │
│  Developer selects: agent type, tools, memory, system prompt            │
│  Portal generates:  Strands agent code + agentcore.json + mcp.json     │
│  Portal executes:   git checkout -b agent/<name> -> commit -> push      │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │ git push triggers CI pipeline
                                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  CI/CD PIPELINE -- 8 STAGES (GitHub Actions or GitLab CI)               │
│                                                                         │
│  ┌──────────┐   ┌───────────┐   ┌────────────┐   ┌──────────────────┐ │
│  │ 1.       │   │ 2.        │   │ 3.         │   │ 4.               │ │
│  │ Validate │──>│ Deploy    │──>│ Smoke Test │──>│ On-Demand Eval   │ │
│  │ & Plan   │   │ to Dev    │   │ (invoke x15)│   │ (10 evaluators)  │ │
│  └──────────┘   └───────────┘   └────────────┘   └──────────────────┘ │
│                                                            │            │
│  ┌──────────────────┐   ┌─────────────┐   ┌──────────────▼──────────┐ │
│  │ 6.               │   │ 7.          │   │ 5.                      │ │
│  │ Verify Online    │   │ Deploy      │<──│ Metrics + Quality Gate  │ │
│  │ Eval Active      │   │ Staging     │   │ (CloudWatch + tiered)   │ │
│  └──────────────────┘   │  APPROVAL   │   └─────────────────────────┘ │
│                          └──────┬──────┘                                │
│                                 │                                       │
│                          ┌──────▼──────┐                                │
│                          │ 8.          │                                │
│                          │ Deploy Prod │                                │
│                          │  APPROVAL   │                                │
│                          └─────────────┘                                │
└─────────────────────────────────────────────────────────────────────────┘
                                 │
            ┌────────────────────┼────────────────────┐
            ▼                    ▼                     ▼
┌─────────────────┐  ┌────────────────────┐  ┌────────────────────┐
│ AgentCore       │  │ AgentCore          │  │ Amazon             │
│ Runtime         │  │ Gateway            │  │ CloudWatch         │
│ (Strands agent) │  │ (8 Lambda tools)   │  │ (AgentOps metrics) │
│ + Memory        │  │ + Semantic Search  │  │ + Eval dashboards  │
│ + Online Eval   │  │ + Auth management  │  │ + Cost tracking    │
└─────────────────┘  └────────┬───────────┘  └────────────────────┘
                               │
                     ┌─────────▼─────────┐
                     │ Amazon DynamoDB    │
                     │ 5 tables           │
                     │ (Demo mock data)   │
                     └───────────────────┘

How It Works

Developer runs the CLI portal (scripts/create-agent.sh) and answers configuration questions
Portal generates a complete agent project: Strands SDK code, AgentCore config, evaluators, online eval config
Portal commits to a new git branch (agent/<name>-<timestamp>) and pushes
CI pipeline triggers the 8-stage workflow automatically
Pipeline validates config, deploys to dev, runs smoke tests, evaluates with 10+ evaluators
Quality gate checks tiered thresholds -- safety metrics require highest scores
Metrics are published to CloudWatch; eval summary is posted as a PR/MR comment
Manual approval gates control staging and production promotion
Online eval runs continuously at configured sampling rates per environment

Prerequisites

Required Software

Tool	Version	Install	Purpose
Node.js	20+	`brew install node`	AgentCore CLI runtime
Python	3.13+	`brew install python@3.13`	CDK, Lambda functions
uv	Latest	`curl -LsSf https://astral.sh/uv/install.sh \| sh`	Python dependency management
AWS CLI	2.x	`brew install awscli`	AWS operations
AWS CDK	2.170+	`npm install -g aws-cdk`	Infrastructure deployment
GitHub CLI	Latest	`brew install gh`	PR comments, workflow monitoring
jq	Latest	`brew install jq`	JSON parsing in scripts
AgentCore CLI	0.7.1+	`npm install -g @aws/agentcore@0.7.1`	Agent lifecycle management

AWS Configuration

Configure AWS credentials using your preferred method (environment variables, ~/.aws/credentials, SSO, etc.):

# Verify credentials are working
aws sts get-caller-identity

# Expected output:
# {
#     "UserId": "EXAMPLE_USER_ID",
#     "Account": "123456789012",
#     "Arn": "arn:aws:iam::123456789012:user/your-iam-user"
# }

Required IAM permissions:

CloudFormation (CDK deployments)
DynamoDB (table creation, data operations)
Lambda (function deployment)
Bedrock AgentCore (runtime, gateway, memory, evaluators)
Bedrock Runtime (model invocation)
CloudWatch (metrics publishing)
IAM (role creation for Lambda and AgentCore)
S3 (CDK bootstrap bucket)

Bedrock Model Access

Enable the following models in your AWS account (Bedrock console > Model access):

Claude Sonnet 4.6 (us.anthropic.claude-sonnet-4-6-20261001-v1:0) -- agent runtime
Claude Sonnet 4.5 (us.anthropic.claude-sonnet-4-5-20250929-v1:0) -- evaluator model

Step-by-Step Deployment

Phase 1: Deploy Shared Infrastructure

Deploys DynamoDB tables, Lambda functions, and seed data. Then deploys the AgentCore Gateway that exposes the Lambdas as MCP tool endpoints.

Step 1: Deploy DynamoDB + Lambdas (CDK)

cd infra

# Create and activate Python virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install CDK dependencies
pip install -r requirements.txt

# Bootstrap CDK (once per account/region)
cdk bootstrap

# Preview what will be created
cdk diff

# Deploy the stack
cdk deploy

What gets created:

Resource	Count	Details
DynamoDB Tables	5	DemoAgentOpsMembers, DemoAgentOpsOrders, DemoAgentOpsOffers, DemoAgentOpsMerchants, DemoAgentOpsSupportTickets
Lambda Functions	8	One per tool (get_member_profile, get_rewards_balance, etc.)
Seed Data	~100 records	15 members, 20 orders, 15 offers, 12 merchants, 10 tickets
IAM Roles	9	One per Lambda + one for seed data custom resource

Step 2: Deploy the AgentCore Gateway

The gateway wraps the Lambda functions as MCP endpoints. Agents connect to it at runtime.

cd infra/agentcore-gateway

# Install CDK dependencies
cd agentcore/cdk && npm ci && cd ../..

# Deploy the gateway
agentcore deploy --target default -y

Verify the deployment:

# Check DynamoDB tables have data
aws dynamodb scan --table-name DemoAgentOpsMembers --select COUNT
# Expected: {"Count": 15, "ScannedCount": 15}

# Test a Lambda function directly
aws lambda invoke \
  --function-name demo-agentops-get-member-profile \
  --payload '{"member_id": "M001"}' \
  /tmp/lambda-output.json && cat /tmp/lambda-output.json

# Verify gateway is deployed
cd infra/agentcore-gateway && agentcore status --json | jq '.resources'

Phase 2a: Set Up GitHub Repository

Initialize and push

# From the project root
git init
git add .
git commit -m "feat: initial AgentOps CI/CD pipeline"

# Create GitHub repo and push
gh repo create agentcore-agentops --private --source=. --push

Configure secrets and variables

Go to: GitHub repo > Settings > Secrets and variables > Actions

Secrets (Settings > Secrets):

Secret	Value	Required
`AWS_ROLE_ARN`	`arn:aws:iam::YOUR_ACCOUNT_ID:role/GitHubActionsRole`	Yes — see OIDC setup below

Variables (Settings > Variables):

Variable	Value	Required
`AWS_ACCOUNT_ID`	Your 12-digit account ID	Yes
`AWS_REGION`	Deployment region (default: `us-east-1`)	No — has default in workflow

Note: GitHub Actions does NOT use the AGENTOPS_CI_IMAGE variable. GH jobs run on ubuntu-latest VMs with tool caching (actions/cache for AgentCore CLI, setup-uv for uv). The ECR CI image is for GitLab only.

Set up OIDC (GitHub Actions > AWS)

# 1. Create the OIDC identity provider in AWS (one-time)
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1

# 2. Create trust policy
cat > /tmp/trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:YOUR_GITHUB_ORG/YOUR_REPO:*"
        }
      }
    }
  ]
}
EOF

# 3. Create the role
aws iam create-role \
  --role-name GitHubActionsAgentOps \
  --assume-role-policy-document file:///tmp/trust-policy.json

# 4. Attach policies (use least-privilege in production)
aws iam attach-role-policy --role-name GitHubActionsAgentOps \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Note: AdministratorAccess is for demo purposes. Use scoped policies in production.

Configure environments

Go to: GitHub repo > Settings > Environments

Environment	Protection Rules	Reviewers
`dev`	None	--
`staging`	Required reviewers	Team leads
`production`	Required reviewers + wait timer	Team leads + manager

Phase 2b: Set Up GitLab Repository (Alternative)

Configure CI/CD variables

Go to: Project > Settings > CI/CD > Variables

Variable	Value	Required	Protected	Masked
`AWS_ACCOUNT_ID`	Your 12-digit account ID	Yes	No	Yes
`AWS_REGION`	Deployment region (default: `us-east-1`)	No — has default	No	No
`AWS_ROLE_ARN`	`arn:aws:iam::YOUR_ACCOUNT_ID:role/GitLabAgentOps`	Only for OIDC auth	Yes	Yes
`GITLAB_AUTH_METHOD`	`instance_profile` or `oidc`	No — defaults to `instance_profile`	No	No
`GITLAB_TOKEN`	Project access token (for MR comments)	Optional	Yes	Yes
`AGENTOPS_CI_IMAGE`	Pre-built CI image URI (see Caching section)	Optional — speeds up ~25s/job	No	No
`AGENTOPS_CI_IMAGE`	`<account>.dkr.ecr.<region>.amazonaws.com/agentops-ci:latest`	Optional — speeds up pipeline ~12 min	No	No

Instance profile auth (runners with IAM roles): Set GITLAB_AUTH_METHOD=instance_profile -- no OIDC setup needed.

OIDC auth (external runners):

# 1. Create OIDC provider (replace YOUR_GITLAB_HOST)
aws iam create-open-id-connect-provider \
  --url https://YOUR_GITLAB_HOST \
  --client-id-list https://YOUR_GITLAB_HOST

# 2. Create trust policy
cat > /tmp/gitlab-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/YOUR_GITLAB_HOST"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "YOUR_GITLAB_HOST:aud": "https://YOUR_GITLAB_HOST"
        },
        "StringLike": {
          "YOUR_GITLAB_HOST:sub": "project_path:YOUR_GROUP/YOUR_PROJECT:*"
        }
      }
    }
  ]
}
EOF

# 3. Create the role
aws iam create-role \
  --role-name GitLabAgentOps \
  --assume-role-policy-document file:///tmp/gitlab-trust-policy.json

# 4. Attach policies
aws iam attach-role-policy --role-name GitLabAgentOps \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Configure environments

Go to: Project > Operate > Environments

Environment	Required Approvals	Tier
`dev`	None	development
`staging`	1 approval	staging
`production`	2 approvals	production

GitLab rollback

Trigger via CI/CD > Pipelines > Run Pipeline, then set variables:

ROLLBACK = true
ROLLBACK_AGENT_NAME = agent name
ROLLBACK_ENVIRONMENT = dev/staging/production
ROLLBACK_RUNTIME_ID = runtime ID from agentcore status
ROLLBACK_TARGET_VERSION = (optional) specific version number

Phase 3: Create Your First Agent

chmod +x scripts/*.sh

# Launch the interactive CLI portal
./scripts/create-agent.sh

The portal opens with an ASCII art banner and walks you through 8 interactive steps with a progress bar:

Pre-built agent types: CustomerServiceBot, MerchantAnalyticsBot, InternalOpsBot, or Custom
Automatic tool selection from available Lambda functions deployed in Phase 1
Memory configuration -- choose short-term only or short-term + long-term
Auto-detection of AWS account ID and region from your active credentials
Project generation into agents/<name>/ with all AgentCore configs, Strands SDK code, and eval setup
Optional git commit + push to trigger the CI/CD pipeline immediately

The portal walks through 9 steps:

Agent name and type -- choose from pre-built personas or define custom
Tool selection -- toggle which Lambda tools to enable
Memory configuration -- short-term or long-and-short-term
Environment config -- AWS account ID, region, git branch
Review -- confirm all settings before generation
Project generation -- creates Strands SDK code + AgentCore configs
Git operations -- commits to a new agent/<name> branch
Push -- triggers the CI/CD pipeline
Next steps -- commands for monitoring and testing

Phase 4: Watch the Pipeline Run

# GitHub: watch in real-time
gh run watch

# GitHub: view in browser
gh run view --web

# GitLab: check pipeline in the web UI
# Project > Build > Pipelines

What happens at each stage:

Stage	Duration	What It Does
1. Validate	~2 min	Detects agent, runs `agentcore validate`, generates deploy plan
2. Deploy Dev	~5 min	Deploys agent to dev environment via AgentCore CLI
3. Smoke Test	~5 min	Generates 15 test prompts via Bedrock, invokes agent, waits for trace propagation
4. Evaluate	~12 min	Runs 10 evaluators in 4 parallel groups (safety, functional-core, functional-tools, quality)
5. Quality Gate	~3 min	Publishes CloudWatch metrics, checks tiered thresholds
6. Online Eval	~2 min	Verifies online evaluation is active and collecting data
7. Staging	~8 min	Manual approval, then deploys + runs lighter eval
8. Production	~8 min	Manual approval, then deploys with strictest thresholds

Note: Evaluate runs 4 parallel jobs -- critical path is ~12 min (was ~35 min sequential).

Phase 5: Promote to Staging and Production

Staging

After the quality gate passes, the staging job waits for approval.

GitHub: Actions tab > pending run > Review deployments > Approve and deploy
GitLab: Pipeline > staging job > Play button (requires Protected Environment approval)

The staging deploy:

Updates online eval sampling to 40%
Deploys to staging target
Runs safety + functional eval only
Checks staging-to-prod quality gate (higher thresholds)

Production

Same approval flow with stricter thresholds:

Safety evaluators must score >= 0.90
Functional evaluators must score >= 0.60
Quality evaluators must score >= 0.70
Composite score must be >= 0.75
Online eval sampling set to 20%

Pipeline Deep Dive

Workflow Triggers

Trigger	Branch	Behavior
`push`	`agent/**`	Full pipeline (dev > staging > prod)
`pull_request`	`main`	Full pipeline with PR/MR comments
`workflow_dispatch`	Any	Manual run, choose agent + environment

Job Dependency Chain

validate --> deploy-dev --> smoke-test --> evaluate --> metrics-and-gate --> deploy-staging --> deploy-production
                  |                                                              ^
                  +---> verify-online-eval ------------------------------------------+

Parallel Evaluation

Evaluators are split into 4 parallel groups to reduce wall-clock time from ~35 min to ~12 min:

Group	Evaluators	Est. Time
safety	Faithfulness, Harmfulness	~5 min
functional-core	Correctness, GoalSuccessRate, InstructionFollowing	~12 min
functional-tools	ToolSelectionAccuracy, ToolParameterAccuracy	~10 min
quality	Helpfulness, ResponseRelevance, Coherence	~8 min

Results are merged in the metrics-and-gate stage before quality gate checks. Groups are configurable in cicd/config/pipeline-config.json under parallel_groups.

Artifact Flow

Artifact	Produced By	Consumed By
`plan-output`	validate	(informational)
`deploy-dev-output`	deploy-dev	smoke-test, evaluate
`smoke-test-results`	smoke-test	metrics-and-gate
`eval-results`	evaluate	metrics-and-gate, post-pr-comment
`quality-gate-report`	metrics-and-gate	deploy-staging
`online-eval-status`	verify-online-eval	deploy-staging

Key Design Decisions

2-minute trace wait: AgentCore emits OTel traces asynchronously. CloudWatch Logs Insights has a short indexing delay; the pipeline waits 2 minutes for trace propagation (configurable in pipeline-config.json).
OIDC per job: Each job runs on a fresh runner. AWS credentials do not persist across jobs.
Concurrency group: A second push to the same branch cancels the in-flight run to prevent racing deploys.
Online eval rate patching: Sampling rates are modified via jq in-memory before each deploy, never committed to the repo.

GitHub vs GitLab Comparison

Feature	GitHub Actions	GitLab CI
8-stage pipeline	Yes	Yes
Auto-detect agent from changed files	Yes	Yes
OIDC AWS auth	Yes	Yes (+ instance profile)
Manual approval gates	Environment protection rules	Protected environments + `when: manual`
PR/MR eval comments	`gh pr comment`	GitLab Notes API
Rollback	Separate workflow file	Same `.gitlab-ci.yml` with variable trigger
Concurrency control	`concurrency` group	`resource_group` + `interruptible`

The post-pr-comment.sh script auto-detects the CI platform via environment variables (GITHUB_ACTIONS or GITLAB_CI). All other scripts are platform-agnostic bash.

Caching and Performance

Default (no setup needed): The pipeline works out-of-the-box with node:20-bookworm-slim + conditional tool installs + caching. After first run, cached tools restore in ~3s per job. Only apt-get (~25s/job) is uncacheable.

GitLab CI caching:

AgentCore CLI cached via NPM_CONFIG_PREFIX redirect to .npm-global/ (keyed on CLI version)
npm download cache (.npm-cache/) — keyed on package-lock.json hash
uv package cache (.cache/uv/)
Conditional installs — which tool || install (skip if already available)

GitHub Actions caching:

AgentCore CLI cached via actions/cache (keyed on version 0.7.1)
uv cached via setup-uv with enable-cache: true
CDK deps use npm ci with committed lockfile

Optional: Pre-built CI image (eliminates ~25s apt-get per job):

A Dockerfile is provided at cicd/ci/Dockerfile with all tools pre-installed. Build it and push to a registry your runners can access, then set AGENTOPS_CI_IMAGE CI variable.

Option A — GitLab Container Registry (easiest, built-in auth):

# Login to your GitLab instance registry
docker login registry.YOUR_GITLAB_HOST

# Build (use --platform linux/amd64 on Apple Silicon Macs)
docker build --platform linux/amd64 \
  --build-arg AGENTCORE_CLI_VERSION=0.7.1 \
  -t registry.YOUR_GITLAB_HOST/YOUR_GROUP/YOUR_PROJECT/ci:latest \
  -f cicd/ci/Dockerfile .

# Push
docker push registry.YOUR_GITLAB_HOST/YOUR_GROUP/YOUR_PROJECT/ci:latest

Then set: AGENTOPS_CI_IMAGE = ${CI_REGISTRY_IMAGE}/ci:latest

Option B — Amazon ECR (for self-hosted runners with ECR access):

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION=us-east-1

aws ecr create-repository --repository-name agentops-ci --region ${REGION}

docker build --platform linux/amd64 \
  --build-arg AGENTCORE_CLI_VERSION=0.7.1 \
  -t agentops-ci -f cicd/ci/Dockerfile .

aws ecr get-login-password --region ${REGION} | \
  docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com
docker tag agentops-ci ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/agentops-ci:latest
docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/agentops-ci:latest

Then set: AGENTOPS_CI_IMAGE = <account>.dkr.ecr.<region>.amazonaws.com/agentops-ci:latest

Note: The AGENTOPS_CI_IMAGE variable is GitLab-only. GitHub Actions runs on VMs and uses actions/cache instead — no Docker image needed. If not set, defaults to node:20-bookworm-slim.

Image rebuilds automatically when cicd/ci/Dockerfile changes (or BUILD_CI_IMAGE=true)

Evaluation Strategy

On-Demand Evaluation (Every Deploy)

10 built-in evaluators execute against traces generated by smoke tests:

#	Evaluator	Category	What It Measures
1	`Builtin.Faithfulness`	Safety	Sticks to facts from tools/context
2	`Builtin.Harmfulness`	Safety	Rejects harmful or inappropriate requests
3	`Builtin.Correctness`	Functional	Factual accuracy
4	`Builtin.GoalSuccessRate`	Functional	Completed the user's request
5	`Builtin.ToolSelectionAccuracy`	Functional	Picked the right tool
6	`Builtin.ToolParameterAccuracy`	Functional	Passed correct arguments to tools
7	`Builtin.InstructionFollowing`	Functional	Follows the system prompt
8	`Builtin.Helpfulness`	Quality	Useful to the user
9	`Builtin.ResponseRelevance`	Quality	On-topic
10	`Builtin.Coherence`	Quality	Well-structured

Tiered Quality Gates

Scores are normalized to 0-1. Different categories have different thresholds because safety metrics are more reliably judged than quality metrics (which have LLM judge variance of +/-0.05-0.10).

Category	Dev to Staging	Staging to Prod
Safety	>= 0.85	>= 0.90
Functional	>= 0.55	>= 0.60
Quality	>= 0.70	>= 0.70
Composite	>= 0.70	>= 0.75

Online Evaluation (Continuous)

Runs automatically after deployment at configured sampling rates:

Environment	Sampling Rate	Purpose
Dev	100%	Catch all issues during development
Staging	40%	Good coverage at manageable cost
Production	20%	Trend detection, anomaly alerting

Monitor online eval:

agentcore logs evals --agent <agent-name> --since 1h --json

CloudWatch Metrics

Published to namespace AgentCore/AgentOps with dimensions AgentName, Environment, and EvaluatorName:

Metric	Unit	Description
`EvalScore`	None (0-1)	Normalized evaluator score
`InputTokens`	Count	Tokens consumed by evaluation
`OutputTokens`	Count	Tokens generated by evaluation
`TotalTokens`	Count	Total tokens used
`EstimatedCostUSD`	None ($)	Cost estimate based on model pricing

Account and Region Placeholders

Template files use __AWS_ACCOUNT_ID__ and __AWS_REGION__ as placeholders instead of hardcoded values. Substitution happens automatically:

In CI: The validate stage replaces placeholders using aws sts get-caller-identity and the AWS_REGION variable
Locally: The scripts/create-agent.sh portal performs the same substitution interactively

Files containing placeholders:

*/agentcore/agentcore.json -- Lambda ARNs in gateway targets
*/agentcore/aws-targets.json -- deployment target account and region
templates/agentcore.json -- template source
templates/aws-targets.json -- template source

Note on deployed-state.json: This file is auto-generated by the AgentCore CLI on first deploy. It contains deployment-specific resource IDs unique to each account. The CI pipeline treats it as a build artifact, not a source-controlled file. Do not rely on a checked-in copy.

Configuration Reference

cicd/config/eval-thresholds.json

Tiered quality gate thresholds. Edit to adjust strictness:

{
  "dev_to_staging": {
    "safety": {
      "evaluators": ["Builtin.Faithfulness", "Builtin.Harmfulness"],
      "min_score": 0.85
    },
    "functional": {
      "evaluators": ["Builtin.Correctness", "Builtin.GoalSuccessRate", "..."],
      "min_score": 0.65
    },
    "quality": {
      "evaluators": ["Builtin.Helpfulness", "Builtin.ResponseRelevance", "Builtin.Coherence"],
      "min_score": 0.70
    },
    "composite_min": 0.75
  }
}

cicd/config/bedrock-pricing.json

Token pricing for cost estimation:

{
  "models": {
    "claude-sonnet-4-6": {"input_per_1m": 3.00, "output_per_1m": 15.00},
    "claude-sonnet-4-5": {"input_per_1m": 5.00, "output_per_1m": 25.00}
  }
}

cicd/config/pipeline-config.json

Central pipeline behavior:

{
  "smoke_test": {"num_prompts": 15, "timeout_seconds": 120, "trace_wait_minutes": 2},
  "evaluation": {
    "builtin_evaluators": ["Builtin.Faithfulness", "Builtin.Harmfulness", "Builtin.Correctness", "..."],
    "staging_evaluators": ["Builtin.Faithfulness", "Builtin.Harmfulness", "Builtin.Correctness", "..."],
    "parallel_groups": {
      "safety": ["Builtin.Faithfulness", "Builtin.Harmfulness"],
      "functional-core": ["Builtin.Correctness", "Builtin.GoalSuccessRate", "Builtin.InstructionFollowing"],
      "functional-tools": ["Builtin.ToolSelectionAccuracy", "Builtin.ToolParameterAccuracy"],
      "quality": ["Builtin.Helpfulness", "Builtin.ResponseRelevance", "Builtin.Coherence"]
    }
  },
  "online_eval": {"dev_sampling_rate": 100, "staging_sampling_rate": 40, "production_sampling_rate": 20},
  "agentcore_cli_version": "0.7.1",
  "ci_image": ""
}

Field	CI Variable Override	Description
`agentcore_cli_version`	`AGENTCORE_CLI_VERSION`	AgentCore CLI version to install in CI jobs (default `0.7.1`)
`ci_image`	`AGENTOPS_CI_IMAGE`	Pre-built ECR image with all tools pre-installed (empty = install from scratch)

Project Structure

agentcore-agentops/
├── .github/workflows/              # GitHub Actions pipelines
│   ├── agent-cicd.yml              # Main 8-stage pipeline
│   └── rollback.yml                # Manual rollback workflow
├── .gitlab-ci.yml                  # GitLab CI pipeline (full + rollback)
│
├── scripts/                        # User-facing scripts + symlinks
│   ├── create-agent.sh             # Interactive agent creator
│   ├── run-smoke-tests.sh ->       # Symlink to cicd/scripts/
│   ├── run-evaluations.sh ->       # Symlink to cicd/scripts/
│   └── generate-test-prompts.sh -> # Symlink to cicd/scripts/
│
├── cicd/                           # CI/CD pipeline infrastructure
│   ├── scripts/                    # Pipeline scripts
│   │   ├── run-smoke-tests.sh      # Invoke agent with test prompts
│   │   ├── run-evaluations.sh      # Run all evaluators
│   │   ├── generate-test-prompts.sh # Auto-gen prompts via Bedrock
│   │   ├── check-quality-gate.sh   # Tiered threshold checking
│   │   ├── publish-metrics.sh      # Publish to CloudWatch + cost calc
│   │   ├── verify-online-eval.sh   # Confirm online eval is active
│   │   ├── post-pr-comment.sh      # Post eval summary as PR/MR comment
│   │   ├── rollback-agent.sh       # Fast endpoint version rollback
│   │   └── generate-ci-report.sh   # CI run summary report
│   ├── config/                     # Eval thresholds, pipeline config
│   │   ├── eval-thresholds.json
│   │   ├── bedrock-pricing.json
│   │   └── pipeline-config.json
│   └── ci/                         # CI runner Dockerfile
│       └── Dockerfile
│
├── infra/                          # Shared infrastructure (CDK)
│   ├── lambdas/                    # 8 Lambda tool handlers
│   │   ├── get_member_profile/     # Each has handler.py + tool-schema.json
│   │   ├── get_rewards_balance/
│   │   ├── search_cashback_offers/
│   │   ├── get_order_history/
│   │   ├── submit_support_ticket/
│   │   ├── get_merchant_analytics/
│   │   ├── update_merchant_listing/
│   │   └── generate_campaign_report/
│   ├── agentcore-gateway/          # AgentCore Gateway (MCP endpoints)
│   ├── seed_lambda/                # Custom resource that seeds DynamoDB
│   ├── seed-data/                  # Mock e-commerce data (JSON)
│   ├── tools_stack.py              # CDK stack: DynamoDB + Lambdas + seed
│   ├── app.py                      # CDK app entry point
│   ├── cdk.json                    # CDK config
│   └── requirements.txt            # Python CDK dependencies
│
├── templates/                      # Agent scaffolding templates
│   ├── agentcore.json              # AgentCore project spec (with placeholders)
│   ├── mcp.json                    # Gateway + Lambda targets config
│   ├── aws-targets.json            # Multi-env deployment targets
│   ├── test-prompts.json           # Fallback test prompts per agent type
│   └── version-history.json        # Deployment version tracking
│
├── agents/                         # Agent projects
│   └── ECommerceBot/               # Demo e-commerce agent
│       ├── agentcore/              # AgentCore config + CDK
│       │   ├── agentcore.json
│       │   ├── aws-targets.json
│       │   ├── cdk/
│       │   └── tool-schemas/
│       └── app/ECommerceBot/       # Agent source (Strands SDK)
│
├── LICENSE                         # Apache 2.0
└── README.md                       # This file

Rollback Procedures

Fast Rollback (Bad Agent Code)

AgentCore versions every deployment. Rollback switches the endpoint to a previous version -- instant, no CDK redeploy.

# Option 1: GitHub Actions workflow
gh workflow run rollback.yml \
  -f agent_name=<agent-name> \
  -f environment=dev \
  -f runtime_id=<runtime-id>

# Option 2: Direct script
RUNTIME_ID=<runtime-id> \
REGION=<your-region> \
./cicd/scripts/rollback-agent.sh

# Option 3: AWS CLI directly
aws bedrock-agentcore update-agent-runtime-endpoint \
  --agent-runtime-id <runtime-id> \
  --endpoint-name DEFAULT \
  --agent-runtime-version <previous-version>

Find the Runtime ID:

cd agents/<agent-name>
agentcore status --json | jq '.resources[] | select(.resourceType == "agent") | .runtimeId'

Full Rollback (Bad Infrastructure)

When you need to roll back memory, gateway, or evaluator changes:

git revert HEAD
git push
# Triggers the pipeline again, deploying the previous config

Troubleshooting

"No session spans found" during evaluation

Cause: Traces have not propagated to CloudWatch yet.

Fix: The pipeline waits 10 minutes after smoke tests. If running manually, wait and retry:

sleep 600
agentcore run eval --agent <agent-name> --evaluator Builtin.Faithfulness --days 1

CDK bootstrap fails

# Check existing bootstrap
aws cloudformation describe-stacks --stack-name CDKToolkit

# Force re-bootstrap
cdk bootstrap --force

"Target not found in aws-targets.json"

Verify target names match exactly:

cat agentcore/aws-targets.json | jq '.[].name'
# Should output: "dev", "staging", "production"

Lambda function returns 500

# Check Lambda environment variables
aws lambda get-function-configuration \
  --function-name demo-agentops-get-member-profile \
  --query 'Environment.Variables'

# Check Lambda logs
aws logs tail /aws/lambda/demo-agentops-get-member-profile --since 1h

Quality gate fails unexpectedly

# View detailed eval results
cat eval-results.json | jq '.results[] | {evaluator, aggregateScore}'

# Run specific evaluator for debugging
agentcore run eval --agent <agent-name> \
  --evaluator Builtin.Faithfulness --days 1 --json

OIDC authentication fails

Verify the trust policy condition matches your repo/project:

# GitHub
aws iam get-role --role-name GitHubActionsAgentOps \
  --query 'Role.AssumeRolePolicyDocument'
# The "sub" condition must match: repo:YOUR_GITHUB_ORG/YOUR_REPO:*

# GitLab
aws iam get-role --role-name GitLabAgentOps \
  --query 'Role.AssumeRolePolicyDocument'
# The "sub" condition must match: project_path:YOUR_GROUP/YOUR_PROJECT:*

Useful Debug Commands

# Agent deployment status
agentcore status --json | jq .

# List runtime versions (for rollback)
aws bedrock-agentcore list-agent-runtime-versions \
  --agent-runtime-id <runtime-id>

# View recent traces
agentcore traces list --agent <agent-name> --limit 10

# Stream agent logs
agentcore logs --agent <agent-name> --since 30m

# Stream online eval logs
agentcore logs evals --agent <agent-name> --since 30m

# Check CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AgentCore/AgentOps \
  --metric-name EvalScore \
  --dimensions Name=AgentName,Value=<agent-name> Name=Environment,Value=dev \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 --statistics Average

Cost Estimation

Per-Agent (Dev Environment, Monthly)

Component	Cost Estimate
AgentCore Runtime (moderate use)	~$15
DynamoDB (on-demand)	< $1
Lambda invocations	< $1
Eval runs (daily CI/CD, Sonnet 4.5)	~$3
Online eval (100% sampling, dev)	~$5-10
CloudWatch metrics	< $5
S3 (CDK assets)	< $1
Total (dev)	~$25-35

Per-Pipeline Run

Step	Tokens (approx)	Cost
Generate test prompts (Sonnet 4.6)	~2K in / 1K out	$0.02
Smoke tests (15 invocations, Sonnet 4.6)	~15K in / 10K out	$0.18
On-demand eval (10 evaluators, Sonnet 4.5)	~10K in / 7K out	$0.14
Total per run		~$0.34

Technology Stack

Component	Technology
Agent Framework	Strands Agents SDK
Agent Model	Claude Sonnet 4.6
Evaluator Model	Claude Sonnet 4.5
Infrastructure	AWS CDK (Python)
Agent Deployment	AgentCore CLI (`@aws/agentcore`)
CI/CD	GitHub Actions + GitLab CI
Tool Gateway	AgentCore Gateway (semantic search)
Data Store	Amazon DynamoDB (on-demand)
Compute (Tools)	AWS Lambda (Python 3.13)
Metrics	Amazon CloudWatch
Tracing	AWS X-Ray + OpenTelemetry

License

Apache 2.0 -- see LICENSE for details.

Name		Name	Last commit message	Last commit date
Latest commit History 139 Commits
.github/workflows		.github/workflows
agents		agents
cicd		cicd
docs		docs
examples		examples
infra		infra
scripts		scripts
templates		templates
.dockerignore		.dockerignore
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
LICENSE		LICENSE
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

AgentOps CI/CD Pipeline for Amazon Bedrock AgentCore

Table of Contents

Architecture Overview

How It Works

Prerequisites

Required Software

AWS Configuration

Bedrock Model Access

Step-by-Step Deployment

Phase 1: Deploy Shared Infrastructure

Phase 2a: Set Up GitHub Repository

Initialize and push

Configure secrets and variables

Set up OIDC (GitHub Actions > AWS)

Configure environments

Phase 2b: Set Up GitLab Repository (Alternative)

Configure CI/CD variables

Configure environments

GitLab rollback

Phase 3: Create Your First Agent

Phase 4: Watch the Pipeline Run

Phase 5: Promote to Staging and Production

Staging

Production

Pipeline Deep Dive

Workflow Triggers

Job Dependency Chain

Parallel Evaluation

Artifact Flow

Key Design Decisions

GitHub vs GitLab Comparison

Caching and Performance

Evaluation Strategy

On-Demand Evaluation (Every Deploy)

Tiered Quality Gates

Online Evaluation (Continuous)

CloudWatch Metrics

Account and Region Placeholders

Configuration Reference

cicd/config/eval-thresholds.json

cicd/config/bedrock-pricing.json

cicd/config/pipeline-config.json

Project Structure

Rollback Procedures

Fast Rollback (Bad Agent Code)

Full Rollback (Bad Infrastructure)

Troubleshooting

"No session spans found" during evaluation

CDK bootstrap fails

"Target not found in aws-targets.json"

Lambda function returns 500

Quality gate fails unexpectedly

OIDC authentication fails

Useful Debug Commands

Cost Estimation

Per-Agent (Dev Environment, Monthly)

Per-Pipeline Run

Technology Stack

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages