⚡ Serverless Async Orchestration Blueprint

A production-grade reference implementation of long-running, parallel, pollable job orchestration on AWS — extracted from a real event-streaming platform serving live audiences.

Important

This is a public, production-ready architectural skeleton. All company-specific business logic has been deliberately omitted for confidentiality. Data-store queries, warehouse SQL, domain metrics, and proprietary algorithms are replaced with clearly marked mocks (// TODO: integrate your data store). The orchestration mechanics, infrastructure-as-code, and reusable libraries are real, typed, tested, and deployable.

---

TL;DR — what is this repo?

This is a runnable blueprint for one hard problem: running jobs that are too long for a normal HTTP request, pull from many data sources at once, and move payloads too large to pass through a state machine — while still being retryable, observable, and pollable.

Think "generate a comprehensive multi-source analytics export": the client fires one request, walks away, polls for status, and downloads the results when ready. This repo is the reusable core of that whole workflow, distilled from a system I built and run in production.

It is intentionally one pattern, done properly — not a kitchen-sink demo.

"Is this just a Step Functions demo?" — no.

A demo wires three states together and calls it a day. This blueprint solves the things that actually bite you in production:

Real-world problem	How this blueprint solves it
Step Functions caps state at 256 KB per transition	S3 "workspace pointer" pattern — tasks write JSON artifacts to S3 and pass back only a tiny { s3Key, count } pointer. State stays microscopic; payloads stay unbounded.
Jobs run for minutes, past API Gateway's 29s limit	Async trigger + poll API — POST /jobs returns 202 instantly; clients poll GET /jobs/{id}. No long-lived connections.
Sources are independent and slow in series	Parallel Parallel branches + bounded Map fan-out — N items gathered concurrently with MaxConcurrency backpressure.
External analytical queries take seconds–minutes	start → wait → poll → choice → hydrate async loop (e.g. Athena) that never holds a Lambda open.
Expensive per-record lookups	Entity-dictionary join — build an id → entity map once, hydrate everything else with O(1) lookups.
Security review	Least-privilege IAM, scoped to this stack's own bucket/table/state-machine — zero wildcard ARNs.
Cost & cold starts	arm64 + esbuild bundling + pay-per-request DynamoDB + S3 lifecycle auto-expiry.

This is the difference between "I've seen Step Functions" and "I've operated durable orchestration at scale."

---

🗺️ The bigger picture — the platform this came from

This repo is one extracted slice of a larger serverless live-event / streaming platform I architected end-to-end. That platform spans real-time interaction, a telemetry data lake, an AI insights layer, and live video — all serverless on AWS. This blueprint showcases the orchestration slice in full; the others are described here at the architecture level (their code stays private), so you can see the breadth behind the depth.

flowchart LR
  subgraph platform ["Serverless live-event platform (architected end-to-end)"]
    direction TB
    rt["Real-time API<br/>AppSync GraphQL · DynamoDB · Valkey/Redis"]
    orch["⚡ Async Orchestration<br/>Step Functions · S3 workspace<br/>(THIS REPO)"]
    tel["Telemetry Data Lake<br/>Kinesis Firehose · S3 · Glue · Athena"]
    ai["AI Insights / RAG<br/>Amazon Bedrock · retrieval + prompt layer"]
    video["Live Video<br/>Amazon IVS (real-time + low-latency)"]
  end
  rt -->|"events"| tel
  tel -->|"queried by"| orch
  orch -->|"feeds reports / insights"| ai
  rt -. presence/state .-> video

Loading

Domain	Stack	What it does	In this repo?
Async orchestration	Step Functions, Lambda, S3, DynamoDB	Long-running, parallel, pollable multi-source jobs	✅ Full implementation
Real-time API	AppSync (GraphQL), DynamoDB, Valkey/Redis	Live subscriptions, presence, multi-auth modes, O(1) caching	🔒 Architecture only
Telemetry data lake	Kinesis Firehose → S3 → Glue → Athena	High-volume event ingestion, partitioned lake, ad-hoc SQL analytics	🔒 Architecture only
AI insights / RAG	Amazon Bedrock, retrieval + prompt orchestration	Retrieval-augmented summaries & insights over event data	🔒 Architecture only
Live video	Amazon IVS (real-time stages + low-latency channels)	Interactive and broadcast live video	🔒 Architecture only

The orchestration pattern in this repo is exactly what stitches several of those domains together: it queries the telemetry lake (Athena), joins it against real-time data stores, and produces artifacts an AI/RAG layer can consume — which is why it's the most representative single slice to open-source.

---

🏗️ Architecture (this repo)

flowchart TB
  client["Client"] -->|"POST /jobs (202)"| trigger["triggerJob"]
  trigger -->|"StartExecution"| sfn["Step Functions"]
  trigger --> jobs[("DynamoDB jobs")]
  client -->|"GET /jobs/{id}"| status["jobStatus"]
  status --> sfn
  status -.->|"presigned URLs"| s3[("S3 workspace")]

  subgraph sfn ["Orchestration state machine"]
    validate["ValidateAndInit"] --> dict["BuildEntityDict"]
    dict --> gather["GatherData (Parallel)"]
    gather --> warehouse["A: start → wait → poll → hydrate"]
    gather --> mapfan["B: Map fan-out over items"]
    warehouse --> agg["AggregateResults"]
    mapfan --> agg
  end

  validate -.->|"pointer only"| s3
  gather -.-> s3
  agg -.-> s3

Loading

The S3 workspace pointer pattern (the core idea)

sequenceDiagram
  participant T1 as Task (writer)
  participant S3 as S3 workspace
  participant SFN as Step Functions state
  participant T2 as Task (reader)
  T1->>S3: PutObject(workspace/<resource>/<run>/artifact.json)
  T1->>SFN: return { status, s3Key, count }   %% tiny pointer
  SFN->>T2: pass pointer (well under 256 KB)
  T2->>S3: GetObject(s3Key)
  T2->>T2: process full payload in memory

Loading

Each run gets a unique prefix — workspace/orchestration/<resourceId>/<runId> — so concurrent runs never collide, and a single S3 lifecycle rule expires every artifact automatically.

Deeper design notes and sequence diagrams live in docs/architecture.md.

---

🧰 Tech stack

• Orchestration: AWS Step Functions (Parallel, Map, Choice, retries/backoff)
• Compute: AWS Lambda (Node.js 22, arm64, esbuild bundling, X-Ray tracing)
• Storage: S3 (workspace artifacts + lifecycle), DynamoDB (job status, TTL)
• API: API Gateway HTTP API (async trigger + poll)
• IaC: Serverless Framework v4 + modular CloudFormation resources
• Language/tests: TypeScript (strict), Vitest

📁 Project structure

serverless-orchestration-blueprint/
├── serverless.yml                 # Service, functions, state machine wiring
├── resources/
│   ├── iam.yml                    # Least-privilege role statements
│   ├── s3-workspace.yml           # Workspace bucket + 30-day lifecycle
│   ├── dynamodb-jobs.yml          # Job-status table (PK jobId, TTL)
│   └── stepfunction/
│       └── orchestration.asl.yml  # The state machine (Amazon States Language)
├── src/
│   ├── handlers/
│   │   ├── http/                  # trigger-job, job-status (async control plane)
│   │   └── steps/                 # validate, build-dict, gather/*, aggregate
│   └── lib/                       # workspace, job-store, sfn-client, types (REAL, reusable)
├── docs/architecture.md
└── tests/workspace.test.ts

The src/lib/* modules are the genuinely reusable parts — kept real and clean. The src/handlers/steps/* files are skeletons with mock data and marked // TODO integration points, so the pipeline runs end-to-end with zero private dependencies.

---

🚀 Deploy & run

Prerequisites: Node.js 22+, AWS credentials configured (aws configure).

npm install                        # install dependencies
cp .env.example .env               # set APP_NAME / AWS_REGION
npm run typecheck && npm test      # verify
npx serverless deploy --stage dev  # deploy

Trigger a job and poll for results:

# Fire it (returns 202 with a jobId)
curl -X POST "$API_BASE_URL/jobs" \
  -H "Content-Type: application/json" \
  -d '{ "resourceId": "demo-123" }'

# Poll until SUCCEEDED — the response then includes presigned download URLs
curl "$API_BASE_URL/jobs/<jobId>"

Teardown: npx serverless remove --stage dev

---

🔒 What was intentionally omitted

To keep this public and confidentiality-safe, the following were stripped or mocked:

• Real data-store access — database queries and warehouse SQL → in-memory mock generators.
• Domain metric/aggregation logic — AggregateResults only tallies counts.
• Proprietary algorithms — e.g. geo/IP remapping and AI prompt templates.
• All infrastructure identifiers — account IDs, ARNs, real bucket/table/stream names, custom domains, VPC subnet/security-group IDs, and cross-account deploy roles. Everything derives from a single APP_NAME + stage or from .env.

---

Built by Bigyat Uladhar — serverless & cloud architecture, real-time systems, data pipelines, and applied AI.

Want a walkthrough of the real-time GraphQL, telemetry-lake, or RAG layers? I'm happy to talk through the architecture.

MIT License