NewsPilot-AI

An autonomous, multi-agent newsletter engine with a Human-in-the-Loop (HITL) control plane — built on LangGraph, Amazon Bedrock, Pinecone Serverless, and AWS RDS.

Overview

NewsPilot-AI is a production-grade agentic system that autonomously crawls live news, drafts a grounded newsletter using Claude 3 Haiku on Amazon Bedrock, evaluates it with a DeepEval faithfulness gate, and routes it to a human operator for final approval before dispatch.

The system is designed for reliability and auditability:

No LLM hallucinations ship — the Auditor enforces a faithfulness_score ≥ 0.8 threshold before a draft exits the pipeline
No blocking HTTP calls — the /run endpoint returns a thread_id immediately; agents execute asynchronously as a BackgroundTask
No stale deduplication — Pinecone Serverless prevents the same story from appearing in consecutive newsletters via cosine similarity filtering
Full human oversight — the HITL /feedback endpoint lets an operator approve, reject, or add rewrite instructions at the LangGraph checkpoint

System Architecture

1. High-Level Cloud Architecture

flowchart TB
    subgraph Client["Frontend Layer (Vite + React)"]
        UI[React Dashboard]
        Tracker[Execution Tracker]
        HITL[HITL Review Workspace]
    end

    subgraph API["Backend API Layer (FastAPI)"]
        Server[FastAPI Server]
        Dispatch[SMTP Dispatcher]
    end

    subgraph Core["Agentic Orchestrator (LangGraph)"]
        Graph[LangGraph State Machine]
        Memory[(MemorySaver Checkpointer)]
    end

    subgraph Cloud["AWS Cloud Infrastructure"]
        Pinecone[(Pinecone Vector DB)]
        RDS[(AWS RDS PostgreSQL)]
        Bedrock[Amazon Bedrock\nClaude 3 Haiku + Titan Embed v2]
        Gmail[Gmail SMTP Service]
    end

    UI -->|POST /run| Server
    Server -->|Generate thread_id + BackgroundTask| Graph
    Graph -->|Embeddings & Deduplication| Pinecone
    Graph -->|Inference & Structured Outputs| Bedrock
    Graph -->|Persist State| Memory

    HITL -->|POST /feedback| Server
    Server -->|Resume LangGraph State| Graph
    Graph -->|Log Metrics & Fetch Recipients| RDS
    Dispatch -->|Send Newsletter| Gmail

    Tracker -.->|GET /status| Server

2. LangGraph State Machine & Self-Correction Flow

stateDiagram-v2
    [*] --> Researcher: Trigger Pipeline

    state Researcher {
        direction TB
        Scrape: Crawl Web Sources (Crawl4AI + DuckDuckGo)
        Embed: Generate Embeddings (Titan Embed Text v2)
        CheckVec: Query Pinecone (Cosine Similarity)
        Filter: "Prune Similarities >= 0.85"
        Scrape --> Embed
        Embed --> CheckVec
        CheckVec --> Filter
    }

    Researcher --> Writer: Pass Filtered Articles

    state Writer {
        direction TB
        Draft: Synthesize Markdown Newsletter
        Ground: Extract Claim-Evidence Mappings
        Draft --> Ground
    }

    Writer --> Auditor: "State Update (draft_markdown, citations)"

    state Auditor {
        direction TB
        Eval: Run DeepEval FaithfulnessMetric
        Score: "Compute Score (0.0 to 1.0)"
        LogDB: Write Metrics to AWS RDS
        Eval --> Score
        Score --> LogDB
    }

    Auditor --> Writer: "Faithfulness < 0.8 & Revisions < 2 (Self-Correction Loop)"
    Auditor --> HumanReview: "Faithfulness >= 0.8 (HITL Checkpoint)"

    state HumanReview {
        direction TB
        Inspect: "Inspect Metrics & Citations"
        Decision: "Operator Decision (React Dashboard)"
        Inspect --> Decision
    }

    HumanReview --> Writer: Reject / Provide Feedback
    HumanReview --> Dispatcher: Approve and Send
    Dispatcher --> [*]: Delivered via Gmail SMTP

📬 Newsletter Delivery — Live Demo

The video below shows a real generated newsletter as received in Gmail.

generated_newsletter_mail_view.mp4

3. Asynchronous HITL Sequence

sequenceDiagram
    autonumber
    actor Operator as Operator (React Dashboard)
    participant API as FastAPI Backend
    participant Graph as LangGraph Engine
    participant Judge as DeepEval + Bedrock Claude
    participant RDS as AWS RDS PostgreSQL

    Operator->>API: POST /run {topic, list_name}
    API->>Graph: Initialize BackgroundTask
    API-->>Operator: 202 Accepted {thread_id}

    loop Real-Time Status Polling
        Operator->>API: GET /status
        API-->>Operator: {status: 'running', active_step: 'researcher'}
    end

    Graph->>Judge: FaithfulnessMetric(draft, context)
    Judge-->>Graph: Score: 0.92
    Graph->>RDS: INSERT INTO newsletter_metrics (...)

    Graph-->>API: Execution Paused at HITL Checkpoint
    API-->>Operator: {status: 'review', draft_markdown, metrics, citations}

    Operator->>API: POST /feedback {thread_id, is_approved: true}
    API->>Graph: Resume LangGraph State Machine
    Graph->>RDS: SELECT emails FROM distribution_lists WHERE list_name = ...
    Graph->>API: Trigger Gmail SMTP Client
    API-->>Operator: 200 OK {status: 'dispatched'}

4. Vector Deduplication & Semantic Filtering

flowchart LR
    A[Raw Web Articles] --> B[Generate 1536-d Embeddings\nTitan Embed Text v2]
    B --> C[(Pinecone Serverless\nIndex: newsletter)]
    C --> D{Cosine Similarity\n>= 0.85?}
    D -- Yes --> E[Drop as Redundant Story]
    D -- No --> F[Upsert Vector to Pinecone]
    F --> G[Pass to Writer Agent Payload]

Tech Stack

Layer	Technology
Agent Orchestration	LangGraph `StateGraph` + `MemorySaver` checkpointing
LLM / Writer	Claude 3 Haiku via Amazon Bedrock (`langchain-aws`)
Embeddings	Amazon Titan Embed Text v2 via Amazon Bedrock
Evaluation Judge	DeepEval `FaithfulnessMetric` + `BedrockModel`
LLMOps	`PromptRegistry` (versioning, rollback, diff) + `ExecutionLogger` (JSONL + PostgreSQL)
Vector Store	Pinecone Serverless (cosine similarity, 1536-d)
Relational DB	PostgreSQL on AWS RDS (SQLAlchemy ORM)
Web Crawling	Crawl4AI + DuckDuckGo search + BeautifulSoup4 fallback
API	FastAPI with `BackgroundTasks`
Frontend	React 18 + Vite + Tailwind CSS (dark mode)
Email Dispatch	Gmail API via OAuth 2.0

Project Structure

newsletter--ai/
│
├── agents/
│   ├── newsletter_agents.py    # 3-agent LangGraph workflow (Researcher, Writer, Auditor)
│   ├── llm_client.py           # OpenRouter client with retry logic
│   └── state.py                # AgentState TypedDict + Pydantic schemas
│
├── api/
│   └── fastapi_server.py       # FastAPI: /run, /feedback, /lists, /status, /llmops/* endpoints
│
├── crawlers/
│   └── web_crawler.py          # Crawl4AI + DuckDuckGo + BeautifulSoup fallback
│
├── dashboard/                  # React + Vite + Tailwind MLOps dashboard
│   ├── src/
│   │   ├── App.jsx
│   │   └── components/
│   │       ├── RunConfigurator.jsx
│   │       ├── ExecutionTracker.jsx
│   │       ├── ObservabilityMetrics.jsx
│   │       ├── ReviewWorkspace.jsx
│   │       └── ActionConsole.jsx
│   ├── package.json
│   └── vite.config.js
│
├── email_service/
│   └── gmail_client.py         # Gmail API OAuth 2.0 + HTML formatter
│
├── evaluation/
│   ├── evaluation_cases.json   # 5 manual faithfulness benchmark cases
│   └── run_evals.py            # DeepEval regression runner
│
├── llmops/                     # ⭐ LLMOps: Prompt Versioning & Execution Logging
│   ├── __init__.py             # Re-exports PromptRegistry, ExecutionLogger
│   ├── prompt_registry.py      # Version-controlled prompt template management
│   ├── execution_logger.py     # Structured LLM call logging (JSONL + PostgreSQL)
│   └── prompt_versions.json    # Versioned prompt store (auto-managed)
│
├── utils/
│   ├── vector_store.py         # Pinecone Serverless client + deduplication logic
│   ├── relational_db.py        # SQLAlchemy: NewsletterMetricLog, DistributionList, LLMExecutionLog
│   └── logger.py
│
├── scheduler/
│   └── newsletter_scheduler.py
│
├── config.py                   # Env-driven config singleton
├── main.py
├── requirements.txt
├── global-bundle.pem           # AWS RDS SSL certificate
└── .env

Setup & Installation

Prerequisites

Python 3.11+
AWS account with Bedrock model access enabled (claude-3-haiku, titan-embed-text-v2)
Pinecone account (Serverless index named newsletter, dimension 1536, metric cosine)
AWS RDS PostgreSQL instance (or local Postgres for development)
Gmail API credentials (credentials.json)
Node.js 18+ (for the React dashboard)

1. Clone & Create Virtual Environment

git clone <repository-url>
cd newsletter--ai

python -m venv newsletter
# Windows
.\newsletter\Scripts\activate
# macOS / Linux
source newsletter/bin/activate

pip install -r requirements.txt

2. Configure Environment Variables

cp .env.example .env

Edit .env:

# AWS Bedrock (Claude 3 Haiku + Titan Embed)
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_DEFAULT_REGION=us-east-1

# Pinecone Serverless
PINECONE_API_KEY=your_pinecone_key

# AWS RDS PostgreSQL (with SSL)
DATABASE_URL=postgresql://postgres:YOUR_PASSWORD@newspilot-db.xxxx.us-east-1.rds.amazonaws.com:5432/newspilot?sslmode=verify-full&sslrootcert=global-bundle.pem

# Gmail OAuth
GMAIL_CREDENTIALS_PATH=credentials.json
GMAIL_TOKEN_PATH=token.json
SENDER_EMAIL=your_email@gmail.com
RECIPIENT_EMAILS=recipient1@example.com,recipient2@example.com

# Newsletter Config
NEWSLETTER_TITLE=NewsPilot-AI Daily
SCHEDULE_TIMES=08:00,18:00
TIMEZONE=Asia/Kolkata
MAX_ARTICLES_PER_TOPIC=5
CRAWL_TIMEOUT=30

3. AWS RDS SSL Certificate

curl -o global-bundle.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

4. Gmail API Setup

Go to Google Cloud Console
Enable the Gmail API
Create OAuth 2.0 Client ID credentials (Desktop App type)
Download as credentials.json to project root
On first run, a browser window will open to authorize the app — this generates token.json

5. Start the Backend API

python -m uvicorn api.fastapi_server:app --host 0.0.0.0 --port 8000 --reload

API Endpoints:

Method	Endpoint	Description
`GET`	`/health`	Health check
`GET`	`/status`	Pipeline status + configuration
`GET`	`/lists`	Fetch all distribution lists from DB
`POST`	`/run`	Dispatch agent pipeline (async) → returns `thread_id`
`POST`	`/feedback`	Submit HITL decision to resume LangGraph
`POST`	`/generate`	Legacy: trigger newsletter generation
`GET`	`/history`	Last 10 newsletter runs
`GET`	`/config`	Runtime configuration
`GET`	`/llmops/prompts`	List all registered prompts with active versions
`GET`	`/llmops/prompts/{name}`	Get a prompt's active version and full history
`POST`	`/llmops/prompts/{name}`	Register a new version of a prompt
`POST`	`/llmops/prompts/{name}/rollback`	Rollback to a specific prompt version
`POST`	`/llmops/prompts/{name}/diff`	Unified diff between two prompt versions
`GET`	`/llmops/executions`	Query execution logs (filterable by agent, model)
`GET`	`/llmops/executions/stats`	Aggregated stats: total calls, avg latency, cost

POST /run payload:

{ "topic": "AI model releases this week", "list_name": "engineering_team" }

POST /feedback payload:

{ "thread_id": "uuid", "is_approved": true, "human_feedback": "Optional rewrite instruction" }

6. Start the React Dashboard

cd dashboard
npm install
npm run dev

Open http://localhost:5173 — the Vite dev server proxies /api/* calls to FastAPI on port 8000.

Agent Workflow Details

Researcher Agent

Queries DuckDuckGo for live URLs based on the operator's topic
Crawls pages asynchronously with Crawl4AI (headless Chromium)
Falls back to httpx + BeautifulSoup if Crawl4AI fails
Generates embeddings via Amazon Titan Embed Text v2 and queries Pinecone for cosine similarity ≥ 0.85 duplicates
Only novel articles proceed to the Writer

Writer Agent

Receives filtered_articles (de-duplicated)
Calls Claude 3 Haiku on Bedrock with a grounded prompt — strictly no external claims
Returns structured WriterOutput (Pydantic): draft_markdown + citations (claim → source URL mappings)
Incorporates human_feedback from the HITL loop on revision passes

Auditor Agent

Runs DeepEval FaithfulnessMetric using Claude 3 Haiku as the LLM judge (via BedrockModel)
Gate: faithfulness_score >= 0.8 AND len(citations) > 0 → approved
On failure: increments revision_count, sends state back to Writer (max 2 retries)
On approval: triggers Gmail dispatch and logs to AWS RDS newsletter_metrics

Observability & Evaluation

Live Metrics (per run)

Metric	Source
Articles Scraped	Researcher agent
Duplicates Pruned	Pinecone cosine similarity filter
Faithfulness Score	DeepEval FaithfulnessMetric
Execution Latency	Tracked in React dashboard

Offline Regression Testing

# Run the 5-case faithfulness benchmark
python evaluation/run_evals.py

Benchmark cases are stored in evaluation/evaluation_cases.json.

LLMOps — Prompt Versioning & Execution Logging

NewsPilot-AI includes a built-in LLMOps layer for production prompt management and LLM call observability.

Architecture

flowchart TB
    subgraph LLMOps["LLMOps Layer"]
        PR["PromptRegistry\n(prompt_versions.json)"]
        EL["ExecutionLogger\n(JSONL + PostgreSQL)"]
    end

    subgraph Agents["Agent Workflow"]
        Writer[Writer Agent]
        Auditor[Auditor Agent]
    end

    subgraph Storage["Persistence"]
        JSONL[("logs/llm_executions.jsonl")]
        PG[("PostgreSQL\nllm_execution_logs")]
        PFile[("llmops/prompt_versions.json")]
    end

    subgraph API["FastAPI"]
        PE["/llmops/prompts/*"]
        EE["/llmops/executions/*"]
    end

    Writer -->|"registry.render()"| PR
    Auditor -->|"exec_logger.track()"| EL
    Writer -->|"exec_logger.track()"| EL

    PR --> PFile
    EL --> JSONL
    EL --> PG

    PE --> PR
    EE --> EL

Prompt Versioning

All LLM prompts are managed through the PromptRegistry. Prompts are stored in llmops/prompt_versions.json with full version history.

Capability	Description
Register	`POST /llmops/prompts/{name}` — creates a new version (auto-incremented)
Get Active	`GET /llmops/prompts/{name}` — returns active version + full history
Rollback	`POST /llmops/prompts/{name}/rollback` — switch to any previous version
Diff	`POST /llmops/prompts/{name}/diff` — unified diff between two versions
Render	Templates use `$variable` / `${variable}` syntax (Python `string.Template`)

Example — Register a new prompt version:

curl -X POST http://localhost:8000/llmops/prompts/writer_prompt \
  -H "Content-Type: application/json" \
  -d '{
    "template": "You are the Writer agent...\n\nGoal: ${goal}\n\nArticles: ${article_context}",
    "description": "Updated writer prompt with improved grounding instructions"
  }'

Example — Rollback to v1:

curl -X POST http://localhost:8000/llmops/prompts/writer_prompt/rollback \
  -H "Content-Type: application/json" \
  -d '{"target_version": 1}'

Execution Logging

Every LLM call is logged automatically by the ExecutionLogger with dual-write to:

Local JSONL — logs/llm_executions.jsonl (always available, no DB dependency)
PostgreSQL — llm_execution_logs table (best-effort, silently skipped if DB is unreachable)

Field	Description
`execution_id`	UUID per call
`timestamp`	ISO-8601 UTC
`agent_name`	`writer` / `auditor`
`prompt_name`	Registry key (e.g. `writer_prompt`)
`prompt_version`	Active version at call time
`model_id`	e.g. `anthropic.claude-3-haiku-20240307-v1:0`
`input_tokens`	Estimated input token count
`output_tokens`	Estimated output token count
`latency_ms`	Wall-clock execution time
`status`	`success` / `error`
`cost_estimate_usd`	Estimated cost based on model pricing
`prompt_hash`	SHA-256 of the rendered prompt
`response_preview`	First 500 chars of the response

Example — Query recent executions:

# All recent executions
curl http://localhost:8000/llmops/executions?limit=20

# Filter by agent
curl http://localhost:8000/llmops/executions?agent_name=writer

# Aggregated statistics
curl http://localhost:8000/llmops/executions/stats

Stats response example:

{
  "total_executions": 42,
  "success_count": 40,
  "error_count": 2,
  "error_rate": 0.0476,
  "avg_latency_ms": 3245.67,
  "total_input_tokens": 128400,
  "total_output_tokens": 52300,
  "total_cost_usd": 0.097425,
  "by_agent": { "writer": 28, "auditor": 14 },
  "by_model": { "anthropic.claude-3-haiku-20240307-v1:0": 42 }
}

Data Model

`newsletter_metrics` (PostgreSQL)

Column	Type	Description
`id`	Integer PK	Auto-increment
`faithfulness_score`	Float	DeepEval score (0.0–1.0)
`claims_supported`	Integer	Number of grounded citations
`duplicates_removed`	Integer	Articles filtered by Pinecone
`is_approved`	Boolean	Human approval decision
`revision_count`	Integer	Writer retry count
`created_at`	DateTime	UTC timestamp

`distribution_lists` (PostgreSQL)

Column	Type	Description
`id`	Integer PK	Auto-increment
`list_name`	String (unique)	e.g. `"engineering_team"`
`emails`	JSON	Array of recipient email addresses

`llm_execution_logs` (PostgreSQL) — LLMOps

Column	Type	Description
`id`	Integer PK	Auto-increment
`execution_id`	String (unique)	UUID per LLM call
`timestamp`	String	ISO-8601 UTC
`agent_name`	String	`writer` / `auditor`
`prompt_name`	String	Registry key
`prompt_version`	Integer	Prompt version at call time
`model_id`	String	Bedrock model identifier
`input_tokens`	Integer	Estimated input tokens
`output_tokens`	Integer	Estimated output tokens
`latency_ms`	Float	Wall-clock execution time
`status`	String	`success` / `error`
`error_message`	Text (nullable)	Error details if failed
`temperature`	Float	Model temperature setting
`max_tokens`	Integer	Max tokens setting
`prompt_hash`	String	SHA-256 of rendered prompt
`response_preview`	Text (nullable)	First 500 chars of response
`cost_estimate_usd`	Float	Estimated cost in USD
`created_at`	DateTime	UTC timestamp

Troubleshooting

Issue	Fix
`BedrockEmbeddings` auth error	Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` in `.env`
Pinecone `dimension mismatch`	Ensure index is created with `dimension=1536` (Titan Embed v2 output size)
Gmail `FileNotFoundError`	Download `credentials.json` from Google Cloud Console to project root
RDS SSL error	Run `curl -o global-bundle.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem`
`deepeval.models.BedrockModel` not found	Upgrade: `pip install deepeval --upgrade`
Dashboard shows no lists	Confirm FastAPI is running on port 8000 and `init_db()` has seeded `distribution_lists`

License

MIT License. See LICENSE for details.

Built with LangGraph · Amazon Bedrock · Pinecone · AWS RDS · FastAPI · React

Name		Name	Last commit message	Last commit date
agents		agents
api		api
crawlers		crawlers
dashboard		dashboard
docker		docker
email_service		email_service
evaluation		evaluation
llmops		llmops
logs		logs
newsletter		newsletter
scheduler		scheduler
tests		tests
utils		utils
.gitattributes		.gitattributes
.gitignore		.gitignore
MEMORY.md		MEMORY.md
README.md		README.md
config.py		config.py
generated-ai-newsletter.html		generated-ai-newsletter.html
generated_newsletter_mail_view.mp4		generated_newsletter_mail_view.mp4
global-bundle.pem		global-bundle.pem
main.py		main.py
project_changes.md		project_changes.md
requirements.txt		requirements.txt
setup.py		setup.py

Latest commit

History

Folders and files

Repository files navigation