Skip to content
AJ576Public

About

Full-stack Retrieval-Augmented Generation (RAG) system that crawls, embeds, and semantically searches a large fictional knowledge base, delivering lore-expert answers via a Next.js frontend and FastAPI backend powered by FAISS + Gemini.

Resources

Stars

1 star

Watchers

1 watching

Forks

Latest commit

 

History

36 Commits

Folders and files

Repository files navigation

Cosmere Archivist (LoreGPT) – RAG System

A resource-optimized, full-stack Retrieval-Augmented Generation (RAG) system that serves as a lore archivist for Brandon Sanderson's Cosmere universe.

The system ingests and indexes 4,740 documents and 19,819 passages (over 2.68M words) crawled from the Coppermind wiki, performing low-latency semantic search and answering user questions in an in-universe scholarly archivist persona powered by Google Gemini. The entire backend is designed to run within 512MB RAM free-tier cloud container limits.


📊 Measured Performance & Corpus Metrics

All metrics below are measured directly from the indexed Cosmere corpus on an Apple Silicon (M-series) environment and are reproducible via benchmark_suite.py and benchmark_retrieval.py:

Metric Measured Value Measurement Details & Scope
Indexed Lore Knowledge Base 4,740 docs • 19,819 chunks • 2.68M words Exact count from meta.jsonl (median chunk: 578 chars, p95: 2,228 chars)
Metadata Table Heap Footprint 0.70 MB RAM (vs 25.32 MB for parsed JSON) Measured via tracemalloc for 64-bit offsets in LineIndexedJSONL (47.4 bytes/record)
Metadata File Seek Latency 0.007 ms p50 • 0.011 ms p95 Direct $O(1)$ disk seek per candidate chunk
Query Embedding Latency (CPU) 9.79 ms p50 • 12.60 ms p95 Dense vectorization using all-MiniLM-L6-v2
Vector Search Latency (FAISS) 0.81 ms p50 • 0.99 ms p95 (top-5) Exact inner-product scan on 19,819 normalized 384-d vectors (IndexFlatIP)
Vector Search vs NumPy BLAS 0.81 ms vs 1.03 ms (~21% lower latency) FAISS C++ SIMD scan compared against vectorized NumPy matmul (vectors @ q)
Cross-Encoder Rerank Latency 118.69 ms p50 • 164.25 ms p95 (20 pairs) Optional ms-marco-MiniLM-L-6-v2 re-scoring (disabled by default)
Retrieval Accuracy (Eval Set) 80.0% Recall@5 • 0.585 MRR@10 Measured on 20 hand-labeled domain queries (Recall@1: 45%, Recall@3: 65%)
Container RAM Budget 512 MB cap (-m 512m) CPU-only PyTorch, OMP_NUM_THREADS=1, single Uvicorn worker

Note

  • Process Memory vs Heap: The 0.70 MB metric is the Python heap footprint of the byte-offset lookup table. The overall running backend process (including the Python runtime, PyTorch CPU libraries, and FAISS C++ binaries) consumes ~150 MB – 220 MB of Resident Set Size (RSS), fitting safely within the 512MB container limit.
  • End-to-End Latency: Vector search (0.81 ms) and metadata lookup (0.007 ms) take under 1 ms combined. Total user-perceived query latency is dominated by local CPU query embedding (~10 ms) and the external Google Gemini API round-trip over HTTPS (~800 ms – 2,500 ms depending on network conditions and output token length).

✨ Features

  • Header-Aware Chunking Pipeline: Crawled raw articles split by Markdown ## headings, then token-bounded into 500-token chunks with 100-token sliding overlap via tiktoken.
  • MD5 Deduplication: Deduplication pass eliminates identical text passages across articles.
  • Custom Byte-Offset Metadata Engine (LineIndexedJSONL): Replaces heavy in-memory JSON dictionaries with a 64-bit file byte-offset table, slashing metadata RAM usage by over 96% with thread-safe sub-millisecond on-demand disk seeks.
  • Memory-Mapped Vector Search: 384-dimensional dense embeddings (all-MiniLM-L6-v2) indexed with FAISS IndexFlatIP using IO_FLAG_MMAP to prevent duplicating index memory in user space.
  • Two-Stage Retrieval Pipeline: Fast vector similarity search with optional Cross-Encoder reranking (cross-encoder/ms-marco-MiniLM-L-6-v2) toggled via environment configuration (ENABLE_RERANKER=true).
  • Persona-Driven Generative Q&A: Formats context into prompts for Google Gemini (gemini-2.5-flash-lite) adopting the scholarly "CosmereArchivist" voice, with dynamic context fallbacks when query relevance is low.
  • 512MB RAM Container Engineering: Containerized using Docker with CPU-only PyTorch, pre-baked model weights, single-threaded BLAS/OMP caps, and single-worker Uvicorn to fit strictly within 512MB cloud free tiers.
  • Modern Full-Stack UI: Responsive Next.js 15 interface with React 19, Tailwind CSS v4, and cold-boot status tracking / auto-retries for container spin-up.

⚡ Tech Stack

  • Backend: Python 3.13 • FastAPI • Uvicorn • Pydantic v2
  • Vector Search & ML: FAISS (faiss-cpu) • SentenceTransformers (all-MiniLM-L6-v2) • CrossEncoder (ms-marco-MiniLM-L-6-v2) • PyTorch (CPU-only)
  • Generative AI: Google Gemini API (gemini-2.5-flash-lite via google-generativeai)
  • Frontend: Next.js 15 • React 19 • Tailwind CSS v4
  • Infrastructure & Testing: Docker • Vercel (Frontend) • Render (Backend) • Pytest

🚀 Architecture & Pipeline

[ Crawled Wiki Docs (4,740) ]
            │
            ▼
[ Ingestion & Header-Aware Chunking ] ──► [ MD5 Deduplication ] (19,819 chunks)
            │
            ├──► Embeddings (all-MiniLM-L6-v2) ──► FAISS Index (IndexFlatIP, MMAP)
            │
            └──► Metadata (LineIndexedJSONL)   ──► 64-bit Byte Offset Table (<1 MB heap)

Query Flow:

  1. User Query: The Next.js chat frontend sends POST /ask with the question.
  2. Embedding: Backend embeds the query text into a normalized 384-dimensional vector on CPU (~10 ms).
  3. Candidate Search: FAISS performs fast inner-product search across 19,819 vectors (~0.8 ms).
  4. Metadata Lookup: LineIndexedJSONL executes $O(1)$ file seeks to fetch chunk text for the candidate IDs (~0.01 ms).
  5. Optional Reranking: If ENABLE_RERANKER=true, candidates are scored via Cross-Encoder (~120 ms).
  6. Prompt Assembly & Generation: Chunks are formatted into a prompt and dispatched to gemini-2.5-flash-lite over HTTPS.
  7. Response: Generated in-universe archivist response is returned to the user.

🔌 API Endpoints

FastAPI server serves the following endpoints (with Swagger UI at http://localhost:8000/docs):

1. POST /ask

Runs full RAG pipeline (search → optional rerank → generate answer via Gemini).

  • Request Body:
    {
      "question": "Who is Kaladin Stormblessed?"
    }
  • Response:
    {
      "answer": "Kaladin Stormblessed is a Windrunner of the Knights Radiant..."
    }

2. POST /search?top_k=5

Retrieves top-k candidate chunks without invoking the LLM generation step.

  • Request Body:
    {
      "question": "What are the Ten Surges?"
    }
  • Response:
    {
      "chunks": [
        {
          "doc_id": "Surgebinding",
          "chunk_index": 2,
          "text": "The Ten Surges are fundamental forces..."
        }
      ]
    }

🛠️ Local Setup

Prerequisites

1. Clone the Repository

git clone https://github.com/AJ576/loreGPT.git
cd loreGPT

2. Backend Setup

cd backend

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Set your Gemini API key
export GEMINI_API_KEY="your-gemini-api-key"
# Or create backend/.env with: GEMINI_API_KEY=your-gemini-api-key

# Start the FastAPI server
uvicorn src.api.app:app --reload --port 8000

The API is now running at http://localhost:8000.

3. Frontend Setup

In a new terminal window:

cd frontend

# Install npm packages
npm install

# Start development server
npm run dev

Open http://localhost:3000 to interact with the Cosmere Archivist interface.


🧪 Running Tests & Benchmarks

All test suites and benchmarks run without external server dependencies.

1. Run Automated Tests

# From backend directory
pytest tests

# Or skip the slow embedding model download test
pytest tests -m "not slow"

2. Run Performance Benchmark Suite

Measures metadata memory footprint, disk seek times, query embedding latency, and FAISS vector retrieval:

python3 backend/benchmarks/benchmark_suite.py

# Optional flags:
# --rerank   include Cross-Encoder latency
# --llm      include real Gemini API calls (uses API quota)
# --quality  include retrieval hit-rate evaluation

Results are saved to backend/benchmarks/measured_results.json.

3. Run Retrieval Quality Evaluation

Evaluates Hit Rate / Recall @ 1, 3, 5, 10 and MRR @ 10 on 20 curated Cosmere queries:

python3 backend/benchmarks/benchmark_retrieval.py

# With cross-encoder reranking:
python3 backend/benchmarks/benchmark_retrieval.py --rerank

4. Run Concurrent API Load Test

Measures throughput and latency percentiles under concurrent client loads:

# Make sure backend is running on port 8000 first:
# uvicorn src.api.app:app --port 8000
python3 backend/benchmarks/benchmark_api.py

🐳 Docker Deployment

To build and run the backend container locally matching the 512MB RAM production profile:

cd backend

# Build Docker image
docker build -t cosmere-archivist-backend .

# Run container with 512MB memory limit
docker run -p 8000:8000 -m 512m --memory-swap 512m -e GEMINI_API_KEY="your-api-key" cosmere-archivist-backend

📄 License

This project is licensed under the MIT License. Fictional Cosmere lore content is based on the works of Brandon Sanderson, documented by the community at Coppermind.net.

About

Full-stack Retrieval-Augmented Generation (RAG) system that crawls, embeds, and semantically searches a large fictional knowledge base, delivering lore-expert answers via a Next.js frontend and FastAPI backend powered by FAISS + Gemini.

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages