Replace FluentBit by Vector on MODE 1

.github/renovate.json5

@@ -140,6 +140,17 @@
       "versioningTemplate": "docker",
       "extractVersionTemplate": "^(?<version>.+?)(?:-distroless-libc)?$"
     },
+    {
+      "description": "Vector image version in collector example DaemonSets",
+      "fileMatch": ["^data-index/collectors/examples/.*/daemonset\\.yaml$"],
+      "matchStrings": [
+        "image: timberio/vector:(?<currentValue>.*?)\\n"
+      ],
+      "datasourceTemplate": "docker",
+      "depNameTemplate": "timberio/vector",
+      "versioningTemplate": "docker",
+      "extractVersionTemplate": "^(?<version>.+?)(?:-distroless-libc)?$"
+    },
     {
       "description": "Elasticsearch image version in Helm values",
       "fileMatch": ["^data-index/helm/data-index/values.*\\.yaml$"],

.github/workflows/data-index-integration-tests.yml

@@ -27,7 +27,7 @@ jobs:
       fail-fast: false
       matrix:
         mode:
-          - name: MODE 1 (PostgreSQL + FluentBit + Triggers)
+          - name: MODE 1 (PostgreSQL + Vector + Triggers)
             script: full-test-mode1.sh
             id: mode1
           - name: MODE 2 (Elasticsearch + Vector + Transforms)
@@ -100,14 +100,8 @@ jobs:
           kubectl logs -l app=data-index-service --tail=100 || true
 
           echo ""
-          if [ "${{ matrix.mode.id }}" == "mode1" ]; then
-            echo "=== FluentBit Logs (MODE 1) ==="
-            kubectl logs -n logging -l app=fluentbit --tail=100 || true
-          fi
-
-          echo ""
-          if [ "${{ matrix.mode.id }}" == "mode2" ]; then
-            echo "=== Vector Logs (MODE 2) ==="
+          if [ "${{ matrix.mode.id }}" == "mode1" ] || [ "${{ matrix.mode.id }}" == "mode2" ]; then
+            echo "=== Vector Logs (${{ matrix.mode.id }}) ==="
             kubectl logs -n logging -l app=vector --tail=100 || true
           fi
 

CLAUDE.md

@@ -2,7 +2,7 @@
 
 **Project:** Data Index v1.0.0 for Open Workflow 1.0.0  
 **Status:** Production Ready (MODE 1, MODE 2 & MODE 3)  
-**Last Updated:** 2026-09-09
+**Last Updated:** 2026-09-10
 
 ---
 
@@ -134,7 +134,7 @@ Claude: I've created the ADR and committed it.  ❌ WRONG - no approval!
 This is a **read-only query service** for Open Workflow (OW 1.0.0) runtime execution data. It provides a GraphQL API for querying workflow instances and task executions.
 
 **What it does:**
-- Captures Quarkus Flow structured logging events via FluentBit (MODE 1) or Vector (MODE 2)
+- Captures Quarkus Flow structured logging events via Vector (MODE 1 & MODE 2)
 - Stores raw events in PostgreSQL (MODE 1) or Elasticsearch (MODE 2)
 - Normalizes events using PostgreSQL triggers (MODE 1) or Elasticsearch Transforms (MODE 2)
 - Exposes normalized data via GraphQL API (SmallRye GraphQL)
@@ -194,8 +194,8 @@ graphify --update
 ## Architecture (MODE 1 - Production)
 
 ```
-Quarkus Flow → /tmp/quarkus-flow-events.log (JSON)
-                      ↓ (FluentBit tail)
+Quarkus Flow → stdout → /var/log/containers/*.log (JSON)
+                      ↓ (Vector kubernetes_logs → postgres sink)
               PostgreSQL raw tables (JSONB)
                       ↓ (BEFORE INSERT triggers)
               PostgreSQL normalized tables
@@ -204,7 +204,7 @@ Quarkus Flow → /tmp/quarkus-flow-events.log (JSON)
 ```
 
 **Key Components:**
-- **FluentBit DaemonSet** - Tails log files, sends to PostgreSQL
+- **Vector DaemonSet** - Tails container logs, writes raw events to PostgreSQL (`postgres` sink; config: `data-index/collectors/vector/mode1-postgresql/vector.yaml`).
 - **PostgreSQL Triggers** - Normalize events immediately on INSERT
 - **Data Index Service** - Quarkus app with GraphQL API
 - **JPA Entities** - Map to normalized tables (workflow_instances, task_instances)
@@ -363,7 +363,7 @@ Quarkus Flow → Kafka (CloudEvents: binary or structured, topic: flow-lifecycle
 - Trade-off: Higher throughput vs manual CE handling (acceptable for high-volume workflow environments)
 
 **NOT used in MODE 3:**
-- ❌ FluentBit (events come from Kafka, not log files)
+- ❌ Vector / log collector (events come from Kafka, not container logs)
 - ❌ PostgreSQL triggers (normalization done in Java via JDBC)
 - ❌ Raw event tables (writes directly to normalized tables)
 - ❌ SmallRye automatic CloudEvent extraction (requires per-message mode, lower throughput)
@@ -406,12 +406,15 @@ data-index/
 │   └── WorkflowInstanceElasticsearchTest.java (Elasticsearch)
 ├── data-index-docs/               # User-facing documentation (Antora)
 │   └── modules/ROOT/pages/        # AsciiDoc documentation pages
+├── collectors/                    # Log-collector configs (Vector) - Go + Maven module
+│   ├── vector/mode1-postgresql/   # MODE 1 Vector config (postgres sink)
+│   ├── vector/mode2-elasticsearch/ # MODE 2 Vector config (elasticsearch sink)
+│   └── examples/                  # Reference DaemonSet manifests
+├── helm/data-index/              # Helm chart (all modes)
 ├── docs/                          # Internal documentation
-└── scripts/                       # Deployment scripts
-    ├── kind/                      # KIND (Kubernetes in Docker) scripts
-    └── fluentbit/                 # FluentBit configurations
-        ├── mode1-postgresql-triggers/ # MODE 1 FluentBit config
-        └── mode2-elasticsearch-transforms/ # MODE 2 FluentBit config
+└── scripts/
+    ├── e2e/                       # Helm-based E2E test scripts (mode1/2/3)
+    └── fluentbit/                 # FluentBit configs (DEPRECATED, ADR-0001)
 ```
 
 ---
@@ -989,11 +992,10 @@ MODE=postgresql ./install-dependencies.sh
 # 2. Deploy data-index service
 ./deploy-data-index.sh postgresql
 
-# 3. Deploy FluentBit (MODE 1)
-cd ../fluentbit/mode1-postgresql-triggers
-./generate-configmap.sh  # Generate from source files
-kubectl apply -f kubernetes/configmap.yaml
-kubectl apply -f kubernetes/daemonset.yaml
+# 3. Vector (MODE 1) is deployed by the Helm chart (values-mode1.yaml,
+#    vector.config=mode1-postgresql). Config source of truth:
+#    data-index/collectors/vector/mode1-postgresql/vector.yaml
+#    Full flow: bash data-index/scripts/e2e/full-test-mode1.sh
 
 # 4. Deploy test workflow app
 cd ../../kind
@@ -1220,7 +1222,7 @@ curl http://localhost:9200/_transform/workflow-instances-transform/_stats
 - Don't add Event Processor service (MODE 1 uses triggers, MODE 2 uses transforms)
 - Don't use polling architecture
 - Don't create staging tables (MODE 1) or separate processing indices (MODE 2)
-- Don't mix MODE 3 Kafka ingestion with MODE 1 FluentBit ingestion in the same deployment
+- Don't mix MODE 3 Kafka ingestion with MODE 1 Vector log ingestion in the same deployment
 - Don't mix PostgreSQL and Elasticsearch in same deployment
 
 ### ❌ Dependencies
@@ -1274,9 +1276,10 @@ curl http://localhost:9200/_transform/workflow-instances-transform/_stats
 ### Deployment Issues (MODE 1 - PostgreSQL)
 
 **"Events not in database"**
-- Check FluentBit logs: `kubectl logs -n logging -l app=workflows-fluent-bit-mode1`
-- Check PostgreSQL connection from FluentBit pod
-- Verify log file exists: `/tmp/quarkus-flow-events.log`
+- Check Vector logs: `kubectl logs -n logging -l app=vector`
+- Check the `postgres_workflow` / `postgres_task` sink health in the Vector logs
+- Enable event tracing: `kubectl set env daemonset/vector -n logging DEBUG_EVENTS=true`
+- Confirm Vector can reach PostgreSQL (`POSTGRES_HOST`/`POSTGRES_PORT` env on the DaemonSet)
 
 **"Raw tables populated but normalized tables empty"**
 - Check triggers exist: `\d workflow_events_raw` in psql
@@ -1353,8 +1356,9 @@ curl http://localhost:9200/_transform/workflow-instances-transform/_stats
 
 **Configuration:**
 - `data-index-service/data-index-service-elasticsearch/src/main/resources/application.properties` - Elasticsearch config (metrics, ILM, smart filtering)
+- `data-index/collectors/vector/mode1-postgresql/vector.yaml` - MODE 1 Vector (PostgreSQL, `postgres` sink)
 - `data-index/collectors/vector/mode2-elasticsearch/vector.yaml` - MODE 2 Vector (Elasticsearch)
-- `data-index/scripts/fluentbit/postgresql/fluent-bit.conf` - MODE 1 FluentBit (PostgreSQL)
+- `data-index/scripts/fluentbit/postgresql/fluent-bit.conf` - MODE 1 FluentBit (DEPRECATED, ADR-0001)
 
 **Testing:**
 - `data-index-integration-tests/data-index-integration-tests-postgresql/src/test/java/.../WorkflowInstanceGraphQLApiTest.java` - PostgreSQL GraphQL tests
@@ -1441,16 +1445,16 @@ curl http://localhost:9200/_transform/workflow-instances-transform/_stats
 → `data-index-elasticsearch-schema/resources/schema/task-executions-transform.json`
 
 **"How does data flow from Quarkus Flow to GraphQL?"**
-→ MODE 1: Quarkus Flow → log file → FluentBit → PostgreSQL raw → triggers → normalized → JPA → GraphQL
-→ MODE 2: Quarkus Flow → log file → Vector → ES raw indices → transforms → normalized indices → ES client → GraphQL
+→ MODE 1: Quarkus Flow → stdout → Vector → PostgreSQL raw → triggers → normalized → JPA → GraphQL
+→ MODE 2: Quarkus Flow → stdout → Vector → ES raw indices → transforms → normalized indices → ES client → GraphQL
 
 **"Which mode should I use?"**
 → MODE 1 for standard use cases, smaller deployments, simpler operations
 → MODE 2 for full-text search, aggregations, large scale, multi-tenancy
 
 **"Can I switch modes later?"**
 → Yes, same GraphQL API, just different storage backend
-→ Need to redeploy with different profile and reconfigure FluentBit
+→ Need to redeploy with different profile and switch the Vector config (mode1-postgresql ↔ mode2-elasticsearch)
 
 ---
 

data-index/Makefile

@@ -12,16 +12,22 @@
 #
 # ============================================================================
 
-.PHONY: help clean setup test-mode1 test-mode2 test-mode3 deploy-mode1 deploy-mode2 deploy-mode3
+.PHONY: help clean setup test-mode1 test-mode2 test-mode3 deploy-mode1 deploy-mode2 deploy-mode3 sync-vector-configs check-vector-configs
 
 # Configuration
 CLUSTER_NAME := data-index-test
 HELM_CHART := helm/data-index
 KIND_CONFIG := $(HELM_CHART)/kind-cluster.yaml
 
+# Vector config: collector source (source of truth) -> byte-identical Helm copy
+VECTOR_MODES := mode1-postgresql mode2-elasticsearch
+COLLECTOR_VECTOR_DIR := collectors/vector
+HELM_VECTOR_DIR := $(HELM_CHART)/configs/vector
+
 # Colors
 BLUE := \033[0;34m
 GREEN := \033[0;32m
+RED := \033[0;31m
 NC := \033[0m
 
 help: ## Show this help message
@@ -30,7 +36,7 @@ help: ## Show this help message
 	@echo "$(BLUE)╚════════════════════════════════════════════════════════════════╝$(NC)"
 	@echo ""
 	@echo "$(GREEN)Main Commands (what you typically run):$(NC)"
-	@echo "  $(GREEN)make test-mode1$(NC)   Full MODE 1 E2E (PostgreSQL + FluentBit + Triggers)"
+	@echo "  $(GREEN)make test-mode1$(NC)   Full MODE 1 E2E (PostgreSQL + Vector + Triggers)"
 	@echo "  $(GREEN)make test-mode2$(NC)   Full MODE 2 E2E (Elasticsearch + Vector + Transforms)"
 	@echo "  $(GREEN)make test-mode3$(NC)   Full MODE 3 E2E (Kafka + CloudEvents + Processors)"
 	@echo "  $(GREEN)make clean$(NC)        Delete KIND cluster"
@@ -51,8 +57,9 @@ help: ## Show this help message
 	@echo "  make e2e-mode2      Just run tests (MODE 2 must be deployed)"
 	@echo "  make e2e-mode3      Just run tests (MODE 3 must be deployed)"
 	@echo "  make status         Show pod status"
-	@echo "  make logs-*         Show logs (data-index, ingestion, fluentbit, vector)"
+	@echo "  make logs-*         Show logs (data-index, ingestion, vector, fluentbit)"
 	@echo "  make graphql-ui     Open GraphQL UI in browser"
+	@echo "  make check-vector-configs  Verify the Helm Vector configs are symlinks to collectors/vector/*"
 	@echo ""
 	@echo "$(GREEN)Shortcuts:$(NC)"
 	@echo "  make m1             Alias for test-mode1"
@@ -65,6 +72,18 @@ clean: ## Delete KIND cluster
 	@kind delete cluster --name $(CLUSTER_NAME) 2>/dev/null || echo "No cluster to delete"
 	@echo "$(GREEN)✓ Cluster deleted$(NC)"
 
+check-vector-configs: ## Fail if any Helm Vector config isn't a symlink to its collector source
+	@rc=0; for m in $(VECTOR_MODES); do \
+		f="$(HELM_VECTOR_DIR)/vector-$$m.yaml"; \
+		if [ ! -L "$$f" ]; then \
+			echo "$(RED)✗ $$f is not a symlink$(NC)"; rc=1; \
+		elif [ ! -e "$$f" ]; then \
+			echo "$(RED)✗ $$f is a broken symlink$(NC)"; rc=1; \
+		fi; \
+	done; \
+	if [ $$rc -eq 0 ]; then echo "$(GREEN)✓ Vector configs are symlinked to collectors/vector$(NC)"; fi; \
+	exit $$rc
+
 setup: ## Create cluster and build/load ALL images
 	@echo "$(BLUE)==> Setting up test environment (all images)$(NC)"
 	@MODE=all bash scripts/e2e/common-setup.sh
@@ -147,10 +166,7 @@ logs-data-index: ## Show Data Index service logs
 logs-ingestion: ## Show Data Index Ingestion logs (MODE 3)
 	@kubectl logs -n default -l app=data-index-ingestion --tail=50
 
-logs-fluentbit: ## Show FluentBit logs (MODE 1)
-	@kubectl logs -n logging -l app=fluentbit --tail=50
-
-logs-vector: ## Show Vector logs (MODE 2)
+logs-vector: ## Show Vector logs (MODE 1 & MODE 2)
 	@kubectl logs -n logging -l app=vector --tail=50
 
 port-forward: ## Port forward GraphQL API to localhost:8080

data-index/README.md

@@ -9,8 +9,8 @@
 Data Index provides a GraphQL API for querying workflow execution data from Quarkus Flow applications.
 
 **Deployment Modes:**
-- **MODE 1** (PostgreSQL + FluentBit + Triggers) - Production ready
-- **MODE 2** (Elasticsearch + FluentBit + Transforms) - Production ready
+- **MODE 1** (PostgreSQL + Vector + Triggers) - Production ready
+- **MODE 2** (Elasticsearch + Vector + Transforms) - Production ready
 - **MODE 3** (Kafka + SmallRye Reactive Messaging) - Production ready
 
 ## Quick Start
@@ -47,7 +47,7 @@ data-index/
 ├── data-index-ingestion/         # MODE 3 Kafka ingestion
 ├── data-index-integration-tests/ # E2E tests
 ├── workflow-test-app/            # Test workflow application
-└── scripts/                      # Deployment scripts (KIND, FluentBit, Kafka)
+└── scripts/                      # Deployment scripts (e2e, Kafka; FluentBit deprecated)
 ```
 
 ## Build