Replace FluentBit by Vector on MODE 1

[mcruzdev]

Changes

This pull request aims to replace the FluentBit to Vector.

Closes #63

Below you can see how to test the entire change with a real Kubernetes cluster with KinD.

Step-by-step walkthrough to manually verify the Vector migration for MODE 1

Starting Colima:

colima stop

colima start --cpu 6 --memory 12 --disk 60

1. Create the KIND cluster

kind create cluster --name data-index-test --config data-index/helm/data-index/kind-cluster.yaml

kubectl cluster-info --context kind-data-index-test

Maps localhost:30080 (GraphQL), localhost:30082 (workflow app), localhost:30432 (Postgres).

2. Build + load the MODE 1 images

( cd data-index && MODE=mode1 bash scripts/e2e/common-setup.sh )

Check docker images:

docker exec data-index-test-control-plane crictl images | grep kubesmarts

3. Install the chart (Postgres + Vector + triggers)

helm upgrade --install data-index data-index/helm/data-index -n default --create-namespace -f data-index/helm/data-index/values-mode1.yaml

Confirm one vector DaemonSet, no fluentbit:

kubectl get ds -A

kubectl get cm -n logging vector-config -o jsonpath='{.data.vector\.yaml}' | head -3

4. Wait for pods

PostgreSQL:

kubectl wait -n postgresql --for=condition=ready pod/postgresql-0 --timeout=180s

Vector:

kubectl wait -n logging --for=condition=ready pod -l app=vector --timeout=180s

Data Index Service:

kubectl wait -n default --for=condition=ready pod -l app=data-index-service --timeout=300s

Workflow Test App:

kubectl wait -n workflows --for=condition=ready pod -l app=workflow-test-app --timeout=300s

kubectl get pods -A | grep -E 'postgresql|data-index|vector|workflow'

5. Verify the infrastructure

bash data-index/scripts/e2e/verify-infrastructure.sh mode1

Expect: Vector running, GraphQL ready, 4 tables, 2 triggers.

kubectl logs -n logging -l app=vector | grep -iE 'healthcheck|postgres|error'

NOTE: If you got some "Timed out." you can restart the daemonset/vector:

kubectl rollout restart daemonset/vector -n logging

6. Turn on event tracing (optional)

kubectl set env daemonset/vector -n logging DEBUG_EVENTS=true

kubectl rollout status ds/vector -n logging

kubectl logs -n logging -l app=vector -f

(leave that streaming in another terminal)

7. Trigger a workflow

curl -s -XPOST localhost:30082/test-workflows/hello-world -H 'content-type: application/json' -d '{"name":"neo"}'

Traced Vector output should show lines shaped
{"tag":"io.serverlessworkflow.workflow.started.v1","time":"...","data":{...}}.

8. Verify raw tables (written by the Vector postgres sinks)

kubectl exec -n postgresql postgresql-0 -- psql -U dataindex -d dataindex -c "select count(*) from workflow_events_raw; select count(*) from task_events_raw;"

kubectl exec -n postgresql postgresql-0 -- psql -U dataindex -d dataindex -c "select tag, time from workflow_events_raw order by time;"

9. Verify normalized tables (written by the V1 triggers)

kubectl exec -n postgresql postgresql-0 -- psql -U dataindex -d dataindex -c "select id,name,status,started_at,ended_at from workflow_instances;"

kubectl exec -n postgresql postgresql-0 -- psql -U dataindex -d dataindex -c "select instance_id,task,task_name,status from task_instances;"

Expect the workflow row status = COMPLETED and its task rows.

10. Verify the GraphQL API

curl -s localhost:30080/graphql -H 'content-type: application/json' -d '{"query":"{ getWorkflowInstances(limit:5){ id name status startedAt endedAt taskExecutions{ task taskName status startedAt endedAt } } }"}' | python3 -m json.tool

WorkflowInstance / TaskExecution fields: id name namespace version status startedAt endedAt lastUpdate eventTimestamp inputData outputData error{ ... } taskExecutions{ ... }. Introspect with { __type(name:"WorkflowInstance"){ fields{ name } } }.

11. Idempotency spot-check (optional)

for i in 1 2 3; do curl -s -XPOST localhost:30082/test-workflows/hello-world -H 'content-type: application/json' -d '{"name":"neo"}' >/dev/null; done; sleep 5

kubectl exec -n postgresql postgresql-0 -- psql -U dataindex -d dataindex -c "select instance_id, task, count(*) from task_instances group by 1,2 having count(*) > 1;"

Expect: 0 rows.

12. Cleanup

kind delete cluster --name data-index-test

Config source of truth: data-index/collectors/vector/mode1-postgresql/vector.yaml.
Full reference: data-index/data-index-docs/modules/ROOT/pages/deployment/vector-config.adoc.

[ricardozanini]

Same as Red Hat currently:

https://catalog.redhat.com/en/software/containers/openshift-logging/vector-rhel9/6447991f84aeefde2c9f2bcf#containerfile

[ricardozanini]

This section can be deleted.

[ricardozanini]

We have a Makefile for this!

[ricardozanini]

FluentBit deprecation can be removed.

[ricardozanini]

Delete.

[ricardozanini]

Can be deleted, this whole dir.

[ricardozanini]

Can be removed.

[ricardozanini]

Instead, perhaps helm can follow symlinks so we don't keep a copy in each dir? (collectors and helm)

[mcruzdev]

Yeah, it is true

[Copilot]

🟡 Changes recommended

Critical Vector liveness and existing-secret credential issues remain unresolved, along with configuration and readiness gaps.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR replaces FluentBit with Vector for MODE 1 event collection while retaining PostgreSQL trigger-based normalization.

Changes:

• Adds Vector PostgreSQL pipelines, Helm deployment, and validation.
• Updates E2E scripts, CI, and operational documentation.
• Deprecates FluentBit configurations and guidance.

File summaries

File	Summary
data-index/workflow-test-app/README.md	Updates collection documentation to Vector.
data-index/scripts/kafka/README.md	Clarifies MODE 3 collector behavior.
data-index/scripts/fluentbit/README.md	Deprecates FluentBit configuration.
data-index/scripts/fluentbit/postgresql/README.md	Deprecates MODE 1 FluentBit documentation.
data-index/scripts/fluentbit/generate-configmap.sh	Adds a deprecation notice.
data-index/scripts/fluentbit/deploy-fluentbit.sh	Adds a deprecation notice.
data-index/scripts/e2e/verify-infrastructure.sh	Verifies Vector infrastructure; readiness checks should validate the full DaemonSet.
data-index/scripts/e2e/full-test-mode1.sh	Migrates the MODE 1 E2E flow to Vector.
data-index/scripts/e2e/common-setup.sh	Installs artifacts for E2E builds.
data-index/README.md	Updates deployment mode descriptions.
data-index/Makefile	Adds Vector sync and validation targets; CI does not invoke the documented config check.
data-index/helm/data-index/values.yaml	Defines shared Vector defaults.
data-index/helm/data-index/values-mode2.yaml	Selects the MODE 2 Vector configuration.
data-index/helm/data-index/values-mode1.yaml	Enables the MODE 1 Vector configuration.
data-index/helm/data-index/templates/workflow-test-app.yaml	Updates logging comments and timestamp guidance.
data-index/helm/data-index/templates/vector-daemonset.yaml	Adds Vector database configuration; host configurability and existing-secret credential wiring require changes.
data-index/helm/data-index/templates/vector-configmap.yaml	Selects mode-specific Vector configuration.
data-index/helm/data-index/templates/fluentbit-daemonset.yaml	Marks the FluentBit deployment deprecated.
data-index/helm/data-index/templates/fluentbit-configmap.yaml	Marks the FluentBit ConfigMap deprecated.
data-index/helm/data-index/templates/data-index-service.yaml	Updates raw-table ownership documentation.
data-index/helm/data-index/README.md	Documents Vector-based deployments.
data-index/helm/data-index/configs/vector/vector-mode2-elasticsearch.yaml	Adds the Helm MODE 2 Vector configuration.
data-index/helm/data-index/configs/vector/vector-mode1-postgresql.yaml	Adds the Helm MODE 1 Vector configuration.
data-index/helm/data-index/configs/fluentbit/README.md	Deprecates Helm FluentBit configurations.
data-index/docs/README.md	Updates collector documentation references.
data-index/docs/MULTI_TENANT_FLUENTBIT.md	Marks FluentBit guidance deprecated.
data-index/docs/elasticsearch/TRANSFORM_OPTIMIZATION.md	Updates MODE 2 collector references.
data-index/docs/deployment/MODE3_KAFKA_INGESTION.md	Updates mode comparison terminology.
data-index/docs/deployment/MODE2_E2E_TESTING.md	Marks the old MODE 2 guide superseded.
data-index/data-index-storage/data-index-storage-postgresql/README.md	Updates raw-table ingestion documentation.
data-index/data-index-storage/data-index-storage-migrations/README.md	Documents Vector raw-table writes.
data-index/data-index-storage/data-index-storage-elasticsearch/README.md	Updates Elasticsearch ingestion references.
data-index/data-index-service/README.md	Updates collector ownership documentation.
data-index/data-index-e2e-tests/README.md	Updates E2E architecture references.
data-index/data-index-docs/README.md	Adds Vector documentation navigation.
data-index/data-index-docs/modules/ROOT/pages/operations/event-reliability.adoc	Reworks reliability guidance for Vector.
data-index/data-index-docs/modules/ROOT/pages/index.adoc	Updates project architecture and prerequisites.
data-index/data-index-docs/modules/ROOT/pages/getting-started.adoc	Updates KIND deployment instructions.
data-index/data-index-docs/modules/ROOT/pages/developers/quarkus-flow-integration.adoc	Updates Vector integration guidance.
data-index/data-index-docs/modules/ROOT/pages/developers/quarkus-flow-apps.adoc	Updates application integration guidance.
data-index/data-index-docs/modules/ROOT/pages/developers/configuration.adoc	Updates Kubernetes collector configuration.
data-index/data-index-docs/modules/ROOT/pages/deployment/vector-config.adoc	Adds Vector configuration documentation.
data-index/data-index-docs/modules/ROOT/pages/deployment/troubleshooting.adoc	Updates troubleshooting commands for Vector.
data-index/data-index-docs/modules/ROOT/pages/deployment/postgresql.adoc	Replaces FluentBit deployment instructions with Vector.
data-index/data-index-docs/modules/ROOT/pages/deployment/overview.adoc	Updates deployment architecture.
data-index/data-index-docs/modules/ROOT/pages/deployment/kind-local.adoc	Updates KIND deployment guidance; removes the stale FluentBit parser troubleshooting command.
data-index/data-index-docs/modules/ROOT/pages/deployment/kafka.adoc	Updates mode comparison tables.
data-index/data-index-docs/modules/ROOT/pages/deployment/fluentbit-config.adoc	Marks FluentBit documentation deprecated.
data-index/data-index-docs/modules/ROOT/pages/deployment/elasticsearch.adoc	Updates Vector RBAC and HA references.
data-index/data-index-docs/modules/ROOT/pages/architecture/postgresql-mode.adoc	Documents the Vector PostgreSQL pipeline.
data-index/data-index-docs/modules/ROOT/pages/architecture/overview.adoc	Updates overall architecture.
data-index/data-index-docs/modules/ROOT/pages/architecture/kafka-mode.adoc	Updates ingestion-layer comparisons.
data-index/data-index-docs/modules/ROOT/nav.adoc	Adds Vector and deprecated FluentBit navigation.
data-index/collectors/vector/mode1-postgresql/vector.yaml	Adds the MODE 1 pipeline; URI password encoding and the documented Maven test phase require correction.
data-index/collectors/src/test/java/org/kubesmarts/logic/apps/dataindex/collectors/VectorConfigValidationIT.java	Validates Vector configuration; the documented mvn test command does not execute this integration test.
data-index/collectors/pom.xml	Updates Vector test/build configuration.
data-index/collectors/examples/mode1-postgresql/daemonset.yaml	Adds a reference DaemonSet; the liveness probe targets the wrong Vector port.
CLAUDE.md	Updates repository architecture and operational references.
.github/workflows/data-index-integration-tests.yml	Collects Vector logs in CI.
.github/renovate.json5	Tracks Vector versions in example manifests.

Review details

Suppressed comments (5)

data-index/Makefile:63

• The help text says CI runs check-vector-configs, but no workflow invokes that target; config-copy drift can therefore pass CI despite this claim. Either add the target to the workflow or remove the parenthetical.

	@echo "  make check-vector-configs  Verify the Helm copies are in sync (CI does this too)"

data-index/collectors/examples/mode1-postgresql/daemonset.yaml:197

• The readiness probe has the same port mismatch: /health is on the Vector API at 8686, not on the Prometheus exporter at 9598. With the reference DaemonSet applied, the pod will remain unready; use port 8686 here as well.

        readinessProbe:
          httpGet:
            path: /health
            port: 9598

data-index/collectors/src/test/java/org/kubesmarts/logic/apps/dataindex/collectors/VectorConfigValidationIT.java:89

• The new MODE 1 validation is in VectorConfigValidationIT; the collectors POM runs **/*IT.java under Failsafe and does not include IT classes in Surefire. Consequently, mvn test -pl collectors (the command documented by this config) will not execute this check. Run/document mvn verify or configure/rename the test so the promised command actually covers the new pipeline.

    @Test
    void mode1PostgreSQLConfigIsValid() throws Exception {
        Path configPath = getConfigPath("mode1-postgresql/vector.yaml");

        String configContent = Files.readString(configPath);
        assertThat(configContent)
                .as("Config should define the kubernetes_logs source")
                .contains("sources:")
                .contains("kubernetes_logs:");

        assertThat(configContent)
                .as("Config should define one postgres sink per raw table")
                .contains("postgres_workflow:")
                .contains("postgres_task:")
                .contains("table: workflow_events_raw")
                .contains("table: task_events_raw");

        validateWithVectorContainer(configPath, MODE1_ENV);

data-index/collectors/vector/mode1-postgresql/vector.yaml:168

• The PostgreSQL connection URI embeds the password without URI encoding. Valid production passwords containing reserved characters such as @, /, #, or % will be parsed as part of the URI structure and the sink will fail to connect; the old pgsql configuration passed the password as a separate option. Use a URI-safe/encoded credential or a connection mechanism that does not embed raw credentials in the endpoint.

    endpoint: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}"

data-index/collectors/vector/mode1-postgresql/vector.yaml:181

• The task sink repeats the raw-password URI construction, so passwords with URI-reserved characters will fail task-event delivery even if the workflow sink is corrected. Apply the same credential-encoding/connection handling to this endpoint.

    endpoint: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}"

• Files reviewed: 59/60 changed files
• Comments generated: 6
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.