Build and publish Kafka ingestion service image

[ricardozanini]

Create container image for the Kafka ingestion service and integrate it into the quay.io publishing pipeline. This service consumes workflow lifecycle events from Kafka (CloudEvents format) and persists them to PostgreSQL via JDBC batch UPSERT.

Supports:

• MODE 3: Kafka + CloudEvents + Ingestion Service (Kafka topics → batch consumer → JDBC UPSERT → PostgreSQL normalized tables)

Acceptance Criteria

• Dockerfile verified/created at data-index/data-index-ingestion/data-index-ingestion-kafka-service/src/main/docker/Dockerfile.jvm
• Dockerfile optimized for size and security (use Quarkus-generated multi-stage build)
• Remove Jib dependency from ingestion service pom.xml (if present)
• Update E2E test scripts to use docker build:
  • data-index/scripts/e2e/full-test-mode3.sh
  • Build command: docker build -f src/main/docker/Dockerfile.jvm -t <image> .
• Image published to quay.io/kubesmarts/logic-data-index-ingestion-kafka:${VERSION}
• Image includes OCI labels (version, revision, created, source)
• Image tested in KIND cluster with Kafka and PostgreSQL (MODE 3 E2E test)
• README updated with image pull instructions and build documentation

Build Configuration

Module Path: data-index/data-index-ingestion/data-index-ingestion-kafka-service

Maven Build (JAR only):

mvn clean package -DskipTests

Docker Build:

cd data-index/data-index-ingestion/data-index-ingestion-kafka-service
docker build -f src/main/docker/Dockerfile.jvm \
  -t quay.io/kubesmarts/logic-data-index-ingestion-kafka:latest \
  .

Image Base: Quarkus JVM mode

Target Image Size: < 400MB

Required Runtime Configuration:

• Kafka bootstrap servers (via env vars)
• Kafka topic name (CloudEvents input: flow-lifecycle-out)
• PostgreSQL connection string
• Dead-letter queue topic (for failed events: data-index-events-dlq)
• Batch size configuration (default: 1000 Kafka records, 1000 DB records)

Testing

Deploy MODE 3 stack to KIND cluster and verify:

# Deploy Kafka, PostgreSQL, ingestion service
kubectl apply -f data-index/deploy/mode3/

# Deploy workflow test app configured for Kafka output
kubectl apply -f data-index/deploy/mode3/workflow-test-app.yaml

# Trigger workflow execution
curl -X POST http://workflow-test-app/... -d '{"test":"data"}'

# Verify events consumed and persisted
kubectl logs -l app=data-index-ingestion-kafka | grep "Processed event"
psql -c "SELECT * FROM workflow_instances;"

Dependencies

Depends on #67