Cloud driver project (EMR Spark + RDS architecture)

This project is a React single-page app plus a FastAPI API on AWS Elastic Beanstalk. Raw driver telemetry lives in Amazon S3 and is processed by Amazon EMR Spark jobs (PySpark). Job A writes per-driver aggregates to RDS, and Job B writes speed feeds to RDS. The API serves summary and speed by reading those precomputed RDS tables.

The emr/ folder contains the PySpark scripts used by this deployment model.

website url: http://cloud-project-web.s3-website-us-east-1.amazonaws.com/

---

Architecture

Component	Role
S3 bucket (web)	Hosts the built React app (frontend/dist) via static website hosting.
S3 bucket (data)	Stores headerless CSV detail files (same column order as the COMP4442 dataset PDF).
Amazon EMR	Runs Spark job A (summary batch) and job B (speed feed / live status writes).
RDS PostgreSQL	Stores driver_behavior_summary, speed_snapshot, and current_vehicle_status/speed_feed_latest.
Elastic Beanstalk	Runs uvicorn + FastAPI and only reads from RDS.

Why this split? EMR Spark handles large distributed transforms; the API remains lightweight and responds quickly by querying RDS.

---

How each component is deployed

1. Data bucket (raw CSV)

1. Create an S3 bucket (e.g. cloud-project-raw-data) in your Region.
2. Upload your detail files under a prefix, e.g. raw-driver-data/.
3. Block public access; only your account (and the EB instance role) should read objects.
4. Note the URI you will pass to the API, e.g. s3://cloud-project-raw-data/raw-driver-data/.

2. Web bucket (React SPA)

1. Create a second bucket (e.g. cloud-project-web) for the frontend.
2. Locally: cd frontend && npm ci && npm run build.
3. Set frontend/.env so **VITE_API_BASE** is your Elastic Beanstalk URL (e.g. https://your-env.elasticbeanstalk.com) before building.
4. Sync the build output: aws s3 sync dist/ s3://cloud-project-web/ --delete
5. Enable Static website hosting (index document index.html; set error document to index.html for SPA routes).
6. Bucket policy: allow public read for s3:GetObject on the website objects (static website hosting uses HTTP; see CORS below).

3. Elastic Beanstalk (API)

1. IAM instance profile for the EB environment: allow read access to Secrets Manager/SSM if you store DB credentials there.
2. Create a Python Elastic Beanstalk application and environment.
3. Ensure the environment can reach RDS (5432) through security groups.
4. Configure environment properties (see **backend/.env.example**):

• **DATABASE_URL** — postgresql+psycopg://app_user:PASS@YOUR_RDS_ENDPOINT:5432/cloudproject
• **CORS_ORIGINS** — exact origin of your static site, e.g. http://cloud-project-web.s3-website-us-east-1.amazonaws.com
• **USE_MOCK_WHEN_EMPTY** — set to **0** in production once data loads correctly

5. Deploy a zip whose root contains Procfile, requirements.txt, **app/**, and **.ebextensions/**. Example from backend/: PowerShell bash
6. Upload **eb-backend.zip** (Upload and deploy) or use eb deploy.
7. Confirm **GET /health**, **GET /api/summary**, and **GET /api/speed?as_of=...** return data as expected.

Runtime: API instances do not run Spark transforms, so scaling concerns are standard API + DB considerations.

4. Amazon EMR (Spark ETL jobs)

1. Create an EMR cluster (or use EMR Serverless with equivalent Spark submit parameters) in the same VPC as RDS, or with routes/security groups so Spark tasks can reach RDS on port 5432.
2. Upload scripts to an S3 artifacts bucket (example layout):

• s3://cloud-project-emr-artifacts/emr/driver_dataset.py
• s3://cloud-project-emr-artifacts/emr/job-a/driver_behavior_summary.py
• s3://cloud-project-emr-artifacts/emr/job-b/speed_snapshot_feed.py

3. PostgreSQL JDBC: EMR Spark needs the Postgres driver on the classpath. Typical approach — pass a Maven coordinate on submit (version can track your EMR release docs):

 --packages org.postgresql:postgresql:42.7.3

4. Job A — spark-submit (cluster or client mode). Example:

 spark-submit --deploy-mode cluster \
   --packages org.postgresql:postgresql:42.7.3 \
   --py-files s3://cloud-project-emr-artifacts/emr/driver_dataset.py \
   s3://cloud-project-emr-artifacts/emr/job-a/driver_behavior_summary.py \
   --raw-s3-path s3://cloud-project-raw-data/raw-driver-data/ \
   --rds-jdbc-url jdbc:postgresql://YOUR_RDS:5432/cloudproject \
   --rds-user app_user --rds-password 'app_user_password' \
   --period-start 2017-01-01 --period-end 2017-01-10

Optional: --source-timezone (IANA id, default Asia/Shanghai). CSV Time is wall-clock in that zone; observed_at is stored in UTC and event_date is the local calendar date there. 5. Job B — same pattern, without period args:

 spark-submit --deploy-mode cluster \
   --packages org.postgresql:postgresql:42.7.3 \
   --py-files s3://cloud-project-emr-artifacts/emr/driver_dataset.py \
   s3://cloud-project-emr-artifacts/emr/job-b/speed_snapshot_feed.py \
   --raw-s3-path s3://cloud-project-raw-data/raw-driver-data/ \
   --rds-jdbc-url jdbc:postgresql://YOUR_RDS:5432/cloudproject \
   --rds-user app_user --rds-password 'app_user_password'

6. IAM / network: The EMR EC2 instance profile (or job execution role for Serverless) needs:

• Read access to the raw-data and artifact S3 prefixes
• Network path to RDS (security group allows EMR nodes → RDS 5432)
• Prefer Secrets Manager or SSM for passwords instead of inline args in production

7. Run order: Job A for the chosen period → Job B for snapshots and current_vehicle_status → then use /api/summary and /api/speed from the frontend.

5. CORS

The browser loads the SPA from the S3 website origin and calls the API on the EB origin. **CORS_ORIGINS** must list the static site origin exactly. For local development, include http://localhost:5173.

---

Local development

From **backend/**:

pip install -r requirements.txt

• Set **DATABASE_URL** to your local/Postgres test DB.
• Run EMR Spark jobs to populate DB tables before querying API in non-mock mode.

**USE_MOCK_WHEN_EMPTY=1** returns demo JSON when table queries yield no rows.

Run the API:

uvicorn app.main:app --reload --port 8000

From **frontend/**: npm run dev with VITE_API_BASE=http://127.0.0.1:8000 (or empty if using a proxy).

---

API behavior (SPA)

• **GET /api/summary** — returns all rows from **driver_behavior_summary** (Spark job A output). Each row includes **periodLabel** / **aggregationPeriodLabel** for that row’s stored aggregation window.
• **GET /api/speed?as_of=<ISO-8601 datetime>** — required **as_of** (UTC). For each known driver (from summary, or from **speed_snapshot** if summary is empty), the API returns the latest snapshot in **[as_of − 1 minute, as_of]**. Each series includes **isSpeeding** on points and **isDriving**: **false** when no snapshot exists in that one-minute window. Demo data is used when **USE_MOCK_WHEN_EMPTY=1** and there are no fleet rows. If ETL stored naive local times as UTC before fixing **SOURCE_TIMEZONE**, set **SPEED_OBSERVED_AT_OFFSET_HOURS** in the API env; after re-running Spark jobs with proper UTC writes, set it back to **0**.

---

Related files

Path	Purpose
emr/driver_dataset.py	Shared Spark reads/normalization for job A and B.
emr/job-a/driver_behavior_summary.py	Batch Spark job writing summary rows to RDS.
emr/job-b/speed_snapshot_feed.py	Spark job writing speed snapshots and latest status rows to RDS.
backend/app/database.py	SQLAlchemy engine/session for API reads.
backend/app/models.py	ORM models for summary/speed tables.
backend/.env.example	Environment variables.