Real-time AI posture detection that runs entirely in your browser β no server, no data leaves your device.
SitRight monitors your sitting posture through your webcam and delivers instant visual and audio feedback. MediaPipe JS detects body landmarks directly in the browser, and a custom neural network (trained from scratch, no AutoML) scores your posture in real time. All inference runs client-side via WebGL β the webcam feed never touches a network. Deploy it as a static site for free.
This project started at a hackathon with a server-side Python architecture, then evolved through production profiling and iterative redesign into a fully client-side system. The architecture history below documents that journey.
The average person spends over 6 hours daily in front of screens. Chronic poor posture during desk work is a leading contributor to neck and lower back pain in young adults. SitRight was built to be a frictionless, privacy-respecting tool β no app install, no account, no data leaving your machine β that makes posture awareness as passive as possible.
- Captures your webcam feed at ~10 FPS
- Runs MediaPipe Pose Landmarker (JS, WebGL-accelerated) to extract body landmarks
- Engineers 7 geometric features (joint angles, distances, ratios) from those landmarks
- Passes features through a custom neural network (929 parameters, plain JS forward pass) to produce a posture score (0-100)
- Displays a color-coded score with an annotated skeleton overlay on the live canvas
All processing happens locally. No network requests during inference.
Browser (all processing local)
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β
β getUserMedia() β
β β β
β βΌ β
β ββββββββββββββββββββββββ β
β β MediaPipe Pose β WebGL-accelerated β
β β Landmark Detection β landmark extraction β
β ββββββββββββ¬ββββββββββββ β
β β β
β βΌ β
β ββββββββββββββββββββββββ β
β β Feature Engineering β 7 geometric features β
β β (angles, distances, β (JS implementation) β
β β ratios) β β
β ββββββββββββ¬ββββββββββββ β
β β β
β βΌ β
β ββββββββββββββββββββββββ β
β β Neural Network β 929 params, 4 dense layers β
β β Forward Pass (JS) β weights hardcoded in JS β
β β MinMax scaled input β <50ms per frame β
β ββββββββββββ¬ββββββββββββ β
β β β
β βΌ β
β Canvas skeleton overlay + Score widget + Chart.js analytics β
β β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The model is not a wrapper around a pre-existing posture API. Every layer was designed and built manually:
Feature Engineering β Rather than feeding raw pixel data or all landmarks into a model (which would require far more training data and compute), 7 semantically meaningful features are extracted:
| Feature | Captures |
|---|---|
dist_nose_shoulders |
How far forward the head is relative to the torso |
ratio_nose_shoulders |
Scale-invariant head forward tilt |
neck_tilt_angle |
Lateral neck angle using ear-nose-ear geometry |
dist_left_ear_nose |
Asymmetry in head position |
dist_right_ear_nose |
Asymmetry in head position |
angle_left_shoulder |
Shoulder roll relative to head |
angle_right_shoulder |
Shoulder roll relative to head |
Model Architecture β A 4-layer fully-connected network with ReLU activations and a sigmoid output. 929 total parameters. Binary cross-entropy loss with label normalization (raw 0-100 scores mapped to 0.0-1.0 for training). Weights are exported and hardcoded directly into the JavaScript inference file β no model-loading step, no external weight files.
Training β 5-fold cross-validation with a cyclic learning rate scheduler (low: 0.0001, high: 0.01) to escape local minima. The final model is retrained on the full dataset after CV confirms generalization. The training pipeline lives in ml/ and runs offline in Python.
- Real-time posture scoring at ~10 FPS
- Color-coded feedback: green (good) / yellow (warning) / red (poor)
- Skeleton overlay drawn on the live webcam canvas
- Configurable score threshold with audio alerts via Web Audio API
- Session analytics with interactive Chart.js line graph
- CSV export of session data
- Complete privacy β webcam feed never leaves the device
| Layer | Technology | Why |
|---|---|---|
| Frontend | HTML5, CSS3, JavaScript (ES Modules), Chart.js | No framework overhead; runs anywhere with a browser |
| ML β Browser | MediaPipe Tasks-Vision JS (pose detection), Custom Dense Neural Network (929 params, 4 layers, plain JS forward pass) | Real-time inference without server round-trip; WebGL acceleration |
| ML β Training | Python, TensorFlow/Keras, scikit-learn, MediaPipe Python | Offline training pipeline in ml/; model weights exported to JS |
| Deployment | Any static hosting (GitHub Pages, Netlify, Vercel) | No server required β zero cost, zero maintenance |
Open the deployed URL in any modern browser with a webcam:
Live: https://sitright-zz9r.onrender.com
Or serve it locally:
cd SitRight/frontend
python3 -m http.server 8080
# Open http://localhost:8080git clone https://github.com/arnavdeepaware/SitRight.git
cd SitRight/frontend
python3 -m http.server 8080
# Open http://localhost:8080 in a browser with camera accessNote: The
ml/directory contains the Python training pipeline for retraining the model. Seeml/sequential.pyfor the training script andml/Predictor.pyfor feature engineering. Retraining requires Python 3.11, TensorFlow, and MediaPipe Python.
SitRight/
βββ frontend/
β βββ index.html # Main app (webcam, score, chart)
β βββ landing.html # Landing page
β βββ script.js # Video pipeline, MediaPipe integration, charts
β βββ posture-model.js # Neural network weights + forward pass (browser inference)
β βββ styles.css / landing.css
β βββ assets/
βββ backend/ # v1 server-side architecture (retained for reference)
β βββ websocket_server.py # aiohttp server: HTTP + WebSocket on one port
β βββ pose_detector.py # MediaPipe landmark extraction (Python)
β βββ requirements.txt
βββ ml/
β βββ Predictor.py # Feature engineering + model inference
β βββ sequential.py # Model training (K-fold CV, cyclic LR)
β βββ model/
β βββ posture_model.h5 # Trained Keras model
β βββ scaler.pkl # MinMaxScaler fitted on training data
βββ Dockerfile # v1 deployment (retained for reference)
βββ docker-compose.yml
βββ .env.example
The entire application is static files. Deploy the frontend/ directory to any static hosting provider:
GitHub Pages
# Push to a gh-pages branch, or configure Pages to serve from frontend/Netlify / Vercel
Build command: (none)
Publish directory: frontend/
No environment variables, no build step, no server process. The app loads MediaPipe from a CDN and runs all inference in the browser.
The migration from server-side to browser-side inference was motivated by production data from v1:
| Metric | v1 (Server) | v2 (Browser) |
|---|---|---|
| Inference latency | ~400-600ms (including network) | <50ms |
| Hosting cost | $7+/month (Render Starter) | Free (static hosting) |
| Privacy | Frames sent to server | Frames never leave device |
| Concurrent users | 1 (MediaPipe not thread-safe) | Unlimited (each user runs locally) |
| Deployment complexity | Docker + process management | Upload static files |
Model details: 929 parameters across 4 dense layers. 7 input features, MinMax-scaled, with weights hardcoded directly in posture-model.js. The forward pass is a plain JS matrix multiply β no ML framework loaded at runtime. MediaPipe JS handles pose detection with WebGL GPU acceleration.
The following metrics were captured during local testing after migration:
| Metric | Observed Value |
|---|---|
| Per-frame inference latency | <50ms (vs ~500ms round-trip in v1) |
| MediaPipe pose detection (WebGL) | ~20-35ms |
| Neural network forward pass (plain JS) | <1ms |
| Effective frame rate | ~10 FPS (throttled via requestAnimationFrame) |
| First-load time (MediaPipe model download) | ~2-5 seconds (CDN, cached after first run) |
| Subsequent load time | <500ms (browser cache) |
| Lag accumulation over session | None β no buffer to fill |
-
Coordinate normalization mismatch (silent wrong output) β The Python training pipeline had a non-obvious coordinate transform: MediaPipe returns landmarks in normalized [0,1] space,
pose_detector.pymultiplied them by frame pixel dimensions to get pixel coordinates, thenPredictor.pydivided by a hardcodedvideo_width=100, video_height=100β not the actual frame dimensions. This means the model was trained on features computed from coordinates in roughly the 0β6.4 range, not [0,1]. Porting this incorrectly would produce wrong scores with no error or warning. Catching it required tracing the full data path from MediaPipe output to model input and running the JS implementation against a known Python test case (score: 88) before any other code was written. -
Model architecture undocumented in source β The training script (
ml/sequential.py) and original README both described a 5-layer network (32β16β8β4β1). The actual saved.h5was a 4-layer network (32β16β8β1). These differ by one layer and ~36 parameters. The discrepancy was caught by extracting and counting weights from the file directly. The.h5is ground truth; comments and docs were wrong. -
ES module vs. global scope β MediaPipe JS requires
importsyntax, which forcesscript.jsto be atype="module"file. Modules have their own scope β globals declared in one module aren't automatically visible in another. Butposture-model.js(loaded as a plain<script>before the module) needs to exposeextractFeatures()andpredictPosture()toscript.js. The fix was intentional: loadposture-model.jsas a regular script (attaches towindow), then loadscript.jsas a module. Both files share the global scope, which is the correct behavior here sinceposture-model.jsis effectively a statically-linked dependency with no imports of its own. -
Weight portability tradeoff β Hardcoding 929 parameters directly in JS eliminates a model-loading step and removes any runtime dependency on TensorFlow.js or ONNX. The tradeoff is that retraining the model requires re-extracting the weights and updating
posture-model.js. At this model size that's a 30-second Python script, not a real concern β but it's the wrong pattern for a model that changes frequently.
The migration confirmed that the bottleneck in v1 was never compute β it was the network. Moving 640Γ480 JPEG frames from a browser to a cloud VM and back at 10 FPS generates ~1.2MB/s of traffic and adds 80-150ms of irreducible latency regardless of how fast the server processes the frame. On local hardware, the same inference took ~30ms end-to-end. That 16Γ difference is purely architectural.
The more transferable lesson: model size determines what deployment options are viable. At 929 parameters (3.6KB of weights), this model can be hardcoded in JavaScript, hosted on a CDN, and run in a browser tab. At 50MB, it becomes a file to download. At 500MB, it needs a server. Choosing the right model size for the deployment target is a design decision, not just a training one.
The original architecture used a Python backend running MediaPipe and TensorFlow on a cloud server. The browser captured webcam frames, Base64-encoded them, sent them over WebSocket to the server, which ran inference and returned annotated frames with posture scores.
This section documents what was learned from deploying v1 in production. These findings directly motivated the migration to v2.
Browser Server (Python / aiohttp)
ββββββββββββββββββββββββ WebSocket βββββββββββββββββββββββββββββββββ
β getUserMedia() βββββββββββββββΆβ Frame Handler β
β Canvas (640Γ480) β base64 JPEG β β
β Score Widget β β βββββββββββββββββββββββββββ β
β Chart.js Analytics ββββββββββββββββ β MediaPipe Pose β β
ββββββββββββββββββββββββ score + β β 26 landmark extraction β β
ann. frame β ββββββββββββββ¬βββββββββββββ β
β β β
β ββββββββββββββΌβββββββββββββ β
β β Feature Engineering β β
β β 7 geometric features β β
β β (angles, distances, β β
β β ratios) β β
β ββββββββββββββ¬βββββββββββββ β
β β β
β ββββββββββββββΌβββββββββββββ β
β β Keras Classifier β β
β β 32β16β8β4β1 neurons β β
β β MinMax scaled input β β
β βββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββ
The following metrics were captured from production logs during a live session:
| Metric | Observed Value |
|---|---|
| TensorFlow inference per frame | ~97-105ms |
Total model.predict() call time |
~195-200ms |
| Estimated network round trip (US East) | ~80-150ms |
| Effective server-side frame rate | ~1.5-2 FPS |
| End-to-end perceived latency | ~400-600ms |
-
model.predict()overhead (~2x inference cost) β Keras'smodel.predict()is designed for batched dataset evaluation. On each single-sample inference call it allocates a result accumulator, runs progress callbacks, and renders a progress bar, doubling the actual compute time. This was caught by reading the production logs and noticing the two timing values Keras prints. The fix was replacingmodel.predict(x)withmodel(x, training=False).numpy()β a direct forward pass with no batch infrastructure overhead. -
Network round-trip is irreducible at this tier β Each webcam frame travels from the user's browser to Render's US datacenter and back. Even with minimal processing, this adds 80-150ms per frame that does not exist in local execution. Combined with inference time, the server cannot keep up with the 10 FPS the client sends, causing frames to queue in the WebSocket buffer and lag to compound over time.
-
RAM ceiling on free tier β TensorFlow + MediaPipe together consume ~450-510MB at runtime. Render's free tier allocates 512MB, leaving virtually no headroom. The container was prone to OOM kills during inference spikes. Upgrading to the Starter tier (768MB) resolved stability.
-
Protobuf deprecation warnings at inference frequency β
drawing_styles.get_default_pose_landmarks_style()internally callsSymbolDatabase.GetPrototype(), a deprecated protobuf API. This fires on every frame and cannot be suppressed without patching the mediapipe internals or upgrading to a version that removed thesolutionsAPI entirely (which breaks the rest of the codebase). This is a dependency version trap: mediapipe >= 0.10.15 removessolutions, but earlier versions carry this warning.
The server-side inference architecture made sense during the hackathon: Python has the most mature MediaPipe and TensorFlow tooling, the model was already in Keras .h5 format, and getting a working demo deployed quickly was the priority.
In production, the core problem was structural: sending raw video frames over a network to run inference and send annotated frames back is doing three expensive operations (encode, transmit, transmit back) that add latency before the user sees any result. On hardware I control (my laptop), total frame processing was ~30ms. On a shared cloud CPU behind a network, it was ~500ms. The gap is not a hosting tier problem β it's an architectural one.
This analysis led directly to v2: move inference to the browser, eliminate the network entirely, and deploy as a static site. The model is small enough (929 parameters) that a plain JS forward pass is faster than even the Python inference was on local hardware.
- HTTPS required in production β browsers block
getUserMediaon non-secure origins - Lighting sensitivity β MediaPipe landmark confidence degrades in low light or with glasses/hats occluding the face
- Fixed camera angle assumption β the model was trained on front-facing webcam data; steep angles (looking down at a laptop) reduce accuracy
MIT
