RehabSystem — Visual Movement Amplification

A real-time upper-limb rehabilitation application that uses an Intel RealSense depth camera, MoveNet pose estimation, and a skinned 3-D avatar to deliver Visual Movement Amplification (VMA) feedback. The avatar exaggerates the patient's impaired-arm motion by a configurable gain factor, providing an augmented visual signal that has been shown in motor rehabilitation research to promote neuroplasticity and re-engagement of impaired limbs.

---

GUI Overview

The interface is divided into three horizontal panels inside a resizable splitter.

---

Left panel — Pose Overlay

A live colour feed from the RealSense camera with the detected 2-D skeleton drawn on top. The right arm is rendered with green bones and red keypoints; the left arm uses cyan bones and orange keypoints. Each keypoint is labelled with its anatomical name (Shoulder / Elbow / Wrist) and its measured depth in metres. This panel always shows the real, un-amplified movement so the clinician can verify detection quality independently of the gain setting.

---

Centre panel — 3-D Avatar

A skinned FBX character mesh rendered in Qt 3D. The avatar replicates the patient's posture in real time, but with the impaired arm's joint displacements multiplied by the current gain factor. The camera is orbitable (left-drag to rotate, scroll to zoom, right-drag to pan) via Qt's built-in orbit controller. The warm directional lighting (three-point setup) is fixed to the scene so the avatar reads clearly from any viewing angle.

---

Right panel — Control Panel

A scrollable dark-themed sidebar divided into four sections:

Visual Movement Amplification (VMA)

• Gain readout — large ×1.50-style display in bright blue on a dark navy tile; updates instantly as the gain changes.
• Gain spin-box — precise numeric entry, range 1.00–5.00 in 0.01 steps.
• Gain slider — horizontal track with tick marks at 1×, 2×, 3×, 4×, 5×; stays synchronised with the spin-box bidirectionally.
• Impaired hand selector — drop-down (Left / Right); switching it redirects amplification to the chosen arm and rebuilds the task list with the correct side badges.

Live Elbow Range-of-Motion

A full-width progress bar showing the current elbow flexion angle in degrees (0° = fully extended, ~150° = fully flexed). The bar updates every frame (~60 Hz) and is colour-coded green-to-dark-green so a clinician can read the arc at a glance without looking at the number.

Session

• Elapsed clock — six-digit LCD display (MM:SS) that counts up while the session is running.
• Start / Pause / Resume button — blue when idle, amber when paused. Clicking toggles the session state and relabels the button accordingly.
• Reset button — red; stops the clock, zeroes all rep counts, and returns the task list to the first exercise.

Exercise Tasks

One card per exercise in the current protocol. Cards are styled differently for the active task (bright sky-blue border, lighter background) versus completed or upcoming tasks (muted border, darker background).

Each card contains:

• Task name — bold white text (active) or muted grey (inactive).
• Side badge — red pill for the impaired hand, sky-blue pill for the healthy hand.
• Rep counter — completed / target right-aligned (e.g. 4 / 10).
• Rep progress bar — thin cyan-to-teal bar; fills as reps are completed.
• Live range bar — amber-to-orange bar showing the real-time elbow arc; visible only on the active task card, hidden on all others.

When the target rep count is reached the system automatically advances to the next task, re-highlighting the new active card.

---

How it works

RealSense D-series  →  Colour + Depth frame
        │
        ▼
  MoveNet (ONNX)    →  17 × 2-D keypoints  (192 × 192 px, ~30 ms)
        │
        ▼
  Depth lift        →  shoulder / elbow / wrist  →  Keypoint3D  (x, y, z metres)
        │
        ├──── Pose overlay widget  (real skeleton drawn on colour feed, no gain)
        │
        └──── VMA amplifier
                  avatar_joint = shoulder + (real_joint − shoulder) × gain
                       │
                       ▼
              Skeleton retargeter  →  IK-style direct bone rotation
                       │
                       ▼
             Skinned FBX mesh  (Assimp + Qt 3D, CPU linear blend skinning)

The healthy arm always renders at 1× gain. Only the designated impaired side is amplified.

---

Features

Area	Detail
VMA gain	Adjustable 1.0×–5.0× via slider and spin-box, mirrored in a large live readout
Impaired hand selector	Left / Right; switches which arm receives amplification
Live elbow ROM bar	Real-time arc-of-motion in degrees (0° = extended)
Session clock	MM:SS elapsed timer with Start / Pause / Resume
Exercise task list	Protocol of named tasks (Elbow Flexion, Shoulder Abduction, …) with per-task target reps, completed rep count, and a live range bar
Automatic rep counting	Hysteresis-based angle-threshold counter; advances to the next task when target is reached
Pose overlay	Colour camera feed with 2-D skeleton drawn on top — right arm in green/red, left in cyan/orange, with depth labels
Dark UI	Navy-dark control panel; all colours chosen for clinical lighting

---

Project structure

RehabSystem/
├── main.cpp                  Entry point
├── RehabSystem.h / .cpp      Main window — VMA logic, UI, session management
├── Scene3D.h / .cpp          Qt 3D scene, bone retargeting, CPU skinning
├── Skeleton.h / .cpp         Bone hierarchy and global transform update
├── AssimpLoader.h / .cpp     FBX loader (Assimp) → Skeleton + MeshData
├── PoseEstimator.h / .cpp    MoveNet ONNX inference + depth lift
├── PoseOverlayWidget.h       QWidget — colour feed with 2-D skeleton overlay
├── DepthFilter.h / .cpp      RealSense depth post-processing pipeline
├── RealSenseWorker.h / .cpp  QThread worker — frame capture loop
├── RehabSystem.vcxproj       MSVC / Qt VS Tools project file
├── RehabSystem.qrc           Qt resources
├── RehabSystem.ui            (reserved — layout driven in code)
└── model/
    ├── model.onnx            MoveNet Lightning/Thunder ONNX model
    └── Body Block.fbx        Skinned character mesh (Mixamo rig)

---

Dependencies

Library	Version tested	Purpose
Qt	6.5	Widgets, Qt 3D (Core / Render / Extras), threading
Intel RealSense SDK 2	2.55+	Colour + depth capture, depth filters
ONNX Runtime	1.17	MoveNet inference
Assimp	5.3	FBX model loading and skinning data extraction

Default install paths expected by the project (editable in the .vcxproj):

C:\Program Files\Assimp\
C:\Program Files\RealSense SDK 2.0\
<repo root>\libs\onnx\

---

Build requirements

• OS: Windows 10/11 x64
• Compiler: MSVC v145 (Visual Studio 2022)
• Qt VS Tools: extension for Visual Studio (QtVS_v304)
• Windows SDK: 10.0

Steps

1. Install Visual Studio 2022 with the Desktop development with C++ workload.
2. Install the Qt VS Tools extension and register a Qt 6.5 installation named Qt 6.5 inside Extensions → Qt VS Tools → Qt Versions.
3. Install Assimp and the RealSense SDK to their default paths, or update AdditionalIncludeDirectories / AdditionalLibraryDirectories in the .vcxproj.
4. Place the ONNX Runtime headers and onnxruntime.lib under libs/onnx/.
5. Open RehabSystem.sln (or the .vcxproj directly) in Visual Studio 2022.
6. Select Debug | x64 or Release | x64 and build (Ctrl+Shift+B).

Note: The Release configuration currently lists only core;gui;widgets Qt modules in the .vcxproj. If you build Release with the 3-D avatar, add 3dcore;3drender;3dextras to the QtModules property for that configuration.

---

Runtime requirements

• The model/ folder must sit one level above the executable (i.e. ../model/ relative to the .exe). This matches the paths hard-coded in AssimpLoader and PoseEstimator:

  ../model/Body Block.fbx
  ../model/model.onnx
  

• An Intel RealSense D-series camera (D415, D435, D455, or compatible) must be connected via USB 3.
• The following DLLs must be on PATH or beside the .exe at runtime: realsense2.dll, onnxruntime.dll, assimp-vc145-mt.dll, and all Qt 6.5 deployment DLLs (windeployqt handles the Qt side).

---

Key parameters (compile-time)

Constant	Location	Default	Effect
kAlpha	RehabSystem.h	0.3f	EMA smoothing — higher = more responsive, less smooth
MODEL_SIZE	PoseEstimator.h	192	MoveNet input resolution (192 = Lightning, 256 = Thunder)
confThresh	PoseEstimator.cpp	0.25f	Minimum keypoint confidence to accept a detection
Gain range	RehabSystem.cpp	1.0 – 5.0	Adjust setRange() on m_gainSpinBox / m_gainSlider
Rep hysteresis low	RehabSystem.cpp	35 % of rangeGoalDeg	Fraction at which a rep completion is registered on return

---

Architecture notes

Pose pipeline thread safety. The RealSense capture runs in a dedicated QThread (RealSenseWorker). Frames are delivered to the main thread via a Qt::QueuedConnection signal, so all Qt widget and Qt 3D updates happen on the GUI thread without explicit locking.

VMA amplification. amplifyPose() applies the gain in camera/pixel space before the skeleton retargeter maps directions into bone rotations. This means the gain is perceptually uniform regardless of the avatar's rest-pose scale.

Skinning. CPU linear blend skinning is performed every frame in Scene3D::applySkinning() and the result is uploaded to the Qt 3D vertex buffer. For a production deployment, this should be moved to a compute shader or Qt 3D's GPU skinning pipeline to reduce CPU load.

Rep counting. The current implementation counts elbow-flexion arcs using a two-threshold hysteresis on the raw elbow angle (shoulder–elbow–wrist dot product). It is intentionally simple — a windowed extrema detector or DTW-based template matcher would be more robust for complex multi-joint exercises.

---

Acknowledgements

• MoveNet pose estimation model by Google, distributed under the Apache 2.0 licence.
• Character mesh rigged with Mixamo (Adobe).
• Depth post-processing via the Intel RealSense SDK 2.0.