Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

ECCV 2026

Siddhant Bansal¹  ·  Zhifan Zhu¹  ·  Shashank Tripathi²  ·  Jiahe Zhao¹  ·  Michael J. Black²  ·  Dima Damen^1
1University of Bristol     ²Max Planck Institute for Intelligent Systems, Tübingen

Paper  |  Project Page  |  Download EPIC-Contact Dataset  |  Models  |  EC-fit

HOPformer is an end-to-end transformer that jointly predicts bi-manual MANO hand meshes, object pose, and object class from a single RGB image in one forward pass. A cross-attention decoder conditions object features on strong data-driven hand priors (WiLoR), making pose estimation robust to the heavy occlusion and ambiguous contact typical of in-the-wild egocentric video. Alongside the model we release EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand–object contact correspondences and posed meshes — the latter produced by EC-fit, our contact-based hand–object fitting pipeline, which we also release. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over the prior state of the art) and, on EPIC-Contact, nearly doubles the success rate while reducing contact deviation by 75%.

The codebase supports two datasets:

• ARCTIC — lab-captured bimanual manipulation of articulated objects.
• EPIC-Kitchens — egocentric, in-the-wild hand–object interaction (rigid objects).

---

Installation

1. Environment

Tested with Python 3.10.13, PyTorch 2.5.1, CUDA 12.6. Dependencies are captured in two files at the repo root: environment.yml (conda — canonical) and requirements.txt (pip-only pins). Create the environment with either:

# Option A — conda (recommended; reproduces the tested CUDA 12.6 / torch 2.5.1 setup)
conda env create -f environment.yml
conda activate hopformer

# Option B — pip / venv (install the CUDA-12.6 torch build first, then the rest)
python3.10 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install torch==2.5.1 torchvision --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt

A small script exports the dataset / body-model download credentials and (optionally) Comet experiment-logging variables — data and log directories are not set here (the code uses repo-relative ./data and ./logs). Copy the template, fill in your own values, and source it before any run:

cp setup_variables.sh.example setup_variables.sh   # then edit in your credentials
source setup_variables.sh                          # required before any run

setup_variables.sh itself is gitignored — never commit real credentials. The ARCTIC_/SMPLX_/MANO_USERNAME/PASSWORD values are used to download the datasets and body models (below); the COMET_* values are optional and only needed if you opt into Comet experiment tracking (off by default — see Training).

2. Patch smplx

Modify the smplx package to return 21 hand joints instead of 16. Uncomment line 1681 in smplx/body_models.py:

joints = self.vertex_joint_selector(vertices, joints)

To locate the file:

python -c "import smplx, os; print(os.path.join(os.path.dirname(smplx.__file__), 'body_models.py'))"

3. MANO (and SMPL-H) body models

The code loads the MANO hand model from ./data/body_models/mano/ (common/body_models.py). Register at https://mano.is.tue.mpg.de (and https://smpl-x.is.tue.mpg.de for SMPL-H, used by ARCTIC), then place these files under ./data/body_models/mano/:

data/body_models/mano/
├── MANO_LEFT.pkl
├── MANO_RIGHT.pkl
├── mano_mean_params.npz          # mean MANO params (from WiLoR — see wget below)
├── SMPLH_male.pkl                # ARCTIC only
└── SMPLH_female.pkl              # ARCTIC only

With your credentials set in setup_variables.sh, ARCTIC's downloader can fetch the MANO/SMPL-X archives for you (scripts_data/download_data.py, see the ARCTIC body-model setup); otherwise download MANO_LEFT.pkl/MANO_RIGHT.pkl (and SMPLH_*) manually from the sites above. mano_mean_params.npz is not part of the MANO release — grab it from WiLoR:

wget https://raw.githubusercontent.com/rolpotamias/WiLoR/main/mano_data/mano_mean_params.npz \
    -P data/body_models/mano/

Cite MANO if you use the hand model.

4. WiLoR hand features

HOPformer uses WiLoR ViT features for the left/right hand crops. The WiLoR model code is already vendored in src/models/wilor/, so you do not need to clone or pip install WiLoR. You only need two files from WiLoR's own releases — the weights and the model config:

mkdir -p pretrained_models/wilor
# model config (small file, from the WiLoR GitHub repo):
wget https://raw.githubusercontent.com/rolpotamias/WiLoR/main/pretrained_models/model_config.yaml \
    -P pretrained_models/wilor/
# weights (from the WiLoR HuggingFace space):
wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/wilor_final.ckpt \
    -P pretrained_models/wilor/

Runs default to --wilor_default_dir ./pretrained_models/wilor, so if you place the two files there no flag is needed; otherwise point --wilor_default_dir at the folder containing them. (You do not need WiLoR's detector.pt — HOPformer crops the hands itself.) WiLoR also reuses the MANO mean params at ./data/body_models/mano/mano_mean_params.npz (see the MANO setup above).

WiLoR is released under CC-BY-NC-ND (non-commercial, no-derivatives). Obtain its weights/config from the links above and cite WiLoR if you use these features.

---

Datasets

ARCTIC

Refer to the upstream ARCTIC repository for dataset download, setup, and splits: https://github.com/zc-alexfan/arctic (see its docs/data/README.md).

Expected local layout (repo-relative):

./data/arctic_data/

EPIC-Kitchens

The EPIC-Contact data is released as the EPIC-Contact dataset (CC BY-NC 4.0, gated). The training pickles + object meshes are on Hugging Face; the RGB frames + hand crops (EPIC-Kitchens derived) are distributed separately as a full-bundle zip on OneDrive. Download both and place them under ./data/epic_data/:

# 1) Pickles + object meshes — from the HF dataset (the hopformer_repro_pkls/ pack):
hf download Sid2697/epic-contact --repo-type dataset --include "hopformer_repro_pkls/*" \
  --local-dir ./data/_epic_dl
mv ./data/_epic_dl/hopformer_repro_pkls/training_pkls         ./data/epic_data/training_pkls
mv ./data/_epic_dl/hopformer_repro_pkls/kps_mesh_3d_bb_v2.pkl ./data/epic_data/kps_mesh_3d_bb_v2.pkl

# 2) RGB frames + hand crops — download the OneDrive full-bundle zip (link above), unzip, then:
mv <unzipped>/source_rgb/epicgrasps_storage ./data/epic_data/epicgrasps_storage

Expected local layout:

./data/epic_data/training_pkls/         # train/test/keys pickles
./data/epic_data/kps_mesh_3d_bb_v2.pkl  # object meshes / keypoints / bbox
./data/epic_data/epicgrasps_storage/    # RGB frames + 256² hand crops (from OneDrive)

See the dataset's DATASET.md (on the HF page) for the full schema, camera convention, and contact format.

---

Pretrained models

The checkpoints are on the Hugging Face Hub (gated, CC BY-NC 4.0): https://huggingface.co/Sid2697/HOPformer. Accept the terms on the model page, then download just the checkpoints into release_models/ (the repo also holds large predictions/ tars — see Evaluation — so download selectively):

hf download Sid2697/HOPformer --include "*.ckpt" --local-dir release_models

This gives one checkpoint per training stage:

File	Stage	Use it for
release_models/p1_exo_epoch22.ckpt	ARCTIC p1 (exocentric / allocentric)	initialising p2 training (--load_ckpt); p1 extract + eval
release_models/p2_arctic_ego_epoch8.ckpt	ARCTIC p2 (egocentric)	ARCTIC extract + eval
release_models/epic_epoch125.ckpt	EPIC-Contact (egocentric)	EPIC extract + eval

To run inference/eval with a downloaded model, pass its path to --load_ckpt (e.g. --load_ckpt release_models/p2_arctic_ego_epoch8.ckpt) in place of the logs/<exp>/checkpoints/last.ckpt shown in the examples below.

---

Training

Run source setup_variables.sh first. All scripts are launched from the repo root.

The recipe is a chain: p1 (exocentric) → p2 (ARCTIC egocentric, from p1) → EPIC (egocentric, from p2). --batch_size is per-GPU; the paper trains on 4 GPUs (p1 = 64×4 = 256; p2/EPIC = 32×4 = 128). The LR uses a linear warm-up from 1e-7 to the peak over the first 5% of steps, then cosine decay back to 1e-7; the released checkpoint is the best-val epoch (early stopping).

GPUs: training uses all visible GPUs via DDP (devices="auto"), so launch on a 4-GPU node — or allocate 4 GPUs through your cluster scheduler (e.g. a SLURM script) — to reproduce the paper's batch totals. Logging: training writes TensorBoard + CSV logs to logs/<exp>. Weights & Biases and Comet are optional experiment trackers — for W&B set WANDB_MODE=offline (as the SLURM scripts do) or disabled to skip it; Comet is opt-in (off unless COMET_API_KEY + COMET_WORKSPACE are set).

ARCTIC — allocentric (p1)

python scripts_method/train.py --setup p1 --method transformer_sf \
  --trainsplit train --valsplit smallval \
  --optimizer adamw --lr_scheduler cosine --lr 5e-5 \
  --backbone vit-g --freeze_backbone --decoder_depth 12 \
  --batch_size 64 --num_epoch 25 \
  --dataset arctic

ARCTIC — egocentric (p2), initialised from the p1 checkpoint

python scripts_method/train.py --setup p2 --method transformer_sf \
  --trainsplit train --valsplit smallval \
  --optimizer adamw --lr_scheduler cosine --lr 3e-5 \
  --backbone vit-g --freeze_backbone --decoder_depth 12 \
  --batch_size 32 --num_epoch 30 \
  --load_ckpt release_models/p1_exo_epoch22.ckpt --precision 32 \
  --dataset arctic

EPIC-Kitchens (egocentric, p2), initialised from the ARCTIC p2 checkpoint

python scripts_method/train.py --setup p2 --method transformer_sf \
  --trainsplit train --valsplit smallval \
  --optimizer adamw --lr_scheduler cosine --lr 3e-5 \
  --backbone vit-g --freeze_backbone --decoder_depth 12 \
  --batch_size 32 --precision 32 --num_epoch 125 \
  --load_ckpt release_models/p2_arctic_ego_epoch8.ckpt \
  --dataset epic

Fast dev run (single-batch sanity check)

python scripts_method/train.py --method transformer_sf --dataset arctic --setup p1 -f

---

Extracting predictions

Extraction writes per-sample predictions to logs/<exp>/eval (ARCTIC) or logs/<exp>/eval_epic (EPIC); pass that directory to --eval_p in Evaluation below. To use a downloaded checkpoint, substitute its path (e.g. release_models/p2_arctic_ego_epoch8.ckpt) into --load_ckpt.

ARCTIC — allocentric (p1)

python scripts_method/extract_predicts.py --setup p1 --method transformer_sf \
  --load_ckpt logs/<exp>/checkpoints/last.ckpt --run_on val \
  --extraction_mode eval_pose --backbone vit-g --decoder_depth 12

ARCTIC — egocentric (p2)

# --precision 32 is required: FP16 can overflow the hand head on egocentric checkpoints.
python scripts_method/extract_predicts.py --setup p2 --method transformer_sf \
  --load_ckpt logs/<exp>/checkpoints/last.ckpt --run_on val --precision 32 \
  --extraction_mode eval_pose --backbone vit-g --decoder_depth 12

EPIC-Kitchens

python scripts_method/extract_predicts_epic.py --setup p2 --method transformer_sf \
  --load_ckpt logs/<exp>/checkpoints/last.ckpt --run_on val \
  --extraction_mode eval_pose --backbone vit-g --decoder_depth 12

---

Evaluation

Point --eval_p at the directory produced by the extraction step.

ARCTIC — allocentric (p1)

python scripts_method/evaluate_metrics.py --task pose \
  --eval_p logs/<exp>/eval --split val --setup p1 --dataset arctic

ARCTIC — egocentric (p2)

python scripts_method/evaluate_metrics.py --task pose \
  --eval_p logs/<exp>/eval --split val --setup p2 --dataset arctic

EPIC-Kitchens

python scripts_method/evaluate_metrics.py --task pose \
  --eval_p logs/<exp>/eval_epic --dataset epic

Reproduce the paper numbers from released predictions

For each model we release its per-sample prediction dump on the validation split — the exact outputs extract_predicts produced — so you can reproduce the reported metrics by running only the evaluator on the dump, without re-running the model. The p2 ARCTIC and EPIC dumps are on the Hub under predictions/ (gated — accept the terms first); the p1 dump is too large to host (~275 GB), so reproduce p1 by running the extraction step yourself. Each dump corresponds to one paper table:

Dump	paper	Eval command	Expected headline metrics
p1_arctic_eval.tar (not hosted — extract locally)	ARCTIC exocentric (Supp.)	evaluate_metrics.py --task pose --eval_p <dir> --split val --setup p1 --dataset arctic	MPJPE 12.7 · CDev 24.5 · MDev 5.4 · AAE 4.8 · SR@0.05 90.3 · Cls 99.7
p2_arctic_eval.tar	Table 1 (ARCTIC ego)	evaluate_metrics.py --task pose --eval_p <dir> --split val --setup p2 --dataset arctic	MPJPE 16.1 · CDev 31.9 · MDev 7.3 · AAE 5.0 · SR@0.05 82.4 · Cls 99.5
epic_eval.tar	Table 2 (EPIC)	evaluate_metrics.py --task pose --eval_p <dir> --dataset epic	MPJPE 19.9 · CDev 20.7 · MDev 11.4 · ACC 2.5/4.1 · SR@0.05 29.8 · Cls 52.9

# download the p2 / EPIC dumps from the Hub (gated — accept terms first), then evaluate one:
hf download Sid2697/HOPformer --include "predictions/*" --local-dir .
tar -xf predictions/p2_arctic_eval.tar         # -> a folder of per-sample predictions
python scripts_method/evaluate_metrics.py --task pose \
  --eval_p <untarred_dir> --split val --setup p2 --dataset arctic

The printed values should match the table above (and the shipped reference results/*agg_metrics.json). Which JSON key is which paper column differs by dataset (EPIC objects are symmetric, so its CDev/MDev/ACC use the symmetric variants and SR@0.05 is ADD-S–based):

• ARCTIC: CDev = cdev/ho, MDev = mdev/h, SR@0.05 = success_rate/0.05, MPJPE = mpjpe/ra/h, AAE = aae, Cls = obj_cls_acc.
• EPIC: CDev = cdev_sym/ho, MDev = mdev_sym/h, ACC h/o = acc/h / acc/o_sym, SR@0.05 = adds_success/0.05, MRRPE·ro = mrrpe/r/o, MPJPE = mpjpe/ra/h, Cls = obj_cls_acc.

The expected numbers in the table above are the reference (rounded to the paper's precision); the full per-metric values are in the results/*agg_metrics.json files on the Hub.

Note on EPIC acceleration metrics: EPIC clips are not uniformly sampled (mixed native fps and dropped frames), so acc_err_pose uses a fixed 15 fps convention and is a relative smoothness score only — not physical m/s². See the comment in src/utils/eval_modules.py.

---

Citation

If you use HOPformer or EPIC-Contact, please cite:

@inproceedings{bansal2026hopformer,
  title     = {Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation},
  author    = {Bansal, Siddhant and Zhu, Zhifan and Tripathi, Shashank and
               Zhao, Jiahe and Black, Michael J. and Damen, Dima},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

References

• EPIC-Grasps / EPIC-Kitchens — source videos and stable-grasp segments.
• WiLoR — per-frame hand features used by the model.
• ContactEdit — contact-axis parametrisation for bijective contact transfer.
• JointTransformer — baseline and previous SOTA on ARCTIC.
• ARCTIC (project page) — this codebase builds on the ARCTIC dataset and baselines.

License

HOPformer code is released under CC BY-NC 4.0 (non-commercial, attribution) — see LICENSE. Bundled third-party components keep their own licenses: the vendored WiLoR code (src/models/wilor/) is CC BY-NC-ND 4.0, and MANO and ARCTIC retain their respective non-commercial terms. WiLoR weights and the MANO model are not distributed here and must be obtained from their sources (see Installation).

Contact

For technical questions, please open an issue.