GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

Abstract

Understanding realistic hand-object interactions from monocular RGB videos is essential for AR/VR, robotics, and embodied AI. Existing methods rely on category-specific templates or heavy computation, yet still produce physically inconsistent hand-object alignment in 3D. We introduce GHOST (Gaussian Hand-Object Splatting), a fast, category-agnostic framework for reconstructing dynamic hand-object interactions using 2D Gaussian Splatting. GHOST represents both hands and objects as dense, view-consistent Gaussian discs and introduces three key innovations: (1) a geometric-prior retrieval and consistency loss that completes occluded object regions, (2) a grasp-aware alignment that refines hand translations and object scale to ensure realistic contact, and (3) a hand-aware background loss that prevents penalizing hand-occluded object regions. GHOST achieves complete, physically consistent, and animatable reconstructions from a single RGB video while running an order of magnitude faster than prior category-agnostic methods. Extensive experiments on ARCTIC, HO3D, and in-the-wild datasets demonstrate state-of-the-art accuracy in 3D reconstruction and 2D rendering quality.

Results

Input Video

Drill

Mixer

Mug

Waffle Iron

GHOST Reconstruction

Drill

Mixer

Mug

Waffle Iron

Method

GHOST pipeline: Preprocessing (segmentation, SfM, hand reconstruction, geometric prior retrieval), optimization, and 2D Gaussian Splatting with hand-object losses.

Qualitative Results

Quantitative Results

3D Reconstruction Metrics (ARCTIC Bi-CAIR)

MPJPE in mm, CD in cm². Green = best, orange = second best.

2D Rendering Quality

GHOST achieves 13-16x faster runtime while maintaining competitive or superior rendering quality.

Citation