Physical Simulation for Vision-based Kinematic Tracking

Marek Vondrak1 Leonid Sigal2 Odest C. Jenkins1

1Brown University 2Disney Research, Pittsburgh


Dynamical Tracking Screenshot

We propose a simulation-based dynamical motion prior for tracking human motion from video in presence of physical and environmental interactions. Most tracking approaches to date have focused on efficient inference algorithms and/or learning of prior kinematic motion models; however, few can explicitly account for physical plausibility of recovered motion. Here, we aim to recover physically plausible motion of a single articulated human subject. Towards this end, we propose a full-body 3D physical simulation-based prior that explicitly incorporates a model of human dynamics into the Bayesian filtering framework. We consider the motion of the subject to be generated by a feedback “control loop” in which Newtonian physics approximates the rigid-body motion dynamics of the human and the environment through the application and integration of interaction forces, motor forces and gravity. Interaction forces prevent physically impossible hypotheses, enable more appropriate reactions to the environment (e.g., ground contacts) and are produced from detected human-environment collisions. Motor forces actuate the body, ensure that proposed pose transitions are physically feasible and are generated using a motion controller. For efficient inference in the resulting high-dimensional state space, we utilize an exemplar-based control strategy that reduces the effective search space of motor forces. As a result, we are able to recover physically-plausible motion of human subjects from monocular and multi-view video. We show, both quantitatively and qualitatively, that our approach performs favorably with respect to Bayesian filtering methods with standard motion priors.


M. Vondrak, L. Sigal, and O. Jenkins, “Physical Simulation for Probabilistic Motion Tracking,” in Computer Vision and Pattern Recognition (CVPR 2008), Anchorage, AK, USA, 2008, pp. 1-8.

M. Vondrak, L. Sigal, and O. Jenkins, “Dynamical Simulation Priors for Human Motion Tracking,” in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2011), pp. in press, 2012.

Source Code

Sources for the physical model can be downloaded from here.


2008 Promotional/educational video with research results

Results for the PAMI 2011 paper

Video attachment and results for the CVPR 2008 paper [video (.divx, 36MB)]


This work was supported in part by Office of Naval Research (ONR) Young Investigator Award N000140710141 and ONR Presidential Early Career Awards for Scientists and Engineers (PECASE) Award N000140810910.