In this paper, we propose a new tracking system based on a stochastic filtering framework for reliably estimating the 3D pose of a user's head in real-time. Our system estimates the pose of a user.s head in each image frame whose 3D model is automatically obtained at an initialization step. In particular, our estimation method is designed to control the diffusion factor of a motion model adaptively. This technique contributes significantly to improving the following performance simultaneously: the robust tracking against abrupt head motion and the accurate pose estimation when the user is staring at a point in a scene. The performance of our proposed method has been successfully demonstrated via experiments.