This paper presents a method for estimating positions of solid balls from images which are captured using a handy camera moving around the pool table. Since the camera moves around by hand in this method, the motion of the camera in 3D space should be estimated. For the camera motion estimation, a homography is calculated by extracting the green felt region of the table-top area that is approximated to a polygon. Then, the balls are extracted from the table-top region for obtaining the positions of the balls. The 3D position of each ball is estimated using a projection matrix determined by the homography. The ball areas are classified by distribution of RGB data in each area. We apply our method to image sequences taken with a handy camera for evaluating the accuracy of the ball position estimation. By this experiment, we confirm that the accuracy of the estimated position is up to 18mm error, which is sufficiently small for displaying the strategy information in the pool supporting system.