Vertigo is a common disease; however, the causes are very complex and wide ranging. This indicates that a great deal of specialized knowledge and skills are indispensable in the diagnosis of vertigo. Regular doctors diagnose vertigo in many cases; however, they have little knowledge and few skills concerning vertigo. In addition, the number of vertigo patients is increasing. For these reasons, the demand for supporting diagnosis of vertigo is growing. The purpose of this study is to develop an automated computer-aided diagnostic system for vertigo using video and image processing. One of the most important indicators in diagnosing vertigo is nystagmus involuntary abnormal eye movement. To provide diagnostic support for a regular doctor, the system must be easy to use and highly accurate. This paper focuses on analyzing nystagmus with a video-oculography (VOG) technique, which is a video-based method of measuring eye movements using an external infrared charge-coupled device (CCD) camera. Previous studies using VOG techniques have encountered two main problems: the noise from blinking and slow performance. This paper proposes a simple method for resolving these problems. The proposed method can be divided into four stages: 1) detect a blink; 2) estimate the pupil position; 3) detect the pupil position and radius; and 4) calculate the rotation angle of torsional nystagmus. A total of 12000 images from three patients were used to evaluate the validity of the proposed algorithm. The algorithm removed the noise from blinking for all patients. The average error of position estimation is 0.86 and 1.4 pixels in the horizontal and vertical directions, respectively. The proposed algorithm detects the pupil with 100% accuracy and occurrence of torsional nystagmus for each patient. In conclusion, the proposed algorithm meets the requirements for diagnostic support of vertigo for a regular doctor.