The driving of unmanned vehicle has been realized step by step by the automatic vehicle in recent years. On the other hand, it is difficult to drive the unmanned vehicle full automatically in the environment in which the information is insufficient. Therefore, people have to drive the vehicle proactively in such an environment, and the teleoperated vehicle is one solution. The defect of teleoperated vehicle is its poor operability, so assist control should be added to drive the teleoperated vehicle safely. However, there is a problem that the effectiveness of assisted control depends on the sensor accuracy. In this paper, a driver assistance system that does not depend on the sensor accuracy heavily is proposed by fusing the human input and the automatic assisted control based on potential method and virtual impedance control. Steer-by-wire is introduced into the steering wheel of the vehicle so as to realize vehicle teleoperation and assisted control. The validity of proposed method is verified by simulations.