Peltier device is a thermoelectric element which can generate a heat transfer. The device is considered a suitable tool for transmission of thermal sensation recently, because it has a relatively fast response among thermal devices. To transmit the thermal sensation, temperature control of the device is thought to be important. However, several elements such as joule heat, parameter identification error, or heat flow from an outside object disturb the control. Moreover, it is also important for the transmission of the sensation to design a system's tracking performance, while ensuring its robustness. To tackle these problems, this research proposes a robust temperature control system based on a heat disturbance observer. The proposed observer uses the structure of a disturbance observer, which is mainly used in the field of motion control. By using the observer, disturbance of the thermal system is compensated. In addition, it is possible to design the tracking performance and the characteristic of disturbance suppression independently. The validity of the proposed method is confirmed by experimental results.