This paper is devoted to designing a binaural sound reproduction system using multiple loudspeakers, which is a virtual realization of three-dimensional audio. The binaural sound reproduction requires the inverse model of a reproduced sound field. It is known that the transfer function of a sound field includes unstable zeros that are originated from the inherent propagation delay. The complete inversion of the sound field is inevitably unstable and cannot be implemented to actual systems. To overcome this problem, we propose a design method of a stable feedforward inversion based on the H∞ model matching problem. In the method, low gain specification is imposed on the inversion, which is important for improving robustness against the modeling error of the reproduced sound field. The design method is applied to actual experimental data of a reproduced sound field to show its effectiveness.