In recent years, a realization of haptic system is strongly desired in industry applications. Also in it, the bilateral control by a master slave system is used widely. We succeeded in realization of "law of action and reaction" among plural system by multilateral control that expanded the bilateral control. Because the plural systems are handled, it becomes important to think of the connection condition among the systems. In this paper, multilateral control considering connection condition based on consensus problem in graph theory. In thinking of the digraph in graph theory as the differential mode of multilateral system, we analyze the haptic transmission by matrix that expresses the graph. The numerical simulation and experimental results show viability of the proposed method.