We consider a framework that learns human habitual behaviors from the data obtained by various kinds of sensors installed in an environment, so that the environment can interact with us based on those patterns. In this paper, we achieved extracting human behaviors by infrared sensor network as an initial step for the framework. Infrared sensor network is able to track us without putting an extra burden on us. Moreover it is able to collect long-term data. However, tracking with it has two problems, that is, link miss and incorrect link. In order to mitigate these problems, we propose the tracking method utilizing estimated "time distances" between sensors from movements' records. We have installed infrared sensor network in our laboratory, and validated the proposed tracking method by test courses. Afterwards, we confirmed that human behaviors can be extracted from longterm data.