Injuries caused by falls are one of the major threats for the elderly. Thus, the demand for a fall detection system without requiring a user to wear or equip any devices is rapidly increasing. In this paper, a fall detection system using Ultra-High Frequency (UHF) passive Radio-Frequency IDentification (RFID) is presented. With the RFID reader on the ceiling and multiple RFID tags on the floor, anomaly scores are computed based on the neighborhood preservation principle. This focuses only on the correlation between RSSI fluctuations between a pair of tags. Therefore, unlike the conventional methods, our method manages to detect falls without requiring a large number of reference data. To detect falls in several areas of the room, our system only requires a simple reference data with one subject walking around the room. Experiments are conducted in an indoor environment with two different setups to show the effectiveness of our method. Our method improves the conventional one, and achieves a high Area Under Curve (AUC) of 0.98.