This paper presents a perceived real-time system for memory augmentation. We propose a realistic approach to realize a memory assistance system, focusing on retrieving the person in front of you. The proposed system is capable of fully automatic indexing and is scalable in the database size. We utilize face recognition to show the user previous encounters with the person they are currently looking at. The system works fast (under 200 ms, perceived real time) with a decent database size (45 videos of 24 people). We also provide evidence in terms of an online questionnaire that our proposed memory augmentation system is useful and would be worn by most of the participants if it can be implemented in an unobtrusive way.