Human memory does not fade, but "rots". Each memory is like a puzzle piece that will distort, proliferate, and even stick to other memories over time, creating false "residues." The protagonist's ability is to smell the "smell" of your memory being rotten.
