Down-and-Out Prodigy

I prefer the type of protagonist who doesn't lick dogs.

Could it be that he actually traveled through time a long time ago and then lost his memory, thinking that he just traveled through time...