When an otaku traveled through time and became Sesshomaru, his life experienced earth-shaking changes.
Naraku didn't turn into a monster, but he was just a hair, so why would he stick to him.
Also, InuYasha always goes to him because of his weakness. His CP is Kagome, Kagome, not Sesshomaru!