The harem has been guarding the Santu River for a hundred years, crossing the present life of the lonely soul and understanding the past life of the soul. There is nothing you can do about yourself.
Some people joked that he deserved to be the matchmaker from hell. The implication, to put it bluntly, is that he is nosy.
It doesn't matter if you break into the City of the Past alone, or fight against a martial judge. He alone takes care of those noisy matters.