He knew that he was just living in a corner among thousands of realms, watching endless people and things being passed down from generation to generation in the six realms. He can leave at will, but there are still some people who still have something to do and are reluctant to let go. So it's better to wait until those things have abandoned him before he leaves.