Perhaps because they lost the ancient Winter Coffin, the Frost Giants still retained some of the ancient Frost Mages' traditions, but not a single Frost Mage was left alive. As a result, their people placed a particularly strong emphasis on martial prowess. They looked down even more on Earth mages like Zhou Yang, who seemed weak and powerless, and would pick fights with him every so often.
You can imagine how miserable Zhou Yang's situation was, even with some magical abilities, fighting Frost Giants in Jotunheim. Only after his magic continued to improve, eventually reaching the level of a legendary mage, did he finally gain enough ability to protect himself. The Frost Giants stopped daring to bother him, and his life gradually settled down.
Even so, Odin had suddenly brought him back to Earth. He would never believe that there was nothing behind it. Heimdall might have dumped him in Jotunheim back then to punish him, but now that he had sent him back to Earth without a word, there had to be some other scheme at play.
Could the Betty Ross standing before him really not be Dr. Bruce Banner's girlfriend?