The Whole Village Wore Them, Leaving Me Alone with My Useless Body

In a modern and peaceful village, all the people except the protagonist travel to the fantasy world overnight and become the bloodline of geniuses. The earth is actually a prison in the upper world where ancient gods are imprisoned. The protagonist stays alone in the original world but awakens the "reverse passage" that can travel in both directions. He relies on reselling the resources of the two worlds and abducting the goddess from the other world to change his destiny.