Ok, figured it out. Next to the 2nd Red pillar, there is a large rock with a rotating switch similar to the one on top of the 3rd pillar. It's well hidden in the sandstorm.
If you can't teleport, some weird flag is set. Best advise is to use the moonjump code and try to get all the lightroots you can. One of them will have a stone alter/building thing, and you can ascend up it.
To remove the teleport flag bug; try entering a shrine, then exiting.