Rabbi Amy R. Scheinerman: What’s What? “Non-Jews who are guests in a synagogue can cover their heads; it is a sign of respect and not at all inappropriate for people who are not Jewish.”
“Some explain that this means that prayers were instituted (..) after the destruction of the Temple to replace the offerings.” Archivierte Kopie (Memento vom 6. Januar 2017 im Internet Archive)