Analysis of information sources in references of the Wikipedia article "Ivory tower" in English language version.
The origin is the Bible, specifically Chapter 7, Verse 4 of the Song of Solomon, in which Solomon is extolling the beauty of his beloved... Not quite the thing today [...] but it struck a chord with Charles-Augustin Saint-Beuve.