I disagree with your statement that tall is relative. If that were the case, the question “How tall are you?” would not invite an answer like, say, “Six feet six,” but one like “Fairly.” I think both words are relative and quantitative but that they simply measure (or compare) different characteristics.
If you look at the opposites of the words, I think you can see where the discrepancy is (at least as I see it). You might think that the opposite of “tall” is “short”, but you’d be wrong. Another way to say “short” is “not very tall” which means that even short things are tall, maybe less tall than average, but still tall. I would say there is no opposite to tall. “High” on the other hand, does have an opposite but it’s not “low”, like you might expect, for the same reason “short” is not the opposite of “tall”; the opposite of “high” is “deep”.