The on’yomi of kanji comes from Chinese, which (ignoring tones) only has a fairly small number of different sounds (though I’m no expert, so I could be completely wrong). As well as こう and じょう, other readings I often encounter are しょう, ちょう, しん, けん and (to a slightly lesser extent) せい.
So how do you tell them apart? Easiest option: don’t learn them in a vacuum. Learn them as part of vocab. Since you know that, for example, 小学生 (しょうがくせい) means “elementary school student”, you’re not likely to confuse 小 with, say, 商 or 将.