Nope, you don't need to distinguish. He might have gone to the hospital as a patient or as a visitor. If you want to specify that he was a patient, that's when you'd say he was "in the hospital."

Actually, now that I say that, I don't know that I usually hear "to the hospital" as the person being a patient? I think I usually hear "to" as going there in a geographic sense, to visit or to work or whatever, and being a patient is nearly always described as "in."

English prepositions, man. I don't envy anyone who has to learn them.