Olomouc is small you only need a day or two to be able to see all the sights. It is a college town so when I was there in September there were many student and foreign students too.
It would help to tie your visit there with a festival. I happen to be there for one and it was fun. I would even recommend doing it as a day trip from Prague if you only want to see the sights.