3-4 days is plenty of time to see this city. The vast majority of travelers come here via the Camino de Santiago, so the cathedral is clearly the high point of the trip.
I recommend getting to the church VERY early if you want to see the crypt of St. James and hug the Saint. My friends went during the daytime, and had to wait 40 minutes for a rushed 1-2 minute experience. I was there when the doors opened, and had it completely to myself for about 15 minutes before just 1 other person showed up!
It is avery small city but unique and special. The cathedtal is a very special place, reminds me of two things, the vatican and harry potter.