There are two places the drill could be off.
1) the chuck could be worn or set badly
2) the shaft could be bent or the bearings worn
Chucks can be replaced fairly easily and cost as much as you want to pay. Decent one is $30 to $50 and a great one could be $250.
So, the main thing to check is the shaft. I have to assume you don't have a Brown&Sharp or Starrett dial gauge. So, to check for bearing play, crank the quill up and down and look for movement side to side, then with the quill all the way down try to move it with your hand. It should feel rock solid, no play. Let the quill back up and turn on the drill on with the slowest setting or pulley combination. Watch the chuck turn and see if you can see any wobble or movement. Set a block of wood on the table and slowly slide it toward the chuck till it is just barely touching. It should make a smooth swishing noise. If it makes a clicking or modulating noise then the chuck or shaft is not concentric.
To determine which one you would have to remove the chuck and repeat with just the shaft. If the shaft runs true then it is the chuck.
If the rest of the drill looks good and works as expected then the chuck can be replaced.
If you do have a dial gauge then you can set it up to measure the amount of non-concentricity in the shaft or chuck.