While the alt key is depressed the focus choice is made each time the (left) mouse key is pressed. Each drag motion (to achieve camera repositioning) involves one ‘mouse down’ and that motion must thus begin with the cursor over the thing about which you wish to rotate.