Not weird at all - the more magnification you have, the more movement you're going to see. The greater the magnification, the easier it is to see how shaky you are. Dial it down, things move less, and you get more comfortable on target. When you see the target moving, at higher magnifications, most people have the tendency to "ambush" the target - jerk the shot right when the target is centered in the crosshairs, and not moving around.
My $0.02.