IMHO, if it gets out of the gun, every single time, it's no big deal. However, if you want to change that ejection pattern - that's buffer weight, just like you thought, or a heavier spring. Sprinco Red spring. Either one will do it. The spring is cheaper, brother.
you also have adjustable gas - put another 1/2 turn in it and see what happens, go 3 turns instead of 2.5 - THAT is the easiest - it's free, since you already have that capability.