I think you just got a bad set, I’d return them.

I have about 4 sets of these on my various benches/carts: LINK

If I’m in certain parts of my shop where the floor is uneven and it’s a small cart, I can get a little wobble. But that’s only because a wheel is off the ground. I’ve never had an actual caster rock side to side like that.