for the magnets, why not set them in the actual hole itself so the ball contacts it directly. Just cut a slot so the magnet pushes in from the side.
It'll give the same amount of movement but with a much stronger attachment.