this is where you need to start thinking about designing for your specific machine.
Try a 1.5mm hole, the odds are it'll come out nearer 1mm than 1.5
I have a 'guide' sheet from 3d hubs for tolearances for different types of 3d prinyter and for FDM they claim a 2mm hole is the smallest you can do.
They are wrong, but it can take a bit of fiddling to get smaller holes. :-)

Speed sounds fine
But the oval columns do indicate that something is not quite right mechanically. a round 4mm column shouldn't be an issue for any printer.

Like wise the variation in length would suggest the same thing. something is slipping or loose somewhere.