Do you have to really ramp up the retraction distance in order to avoid oozing when printing with the other hotend? Pull the filament completely out of the melt zone?