I am not using any USB 2.0 hubs to down-convert the USB host.

I am, but I'm using a Thunderbolt adaptor & with native USB 3.0 on the iMac, and I didn't think it was worth adding more incompatibilities.
I'm using a D Link USB 7 port USB 2 hub no problem on a USB 3 port, but I might try the native USB 3.0 after your post to see if there is any effect.

Mike