So you've tried adjusting the tightness of the bolts holding the pulley clamps to the front plate? Adjusting the angle of the bolt with respect to the belt should affect where the belt rides. My pulley clamps were glued in place prior to assembly, so all I could do was use a round needle file to ream out the bearing bolt hole in the side opposite from where the belt tended to creep to. It's still not perfect, but it helped.

The way the belt can rub against wood is kind of rinky dink. At least on the X idler you can adjust the angle of the bearing bolt pretty easy and adjust where the belt rides. I still hope to eventually retrofit something like what you're talking about to both the X and Y idler.