I'm so stuck that I think I'm going backwards.

Ok, here goes.

I have a 2k (27") monitor running on an ATI FirePro V4900.

I build my own computers, so it's fast enough.

I setup both displays to 800x600@60Hz.

Main display is running on display port, scanner is through a
DVI to VGA cable.

I managed to get one scan! One! The quality was horrible,
then it hit me that I couldn't see the quality settings because
of the 800x600 resolution. It just doesn't fit on the screen. This
is an issue but not why I'm here.

I decided to change the main resolution higher so I could see
and keep the scanner at 800x600 (still in extended mode).
This did not work... I then roll all my changes back so both
are at 800x600. Now I cannot get it to calibrate or scan at all.

It just mirrors the main display. Ok, it doesn't mirror, it extends
but I have so many desktop icons it seems that way. It is extended.
I place a piece of paper in front of it and see that at least it is
projecting properly, and it is. I just cannot get it to go into what I would
call "pattern mode". During calibration and scanning, the thing
rotates and software seems to be calibrating or scanning but it's
not. Just throwing the extended display image.

This is is my first venture into scanning and it's giving me a headache.

How did I manage to get it to work once and now I can't?

I'd rather get community support before contacting tech support.

I'm using Windows 10.

Can someone slap me and help me figure this out :(

If I read correctly, you problem most likely is that you set both displays at 800X600,
only the projector/scanner should be set at 800X600, your own/main display can be set at any native resolution you want.

and if you cannot do a first calibration scan, the scanner will not work correctly.