A very simple method to eliminate any software questions is to program a circle manually, then run it.
In Mach you would:
Position the spindle in the center of a piece of material.
Set both X and Y to zero at that position
Zero Z axis to the surface of the material
Enter the following sequence:
My comments are in parenthesis and should not be added.
M3 (spindle on)
G0X4Y0 (rapid to X4 Y0)
G1Z.1F25 (feed to .1 inch above work at 25 IPM)
G1Z-.1 (cut .1 inch into material at 25 IPM)
G3X4Y0I-2J0 (cut full circle with the center -2 inches in X from current location)
G0Z1 (rapid to 1 inch above material)
M5 (turn spindle off)
Unless you changed a bunch of stuff in Mach, your machine will power up in absolute mode which the above code uses. A lot of the stuff in the beginning of a program is to make sure things are set properly in case they were changed. That is why this code seems so short, no need for it. Of course, a good thing to do to make sure you do not have problems is set the Z zero a few inches up in the air and set zero to do a test run, or you can use the dry run function. I prefer the sky cutting method myself.
That G3 line instructs the machine to cut a full circle in one go. Knowing a little bit of G code goes a long way in solving lots of "issues". Instead of sitting around having no clue whether it is mechanical or software, you can eliminate the software portion of the issue very quickly.
If your machine still cuts a crummy circle, you have machine issues. BTW, you list your machine as homebuilt. Did you design it yourself or follow some of the plans available on line?
But, all that said, it is your machine. You just need to verify that for yourself. Then you can get on with fixing the real issue. Like Martin said above, I have been cutting circles on my homebuilt machine for over 10 years now. They are all round and do not look like yours, even when I use tabs.
Edit: I just saw your post about having cut good circles with Linux CNC. Sooo, thousands of users are using Vectric to program circles everyday, so you can be pretty sure it is not the software. That leaves one thing, the way your Mach 4 is set up. I would jump on a control forum to figure it out.
"If you see a good fight, get in it." Dr. Vernon Jones