An old collet could be and probably is the culprit. If that's not it, the stepper motor for the Z axis could also be the problem. If you are not losing height when using different sizes of mills (different collets), the stepper motor would be the next thing I would check. If that is, a quick solution could be adding springs to help hold the weight of the spindle. If that don't work, replacing with a higher torque stepper motor would be suggested.