The allowance offset should work, on a male inlay part the a positive allowance should make the letter bigger & a negative make it smaller. I can never remember which way round allowances go for each type of toolpath so I always set a really large offset & then look at the 2D toolpath view to see how it has cut
I have just done that to make sure I posted correct way round above
I always try to cut the male parts first, this way if they are slightly to large then you can increase the allowance a little on the female pocket. Are you using the same vectors & the same tool? Not just the same size tool but actually the same tool? If you are not then that may be part of your problem, Cutting bits are often not exactly the size they claim to be.
It will obviously depend on what size you have overcut the female pocket if the letters are to small, that's another reason for cutting the male letters first, if you cut without any allowance what so ever then the letters are never going to fit in the pockets so normally only a small allowance is required.
0.04 inch is quite large but obviously depends on what sort of fit you are after & what size the letters.
I have just done some Name signs using Oak with 3mm acrylic letters, there were 10 of these in total, the letters are approx 3 inch tall, this was using a 3mm bit with a 0.15mm offset allowance, letters were all a very snug fit