is it a fixed touch plate or one you place on top of the material? If it is the latter then are you placing it in exactly the same place & orientation after your bit change? Wood often isn't as flat as we think it is so even with a touch plate it is possible to get different z heights when changing bits
As Randall says changes in humidity can affect things as can tension within the wood that is released when a large amount is removed from one side. Have you experimented with different types of wood ?