What do you mean by expanded PVC ? To me over here in the UK means something like foamex (Brand name) which is not a solid PVC material. It is a mixture of PVC & air which produces a light weight material. Because of this once you machine away the top solid layer & expose the core you can get very mixed results. Brands seem to vary enourmously & the cheaper it is generally means it has more air than PVC in the mix lol Signmakers use it quite a lot & if you see boards that look like waves on the sea then it is normally due to using a cheap board & leaving no room for expansion on fixings. More air it contains the more it is affected by changes in temperature.
The amount of air (& probably size of air pockets) it contains can make a big difference to surface finish. If left like that it can also get dirty very quickly & never look good again.
Trying different brands might be an option for you, can't say I have ever done anything other than profile cut it & as you noted yourself a good single flute bit will do that. Only other thing I can suggest is go back to the drawing board with speed & feed settings & epxperiment. Much more difficult to get good settings with something like this than it is with wood, plastics are far less forgiving & it may just be you haven't got the settings right yet.