Jagged trench around model

[Xxray]

Am in the process of making this model, I used a wavy background and what I did was create a vector boundary around the model then subtracted the waves within the model. That worked but as you can see, it left a jagged trench around the whole model.

Pretty basic stuff I'm sure, but what is the best way to go about eliminating or at least reducing this flaw ?

[dealguy11]

Rather than cut out the model from the background, you could leave the background alone and try setting the combine mode of the model to "Merge". You might have to adjust the depth of the model a bit to make it stand out properly from the background.