ah.you used the magic wand... did you smoothen out the edges after it was selected? click on refine edges to do that. also..you might want to try increasing the tolerance level. 

work at 200-300% of the size to see exactly whta was selected and smooth the edges become. 

for something like this.. i'd use the marquee tool and work my way around it.. or a mask.. 

cheers and hope that helps. 

• rosie