The comment above about higher vertex count is spot-on in my experience. RDS's boolean operations are good, but they still do have problems, and I've found that if I increase the amount of vertices around the area I'll be performing the boolean, it'll be much cleaner in the final result. Hope this is helpful....