@Paloth
Just quadrangulate (or triangulate) your model. It is the n-gons that are causing the problem for you. Holes are fine, as long as they are surrounded by quads (or triangles) with angles less than 180 degrees.

@Khai
I have no idea what the garbage topology of the bottom wireframe is supposed to demonstrate. The top one has correct topology, and will work fine. The bottom just would need to connect the vertices to work...
I don't know what else to say about the subject; but don't want to argue with anyone. That is not why I come here.

Friends don't let friends use booleans.