'Point At' works along a line from the 'origin' of the pointing object through its 'endPoint' and finishing (or starting, depending how you look at it) at the origin of the Point At target. Thus the orientation of the pointing can be set via the endPoint of the item.

None of the above seems much help with your problem. I don't know a solution to your problem off the top of my head, but I have a couple of ideas that might work. But rather than just sprouting off with ideas that might come to nought, I want to test them out first. I will get back to you when, and if, I have something interesting to report.