Love to have that billboard prop. I tried to do the same thing with point-at years ago with the same results and gave up.
Would be useful for flames, smoke, foliage, crowds, etc... (Point-at/maintain X/Y/Z angle relative to camera plane should have been built in years ago...)
Also, you might be surprised how realistic billboards can be when they are normal mapped.
https://www.blendernation.com/2017/03/03/fake-large-scale-forest-blender/