The best way to clear out individual brushes is to use your Preset Manager. You can then create your own Brush Set.

If you want to clear out all the brushes you click Reset Brushes. This will default to the first Brush Set in the list (Assorted Brushes).

If you don't want a Brush Set to appear in the drop down list then it has to be taken out of Preset/Brushes directory. You then have to restart Photoshop.