It won't do much damage if you use more threads than you have cores, just a little. Every thread adds a little bit of system load, this will slow your render down a tiny bit. You won't be slower if you use just two tasks, if your CPU does not have Hyperthreading. If it has Hyperthreading, 4 will be the best choice, all IMHO. You can see it if you have an CPU monitor ( in Windows, you get it by pressing [Ctrl][Alt][Del] and choosing the Task Manager) - if the CPU load is 100%, all is fine, if it's lower, add one thread.

A ship in port is safe; but that is not what ships are built for.
Sail out to sea and do new things.
-"Amazing Grace" Hopper

Avatar image of me done by Chidori.