Absolutely.
You can even see considerable difference in render time just by modifying the resolution of a garphical mat.
There are times when I'd be willing to bet you can get away with uber high poly count if you use simple mats one no additional lights.
Reflective mats like mirrors, metals, glass...all increase render time.
Add extra lights to those and it gets worse.