Tiff or Jpg doesn't make any difference whatsoever for render memory consumption. And neither does texture size as of P7 or newer. It must be something else in your scene or render settings that's causing the headaches. If you share more info about that, maybe we can help you get that render that you want?