No matter how much RAM you stick in, Windows 32 bit will not see more than 3 GB.  To get beyond thant you have to go to 64 bit on OS and application.

Critical evaluation of textures is in order.  You might need to reduce size of some textures.  It is wasteful to have a 2048 x 2048 eye texture on character that is in distance farther than 10 feet from camera.

You have to reduce texture dimensions to save memory, not just apply higher compression to file.  Some people thinks by reducing file size, you reduce memory.  Not so because application has to uncompress file to use it.

Cutting in half dimension of map means you only use one-fourth the memory, and this can add uo rapidly.