Without knowing how many layers, or anything about your system, it's difficult to say. It does sounds like maybe you've reached as much as can be handled. Is it possible to split the file into two or more, sharing the layers (and so the load) between files?