I'm running XP with 2 gigs and I can use the poser shader tree alright.
Bear in mind that there is no need to go over 2.5 gigs really, since XP 32 bits can't address more than 2 gigs per application.
Maybe your ram is faulty here, maybe you should only have 2 sticks of 1 gig, unless your ram is dual memory, in this case it works with 4x512 better than only 2x 1gig.
Or maybe your video drive is a tad too old? Mine has 512 megs (new), and it does make a huge difference in handling textures compared to previous machine.