In the process of trying to copy morphs between actors, I'm having to do some complex organizing of variables in lists.  I then have to reference the lists (of course) and sometimes iterate over them.  I've come across a situation where a series of loops is consuming increasing amounts of RAM and not releasing it until Poser is closed.  I've done some research, and this sounds like a pre-2.5 Python problem with memory allocation using malloc (whatever that means) when dealing with objects which are containers.  Umm.  This is Greek to me, more or less.

I assume that this is something relating to the deep copy/shallow copy problem, however.  Some of my lists are lists of lists (homemade arrays, perhaps).  Some of the lists are constructed using others of the lists.  I'm wondering if I'm ending up with lists full of references to the other lists, rather than lists full of the data from the other lists.

Umm.  Clear or confusing?  I think I'm making deep copies, if I understand what that means, when I need shallow copies.  Is that right?

Under what circumstances does this sort of thing commonly occur, and how can one avoid it?  I've been restructuring everything until I've made a real mess of the code, but the problem is not going away.  If anyone has any ideas, please respond.  I'm getting gray hairs, fighting with this.  #$%&^!!  :)

I suspect your problem is that your using global variables to store your lists, and your not assigning None to them before the script finishes, hence there is still a reference to the whole of the contents and so nothing can get garbage collected.

Another possibility is that the memory will be garbage collected (provided there are no in-scope references to it) once you actually use enough to trigger the GC.

Well, all the lists are assigned to local variables at this point.  In my first version, they were stored in instances of a class object, which I assume were global.  I removed the class, but that hasn't fixed the problem.  I've also tried using copy.copy() on just about everything now, with no results.  Perplexing.

I am not assigning None to the lists at any point, however.  Is that common practice?  I've never seen it done with lists.  I think I've seen it with classes.  I'll give that a shot.  Thank you.

When I've run the garbage collector, it's come up empty - claimed there was no garbage to collect - and has failed to clear any memory.  I may be implementing it badly, however.  This is another area where I haven't really seen any examples of good implementation.

A very frustrating situation.  I have a gigabyte of RAM and I'm trying to compare verts and polys between two objects, and there isn't enough RAM to compare objects of over 1000 verts apiece.  When I was comparing verts to verts, in a previous script, there was no problem like this.  In my experience, more intensive calcualtions lead to slower processing, not more RAM consumption.  I can't figure out quite waht I'm doing that's giving rise to this result.  Hmm.

I'll try what you suggest.  Thank you.

This is what I'm babbling about, with the malloc and version 2.5....

http://evanjones.ca/python-memory.html

You might have to use arrays from the Numeric module to hold your data, see -> http://numpy.scipy.org/numpydoc/numdoc.htm
The version included with Poser is 21.0, so keep that in mind when looking at the documentation.

PoserPython is 2.1?!?  I thought it was 2.23.  

No wonder I'm confused....

Thank you.  I'll look into numeric.  And gc.  And None assignments.  

Upon reading a bit more about garbage collection, I think the gc module may have failed for me because I have circular references in place.  Drat, drat.  Cage needs to remind himself: don't take that shortcut approach that seems so feasible.  Better to repeat a few steps than to end up with a mess like this one.  Gah.

If all the references to objects are not deleted then you won't see it in the garbage because there are still outstanding references.

You might test this by laboriously assigning all references to None and then deleting them, this might get it under control.
This is also very important when using dictionaries in python, where it is the prime source of memory leaks.

I also notice that poser's python is rather greedy about not freeing memory. However, I've never had it get really out of hand.
It hurts me a little because I'm a very lazy programmer and don't always clean up my garbage.

*You might test this by laboriously assigning all references to None and then deleting them, this might get it under control.

*Hmm.  Could one query locals() or globals(), loop through them, and assign every single one as None, when the script is done?  Or is that not possible or not advisable for some reason?

Personally, I had never paid much attention to garbage.  Like a fish with water, perhaps.  So much garbage around me that it's invisible.  Hmm. 

Any idea if and/or how there are circular references here?  First, I create a list from Geometry().Sets(), then I use that list to create a list of edges.  The list created by polyverts() is the pverts used by polyedges().

#--------------------------------------------------
def polyverts(geom,polys):  #return the sets list broken down by polygon       
    pv = []
    gset = geom.Sets()
    for p in polys:
        l = p.NumVertices()
        s = p.Start()
        e = s + l
        vl = [v for v in gset[s:e]]           
        pv.append(vl)       
    return pv

def polyedges(geom,pverts): #return the start, end vertices for each edge in the polygon       
    edges = []
    for p in pverts:           
        poly = []
        for v in range(len(p)):
            if v != len(p)-1:
                e = p[v], p[v+1], vecsub(coord_list(geom,p[v+1]),coord_list(geom,p[v]))
            else:
                e = p[v], p[0], vecsub(coord_list(geom,p[0]),coord_list(geom,p[v]))
            poly.append(e)
        edges.append(poly)       
    return edges
#--------------------------------------------------------------------

So does the generator object used in polyverts() create shallow copy references to the values in Geometry().Sets(), or does it create deep copy references back to the Sets() list itself?  And when I append p[v] (etc) in polyedges(), do I create deep or shallow references?  Just exactly where am I making circles?  Any ideas?

Sorry to be a pain.  This is a rather new and mysterious topic for me.  Python keeps surprising me....

Quote - PoserPython is 2.1?!?  I thought it was 2.23.  

No wonder I'm confused....

I was referring to the version of the Numeric module not Python.
The reason I mentioned it, is because this is a very old version (4+ years, source code only available from the numpy CVS at sourceforge).

Thank you.  I'll look into numeric.  And gc.  And None assignments. 
Upon reading a bit more about garbage collection, I think the gc module may have failed for me because I have circular references in place.Python should cope with reference cycles, but extension modules have to correctly use the relevant API calls, so possibly Numeric and the poser module may not be doing things that they should be to avoid trouble.

*possibly Numeric and the poser module may not be doing things that they should be to avoid trouble.

*I see.  I'm not using Numeric at this point.  So I'm shaking my fist at Poser, yet again.  

Malloc is a function from the underlying C-library used to collect/reserve memory.
Newer Pythons are not kown as having problems with this.

Python is using references to variables where even possible.

http://www.python.org/doc/2.5/lib/module-copy.html describes how the copy module works.

To see whats up with your variables, use:

gc.get_referrers( *objs)

(see  http://www.python.org/doc/2.5/lib/module-gc.html)

The Numeric modul gives you "real arrays", fast math functions and easier handling while dealing with 3D-data.
Here is a starting point: http://wiki.python.org/moin/NumPy?action=show&redirect=Numeric

Right on.  Thanks, adp001.

Will using the numeric arrays rather than lists of lists help avoid these circular references/cycles which I apparently have?  I'm really trying to puzzle out when this probelm can occur.  The examples I've seen are generally much simpler than what I have in the code sample, above.

The frequent example is:

a=[]
b=[]
a.append(b)
b.append(a)

But in which portion of my code above do I achieve similar results?  Do I have the problem in polyverts(), or polyedges(), or both?  It seems to me that if I want to really avoid and fix this problem, I need to understand how it's happening, before I dodge the current trouble by switching to gc and numeric and the copy module.

I do wonder if using a copy.copy(pverts) in polyedges() would help....

Thanks, everyone. 

Hmm.  I've had copy.copy and copy.deepcopy backwards, haven't I?  How like me.