[as3] Flash Memory Use, THE SEQUEL!

[Rezmason]

Hi again! I'm back, this time with some graphs!

A short while ago I started a thread on Flash's memory usage, and how to track down memory leaks. Toward the end of the thread, I posted a link to ActiveGraph, a class I made that helps you keep track of what should be the active memory in use by the Flash Player. (Quick recap- active memory is the memory Flash is actually using; when memory is no longer active, it is called garbage, and doesn't go away until the memory limit imposed by Flash or the OS is reached, and the "garbage collector" is run.) You can read about all that and download ActiveGraph here (http://kirupa.com/forum/showthread.php?t=258129).

Since then, I've been using ActiveGraph to track down memory leaks in my latest project, BitmapEditor. (Eventually I'll open-source it, but first I've got to eradicate potential leaks.)

To just quickly restate how ActiveGraph decides what's active memory and what's not, the code in ActiveGraph looks at System.totalMemory (active memory + garbage) every 100 ms or so, and if the totalMemory ever goes down, ActiveGraph assumes that the garbage collector is responsible. That means, at that moment in time, there should be no garbage, so totalMemory should equal active memory. So ActiveGraph only outputs data when there's been a totalMemory decrease.

If you graph totalMemory plain and simple, you should get a "sawtoothed" graph- it steadily increases, but every now and then (when the GC runs), it sharply goes down. ActiveGraph "overlooks" these sawtooths, drawing a line beneath the teeth (which are made of accumulating garbage). So, if ActiveGraph really is outputting active memory, then the graph it makes should be stable- wavering up and down a little, without "sawtoothing".

Are we all on the same page? Cool.

Now, take a look at these two graphs. Their data came from looking at the same program- BitmapEditor- through ActiveGraph, but during one of them, I had commented out the code that draws some animated content. When I left it uncommented, the graph still sawtooths, but slower. When I comment it out, the graph stays relatively stable (or so it seems).

Graph 1. (http://www.rezmason.net/other_files/forum_resources/graphs/be1.png)
Graph 2. (http://www.rezmason.net/other_files/forum_resources/graphs/be2.png)

I decided to see if this happened outside of my project, and I created a simple Flash file called Barebones3, which contains ActiveGraph and does nothing; it just has a bunch of instances of a movieclip on the stage. You can see some of those graphs here:

Graph A. (http://www.rezmason.net/other_files/forum_resources/graphs/b3a.png)
Graph B. (http://www.rezmason.net/other_files/forum_resources/graphs/b3b.png)
Graph C. (http://www.rezmason.net/other_files/forum_resources/graphs/b3c.png)
Graph D. (http://www.rezmason.net/other_files/forum_resources/graphs/b3d.png)

I tried to switch things up, to see whether the graph changes at all if, say, bitmap caching is turned off or something. But there's no change- all the animated content I've seen so far causes a sawtoothed graph. And a sawtoothed graph means, there's still garbage.

Remember, these graphs are supposed to be garbage-free! I think what's happening is, data for animated content gets handled by a different garbage collector than other data. The sawtooth graphs we're seeing here are caused by a second, slower GC. Or, there could be one GC event that deals with reference counting, and a slower one for mark and sweep- two garbage collecting strategies that, optimally, happen at different times from one another.

But that's speculation, and it's not as important as the implications of this: your program may be slowed down by your animation, or by something else, and in order to test your program for memory leaks, it's wise to have in place a system that'll replace all your timeline animations- especially the ones that use filters and such- with placeholders, at least until you're done debugging.

Frankly, I think I'm coming close to over-thinking this, and I doubt it's that important to you all. As long as you keep your programs kind of slim, you'll probably never even notice these things. But seeing this sawtooth pattern in the graphs, after already compensating for another sawtooth pattern, is a big surprise.

Finally, does anybody know someone from Adobe who I can talk to about this, who's generally open to questions? If anyone here's had success talking to a bigwig like Grant Skinner, Mike Downey, Mike Chambers, or just somebody who could give these graphs some more meaning, I'd like to know who that was. :)

And now I'll open the floor to questions.

[Breen]

Very interesting find. Congrats on this. Anyone can confirm this is a problem?

[Gerrit]

This is indeed very interesting, and concerning. Seeing the memory-weight of some of todays modern Flash apps, this might give us some issues in the future. It would be great if you could get exclusion on this from higher up indeed.

Keep us updated.

[Rezmason]

Alright, here's an update.

I found a powerpoint presentation (right here (http://www.onflex.org/ACDS/AS3TuningInsideAVM2JIT.pdf) in PDF format) given by Gary Grossman a while back, and it distinctly says that there's a separate garbage collector for AVM1, AVM2 and the display list. The source code for this garbage collector, called MMgc, happens to be included in the code open sourced as Tamarin (http://www.mozilla.org/projects/tamarin/). (Which is largely irrelevant.)

So the Display List has its own GC. Whoop de doo, we've already seen that. But again, the point is that in order to get relevant information about our programs' memory use, we need to do away with GCs. And it seems the only way to do that is to give the display list nothing to look at.

So if you'd like to use ActiveGraph to monitor your program's memory usage, I'd suggest making your document class invisible. The Display List will have nothing to do, so it'll never accumulate garbage and you won't get a sawtooth graph.

But that also means you'll have nothing to look at while you're checking for memory leaks. It's a kind of lame solution. Another strategy is to replace the most processor-intensive animation in your program with placeholders. If you're a game developer, that could be tedious, and isn't nearly as thorough as doing a memory test with the "lights turned off".

Fortunately, ActiveGraph can be useful whether you can see it on the screen or not. If your program doesn't tax the Display List, you can "leave the lights on" and keep an eye on the ActiveGraph. If the Display List is getting clobbered, try "turning the lights off" and setting ActiveGraph to verbose mode.

One more mystery that Gary Grossman's powerpoint prezo solves is that sometimes, the totalMemory goes down slowly (that is, not instantly). That's because MMgc is incremental, and so sometimes it won't eliminate all the garbage at once. Sometimes, it takes two trips, so to speak, to carry it out to the curb. ;) Apparently these trips occur at 30ms intervals, but that's more academic than anything.

So. Recap. The Display List has its own GC. Give it nothing to do, and ActiveGraph will give you a better graph, one that'll help you find memory leaks. And now, I'll email someone from Adobe to confirm all of this.

[dthought]

senocular, apart from being totally awesome, works for Adobe if I'm not mistaken. Perhaps he may be able to offer some ideas regarding who to speak to?

[Rezmason]

Really? senocular? Are you serious?? :)

I just thought he had too much time on his hands! :trout:

Also, I'm going to see if Tom Reilly, the "garbage collection developer" on Tamarin, would be willing to entertain my questions.

[Rezmason]

One step forward, two steps back. Here's what Mr. Reilly said, in an email:


Cool graphs. Actually there is only one GC in Flash, I'm not sure how
you convinced yourself there is more than one. As far as developers are
concerned there is a lot more we could do to enable developers gain a
clearer picture of how memory is being used. Particularly we are
working on a memory profiler (AS3 only) that will give precise object
allocation and deallocation information. Stay tuned for announcements
on labs.

So I'm wrong about there being multiple GCs, and the expert says so. That's good news– it means as long as you have an effective strategy for interpreting the System.totalMemory, you should be able to track the garbage collector without worrying about other GCs laying around. Clearly my strategy is not so effective. Still, ActiveGraph should be of some help until more adequate profiling solutions arrive, and I'll keep working on it. :)

...Maybe there's one GC, but different sets of memory that are managed at different times, or something...

[Breen]

Hey Rezmason.

I've been following this topic as it progressed, and have been very interested in your findings. After your last post, I decided to re-read the starting post, and I think the problem you are facing is far simpler than what you are suggesting.

It's not that there are multiple GC's or different memory segments active. It's just your presumption that, when the System.totalmemory goes down all garbage is gone, is incorrect (in retrospect, I didn't realize this when first reading your post).

As is explained here (http://www.createage.com/blog/?p=53) or if you want to have a look on your own, google for (actionscript garbage collection incremental). You'll see that GC in Flash Player is incremental, which means that not all garbage is necceseraly collected in 1 go, it might take several goes, which explains your sawtoothed graphs, if my logic is sound.

If i'm wrong, please correct me :).

[Rezmason]

If you are wrong, Breen, I wouldn't know. :D

As stated here (http://www.onflex.org/ACDS/AS3TuningInsideAVM2JIT.pdf), only part of the GC is incremental; that's the part that uses mark and sweep, a longer process than reference counting. Perhaps the reference counting algorithm, which runs more frequently, leaves garbage behind that accumulates until the ceiling is reached and the mark/sweep algorithm removes what it can. That could definitely result in a saw-toothed graph.

But that would mean that the garbage that's being left behind is the kind that a deferred reference counter can't deal with, but a mark/sweep collector can. Furthermore, because the saw-teeth were more prominent in the graph during which intensive animation occurred, I assume that the garbage has something to do with what Flash is displaying on the screen. So I'm looking for an object that'd trick the DRC, but not a mark/sweeper, and possibly involves animation and very well may be created by Flash from the Timeline animation and not my Actionscript code.

EDIT: YES! Mr. Reilly has just gotten back to me again, saying essentially what I suggested above. Breen, you're correct; deferred reference counting can happen ridiculously quickly, but leaves objects with cyclic references behind. And, according to Reilly, DisplayObjects always have this sort of cycle: A parent in the display list has a reference to the child, and vice versa.

So here's a new strategy. Before deleting an object that's a subclass of DisplayObjectContainer, make sure it has no children. You can do this by creating a function– let's call it "abandon()"– that loops through all children of your object, until there are no children left:


function abandon():void {
while (numChildren)
removeChildAt[0];
}

That ought to disassemble the display hierarchy in the DisplayObjectContainer you're deleting, which should remove cyclical references between parents and children. You might want to create subclasses of Sprite and MovieClip, so that the symbols you have in the Library can abandon their children when they're removed (in response to Event.REMOVED).

I feel like it's time to make a new graph...

[Rezmason]

Never mind, I'm giving up. Or at least, I'm setting the dissolving DisplayObject classes aside. ActiveGraph may not take into effect the incremental behavior of the garbage collector (yet), and cyclically referenced DisplayObjects can still accumulate a little as garbage until being caught by the mark-and-sweep (which is its purpose, after all). But this discussion thread and the last one have already come to their logical conclusions, and if you read them through and through you'll have a fair enough understanding of the garbage collector to discern where a memory leak might be in your code.

If any of this ever confuses anybody, I'll be happy to talk with them about it. Future work and experimentation with ActiveGraph and the dissolve package I mentioned above will be posted where all this probably should've been in the first place, which is the Source/Experiments section.

[Breen]

RezMason, this isn't defeat neccesarily. As a developer your class still can prove very usefull. It might not account for DisplayObjects being cleaned up incrementally, but from looking at the raw graph, it's still possible to see any trends where display objects cannot be responsible. In any case, a good thing to have active in debug mode in ;).

[Rezmason]

Thanks, Breen. :) I didn't mean to say I lost a fight, per se, it's more like there hardly was one in the first place.

One feature I'll be adding to ActiveGraph, is a secondary sawtooth-remover. That should allow it to approximate active memory again, but will only allow AG to update each time the mark-and-sweep part of the GC kicks in. People will be able to choose whether to have the secondary working, in case they've got a lot of cyclically referenced garbage, like parented DisplayObjects. And I'll document the class, because now it's getting complicated.

[dthought]

Rezmason, your research into the GC is fascinating, even if I only understand about half of it :)

[Breen]

If you're gonna include some extra info, you might consider including a running average, over some of the sawtooths. Always nice to see trends.

[Rezmason]

I did consider that before, Breen, but a graph of averages won't show any trends useful enough to be drawn to the screen.

I guess knowing the overall average memory use of your program would be nice, though. I'll try adding that to the interface, somehow, without cluttering ActiveGraph.