Blog

Skip Intro

Had the honor of meeting up with an old friend yesterday. Chris Skogen, one of the talented gents at Gizmolabs, was my manager & tech lead when I worked at BellSouth. He’s an extremely talented programmer, doing a lot of C, C++, assembly, and Java. Below is some of his team’s coding + hardware work in action. This is not actually the record music playing, but rather, the record’s time code being used to scrub an MP3. Plethora of other features but… isn’t this one alone enough to make you go, “OMFG!!!”

He’s been doing it for a long time, and since I respect anyone who has a lot of experience in this industry, I pretty much spent the day absorbing every word. We were supposed to meet for lunch since her majesty and I skipped out on going to a Christmas party of a friend of his that we RSVP’d too. Figured, I could buy him lunch and he’d forgive me. Anyway, instead of staying an hour at noon, stayed till 7pm mainly talking shop, finishing with Gears of War coop. He’s the guy that got me typing my brackets on a new line after he and another C coder at BellSouth started making fun of my if statements (Flex Builder is biased against this way of coding, btw…bleh !). He was also the guy who first introduced me to the Command pattern, and wondered if the Flash community had any framework like that. Four months later, I was all into ARP, which has the Command pattern as its central crux.

I spent some of the time asking about invalidation methods. Invalidation being defined as “painting the screen with pixels in an efficient manner”, also known as redraw. Since he develops for multiple platforms, he knows the libraries and strategies involved for screen redrawing on Linux, OSX, Windows, and various devices. I remember one of Bruce Epstein’s books, Director in a Nutshell describing “The Great White Flash”. It’s been so long I can’t really remember the cause, but the redraw of the window caused the application to start with this white flash, basically a redraw of the screen that sabotaged the initial experience with various ways around it. When Chris would describe Windows redraw events compared to other platforms, it sounded all too familiar. The Director mantra back then was, “If you build it on PC, you know for a FACT it’ll work just fine on a Mac. If you develop on a Mac, it will NOT work on a PC.” A lot of the intricacies of when something redraws an area, how you have to manage it, etc. seemed really gross, and painful. However, like a lot of C guys I’ve talked to, he seemed to take it stride, view it as a conquered fiefdom, and begin to relay the double-buffering strategy employed in some cross platform library. Must… get… that… attitude!!!!

In particular, I recently had the good fortune of getting Greg Burch’s, bloke employed at Adobe on the Mobile Team, feedback on my Flash Lite 2 component set. The guy clearly knows his stuff, has massive knowledge in ActionScript optimization, and is very focused in achieving that goal. He also understands mobile GUI development as well as the experience users expect on devices. After talking to him on the phone, I can clearly see why he is employed with Adobe on the Mobile Team. In our discussions, both offline and on, he basically set out in the opposite direction as me. While I was originally interested in creating a nice API as a tertiary benefit to the original 3 goals, he instead approached it from a “we must make it lightweight”. Even 10 seconds of his recounting of past projects with no supportive details clearly indicates that he has earned in blood a lot of great techniques and understandings. Basically, he is making the code look as much like ActionScript 1 as possible (even C for that matter), not as an intention, just a result that optimization and ActionScript 2 & OOP do not go hand in hand. While he is confident and happy with these potential changes, I was somewhat disheartened. His suggestions all met the first goal of “if it runs Flash Lite 2, the framework needs to work”, but I felt it was losing a lot of it’s utility to developers; they would now have to do a lot more work (typing basically, and not all of it comfortable). However, if you are going to run under the insane CPU & RAM restraints, no one is left unscathed. Welcome to the suck.

After talking to Chris, however, he pretty much corroborated everything Greg was suggesting and had little empathy for my forlorn yearning for OOP. This lead to a very interesting discussion about inheritance vs. interfaces. He’s a big believer in programming by contract; having classes implement interfaces instead of using the extends keyword. After hearing him recount the challenges of being “a slave to the base class”, I totally understand and agree… for large scale projects with uber-knowledgeable individuals. If you are on a deadline, and have a variety of skill sets on your team, abstracting details seems to me like a good thing. I hate those projects, though,hehe! Flash Lite, however, has no room for interfaces as they too are basically objects like classes, and take up RAM. I digress I know, but it’s fun stuff.

Anyway, let me give some context first. There are multiple ways to redraw things to the screen in the Flash Player, with different repercussions in performance and developer approach. You basically have the following options in Flash Player 9:

1. DisplayList
2. Blitting
3. Drawing

Both blitting and drawing depend on the DisplayList, but they are not mutually exclusive. The DisplayList is the act of adding a graphic object to a list of “things to draw”. If the Flash Player see’s it in there, it’ll get drawn. You can remove the object from the list, and it’ll no longer be drawn. This has a lot of positive repercussions in that a graphical object can exist in RAM without taking up CPU resources being drawn. Flash Player 8 and below did not do this. If you created a graphical object, like a MovieClip for example, even if it was invisible, it would still take up system resources being drawn every frame.

Suddenly, the programmer now has control over what is drawn, and when. This makes coding easier, too, because you can choose when to draw things, therefore writing more scalable & efficient applications… just what Flex needed when you started writing these gigantor Enterprise apps that made Flash Player 7 & 8 scream in pain.

There is something, however, more efficient than the above. This is called blitting. It is the act of painting a bitmap to the screen. A Bitmap is a DisplayObject in Flash Player 9, meaning it can be painted, or not painted, to the screen at the developer’s discretion. While Flash Player 9 has removed the problem of “too many objects == too much system resources”, Bitmap never has this problem even if those objects are in the billions. This is because you only ever have 1 bitmap (usually) that you paint on top of. Therefore, you actually have multiple small objects in memory that are then redrawn at some interval to the screen, and ONLY those areas that have changed. This is actually what is probably under the hood of the Flash Player, written in C/C++, that is redrawing all of your DisplayObjects. However, if you know going in that you’ll have a lot of objects which aren’t interactive, blitting is a wonderful technique that scales well, uses less RAM than the DisplayObject , and CPU is minimal because Bitmaps don’t use a lot of CPU to redraw compared to vector graphics. This technique can be used in the Flash Player 8 as well, although, there is no DisplayList so even if the Bitmap isn’t being visibly shown, it is still taking up CPU cycles to redraw it. Both can be combined with double-buffering where you paint to an off-screen bitmap, and copy pixels onto a on screen, smaller one.

Finally, there is drawing. Drawing is the act of using the Flash Player’s vector drawing API to create vector graphics. These are wonderful in that the engine to create them is small, and they take next to no RAM. They look nice in terms of anti-aliasing, and the API is simple enough you can create some pretty sophisticated results by building atop that foundation. The downside is that drawing doesn’t scale very well. A lot of “anything” in Flash Player 8 and below used a lot of CPU, and vector content is definitely a prime candidate to use those resources. There are various techniques you can apply in Flash Player 8 and 9 using cacheAsBitmap, scrollRect, or even blitting the resulting drawing and thus removing the vector part.

The way each of the above is rendered is doing on an interval, called a “frame”. Flash Player was built originally as a vector to pixel renderer , and later incorporated a timeline to show changes in these images over time. In effect, a lightweight animation tool was the result, hence Flash Player’s rising popularity for design in the Internet during the modem, low bandwidth days. These frame ticks run on some developer / designer set time code, like “15 frames a second”, or “120 frames a second”.

What that means is that the Flash Player will update the screen 15 times a second, but more no more. It is not guaranteed to update the screen that many times for various reasons such as screen refresh, CPU usage, etc., but it IS guaranteed not to update more than the set frame rate. This is useful for animators to convey motion. As Ted Patrick has written about in the past, the introduction of the ActionScript Virtual Machine share’s this time between rendering. The important thing to understand, though, is that ActionScript for the most part is single-threaded. This means that once it starts, it doesn’t stop till it’s done running all of it’s code. If you have a lot of “stuff” happening when a user clicks a button, that code is run, and nothing else can really happen in the meantime. There are various threads that the Flash Player spawns to handle network operations, keep tabs to see if theAVM stopped responding for whatever reason, but none of these are accessible to the developer. Bottom line, if you’re code is busy doing things, you’ve allocated less time to the Flash Player graphics renderer to redraw the screen, and thus you’ll lose redraw fidelity.

How do you then lessen this impact, and create an effective invalidation solution given the technology?

In Flash MX and Flash MX 2004, an invalidation routine was created that ensured that components were only ever changed visually once per frame. This was done because while the developer was more than capable of changing a MovieClip‘s x and y coordinates on the screen multiple times in a few milliseconds, it would only actually render once. Changing those properties directly actually triggers a need to redraw event in the Player, and thus you are being very inefficient in your use of the Flash Player. To remove this problem, the first technique was a convention called drawing and invalidation. You would have a few private properties in a base class that abstracted MovieClip , and allowed you to change it’s properties with the ability to then invalidate on your own time. Some would invalidate immediately. It was a good first attempt, but was more formalized and better implemented when ActionScript 2 came onto the scene in Flash MX 2004. While AS2 still outputted the same bytecode as AS, it still implemented the main function that made this possible, introduced flexibly in Flash Player 6: onEnterFrame.

The enter frame event is something emitted every frame. In Flash Player 9, DisplayObjects do this regardless of whether something has a timeline or not, 1 frame or 10 billion such as MovieClip & Sprite. While the name itself is somewhat legacy, it is useful to understand what it really means. First, it is guaranteed to be fired once per frame. Unlike setInterval which can fire randomly (at most 10 times per frame, or less for longer intervals), you can depend on the onEnterFrame function to run when you expect it to run. When it actually fires within a frame context isn’t important. What is important to know is that it is synonymous with “redrawing now”. Since you can only redraw once per frame, it fits in nicely with knowing when you are best suited to do so effectively. FlashMX & Flash MX 2004 would use the invalidate method for graphical components. This meant, “redraw next frame”. So, the developer could change a bunch of properties, styles, etc. and theMovieClip wouldn’t actually redraw itself visually (by setting _x, _width, applying a color, etc.) until the next screen redrawn, no matter how many were changed, or how many times the same one was changed.

This was great in theory, but a lot of the race conditions with measurement and styles made it hard to effectively do. Regardless, it was THE best way to ensure effective redraw. This was slightly improved upon in the Flex 1.5 framework with a more formal system:
– set a property
– if it causes a redraw, invalidate a draw function, otherwise have that property’s dirty flag set
– if a dirty flag was set, determine what type of redraw to perform, a draw, size, measurement, or style change
– invalidate one of those functions if applicable
– one of the invalidated functions would run at the next redraw

This made possible larger Flash Player applications. The problem was, though, that even if things weren’t redrawing and just sitting there, they were still taking up resources, regardless of the wonderful invalidation implementation. Hence Flash Player 9’s DisplayList being invented.

Blitting is kind of a low-level, boilerplate affair, and it’s currently not used a lot by the Flex framework. This is a shame. While the Charting components and Flex components both make extensive use of caching as bitmap and setting effective scrollrects , I think for some of the non-interactive parts of charting, they could scale effectively better. I have not, however, read reports of problems with the charting components not scaling. Props to the power of ActionScript 3!

A significant amount of the framework is being drawn with vector graphics via the Drawing API, and all uses extensive use of theDisplayList.

So, returning to my invalidation implementation, I chose to use the Flex framework invalidation methodology in my implementation of a Flash Lite 2 component framework. Granted, I didn’t have the added or render event’s, but enter frame is all you really need. Seemed logical enough; you want something to efficiently redraw on a device that REALLY needs to be efficient. As I later found, some on my own, some with Greg’s help, there are a few problems with this.

1. Most dirty flags in the Flex framework start out with default values of false. This default value takes up RAM, RAM that I don’t have.
2. I am making an assumption that a developer wants to have the component redrawn when they change a property. This is an incorrect assumption to make.

Regarding #1, one solution is to simply not define default values for the dirty flags. The if ( flag == true ) still results in a correct evaluation if the default value is false, undefined, or null. That way, the flags would only last for 1 frame. Unfortunately, that is still a lot because Objects, what all data-types derive and thus extend in pre-AS3, take up a lot of overhead so even a Boolean has a valid cost to think about.

Regarding #2, there are numerous cases where data isn’t ready for consumption by GUI controls, or some other process hasn’t completed yet. Some of that data or relevant process may be needed to do internal calculations for redrawing. I’m slowly to see ways for developers to implement their own call to invalidate. So, instead of doing this:

attachMovie ( ComboBox.SYMBOL_NAME, “cb”, 0 );
cb.prompt = “– select a value –“;
cb.dataProvider = someArray;
cb.setClickHandler ( this, onClick );

You would instead do (per Greg):

attachMovie ( ComboBox.SYMBOL_NAME, “cb”, 0 );
cb.prompt = “– select a value –“;
cb.dataProvider = someArray;
cb.setClickHandler ( this, onClick );
cb.invalidate();

Notice the only difference is the addition of a invalidate method call. This triggers a redraw. The former, every single public property set, and some methods may trigger a redraw, usually via getter / setters (which have a lot of overhead…). Yes, it’s efficient in that it supports an infinite amount of property setting with only one redraw, but again, I’m assuming the developer WANTS to redraw. Maybe they have the data, want to set it, throw it away, but are awaiting more. Maybe they need the above, plus they are doing it to multiple controls over a sequence so they can effectively deliver a large amount of data through a small series of steps the device can handle. Later, they can then redraw when they deem it necessarey to do so.

In discussing this stratgedy with Chris, he’d recount a lot of techniques Apple’s Cocoa uses as well as some of his own implementations. He found the most successful (and I’m paraphrasing badly here) is when parents are responsible for drawing their children. This allows you can redraw just one section of the GUI, or if you need to redraw everything, you can call invalidate on the main parent, and it’ll cascade down the tree of children, who in turn call invalidate on their children.

What I find interesting is that this technique could not only work in Flash Lite 2 / 2.1 which is basically Flash Player 7, but also in blitting in Flash Player 8 as well as utilizing the DisplayObject API in Flash Player 9. In fact, it’d be really nice if Flex 3 gave us this option. I think the reason it’s not in Flex 2 is a lot of people need an accessible framework; meaning, low learning curve. Some people have never developed or programmed before, or not done that level of heavy GUI work, etc. Bottom line, I think the decision by the Flex team is “developers shouldn’t have to know how the Flash Player works to effectively write applications atop it”. I totally agree. I also believe, however, that there should be a way to optionally allow those of us who know what we’re doing to redraw when we see fit. It’d really help those larger applications, or when you are trying to squeeze all you can out of an implementation, say, Flash Lite 3 which could possibly use the new AVM on a device :: crosses fingers ::.

While it’s more work for the developer, I understand why Greg suggested what he did now, and have more context as to the ramifications what power this really gives developers. I guess I just don’t really have in my head best practices on implementing this. The abstracted invalidation model that Flex has removes any care, and provides a nice convention for me to build atop of. Now, I suddenly have to care when I redraw things which could be different for each GUI control. Furthermore, the less utility I add to a framework by making it more efficient instead, the more the expectations of the Flash Lite community shifts to wanting more features out of controls. Not sure the best approaches to either. Fun, regardless!