There are a lot of good articles already written & screen casts about memory management in Flash Player.  I encourage you to read them first before reading the below.  I cover memory management in a large application using a PureMVC architecture, typically on top of Flex.

• Unloading Flex Modules
• Resource Management in AS3 Session Notes Available
• AS3 Resource Management: Part 1
• Garbage Collector Interactive Simulator
• Garbage Collection and Memory Leaks
• Memory Performance in the Land of References
• Flash Player Memory Management
• Using Object Pools
• Object Pool Class
• Tweening and Object Pools
• Garbage Collection with Flex and Adobe AIR
• Screencast: Performance and Memory Management

Introduction

Memory Management, referring to making sure your application doesn’t leak memory through the use of responsible coding practices, is hard.  It’s even harder if you don’t have a history of having to deal with such things (*cough* Flash Developers *cough*).  It was challenging for me to learn… and I’m still learning.  As the Flex community has grown, we’ve seen the influx of more traditional programmers to help write blog entries and white papers, especially for AIR apps, on better memory management, how to prevent memory leaks, and how memory in the Flash Player works (and doesn’t work).

What I haven’t seen is this applied to Enterprise programming and frameworks.  Most articles are either from a Flash game developer, or a Flex dev referring to AIR applications.  From my consulting, the people who need the biggest help are doing the web applications that need to run as long as a normal desktop application.  Below, I’ve documented how you do this using PureMVC.

Why Memory Management?

Many Flex & Flash projects don’t need to use good memory techniques.  Memory leaks on a small scale are ok.  If the application is responsive to the user, you have no problem.  The purists will argue that creating a bunch of things and not removing them from RAM is bad.  However, if your application works, is responsive to the user, and doesn’t adversely affect other applications, then the purist is wrong.

For things like games that use a lot of system resources, or applications that need to run on your computer longer than 10 minutes, memory leaks will eventually use all available memory that a SWF is allowed, and cause it to stop working, or the browser to crash.  The application will be slow to respond, Garbage Collection may freeze Flash Player while it attempts to clean up, and/or your memory will become fragmented over time.  Fragmented memory makes things slower to access it, and thus can slow down your application.

Another side effect of not cleaning up the memory you’ve used yet no longer need is weird bugs.  You’ll sometimes have objects around that are doing things, yet you have no strong-references to.  Examples include timers still running, service classes that were “removed” yet receive a response from the server and no one is still listening for the response, or duplicate PureMVC classes.  Weird bugs can arise from these situations, causing your debugging time for problems to take longer.

Bottom line: If you want  your application to run for hours at a time, not to adversely affect other applications, and remain responsive, you need to employ good memory management.

Smaller projects that have memory leaks can often be re-factored.  Enterprise ones, however, will often take months to fix.  Worse, those months will have a major lack of visible project progress as things are broken on purpose to get things back into shape.  This is sometimes a contributor to the reason larger projects have the infamous “last 10% is 90% of the work”.  If you don’t start early with easy to follow conventions, it’ll be really hard to add them in later in; “add” is too nice a word since those projects don’t just stop to wait for you to clean up some memory; the logistics are a lot more complex than that.

This is even worse in Agile.  Where 80% of your time is re-factoring and you’re constantly creating new code, it’s quite challenging to keep up.  The best you can do is have a good understanding of the life cycle of your app.  The better understanding you have, the better you’ll be able to know what you don’t have to clean up, and what you do; i.e. when you have to write code and when you don’t.  Remember, designers / UX people can help here.  They’ll be able to give you a better understand of user paths into your app, and you can focus on what truly matters and knowingly let the rest remain in memory.

Example: If the user is constantly creating 3 types of forms, you can focus your efforts on ensuring they are cleaned up.  If the user hardly ever goes to a FAQ page, you can spend little to no time ensuring it’s cleaned up.  The reverse can be true as well.  If the 3 forms are accessed often, you can ensure that only the data they create is unique, but the forms themselves remain in RAM.  You can instead focus your efforts on those items that are rarely accessed.  If you user does go to the FAQ page, they shouldn’t be punished when they leave; it should remove from memory.

Why NOT memory management?

Memory management, sadly, requires you to often write more code.  More code is a negative contributor to projects.  For Enterprise code bases, they are often quite large.  Even when all features are completed, you are often adding more code for modifying features, or fixing bugs.  The last thing you want to do is knowingly add more code if you can help it.

A common practice for classes is to add destructor methods.  These are basically methods that are called “public function destroy():void” and clean up the class.  It’ll remove strong event listeners, remove child display objects, and stop asynchronous classes such as Timers, Sockets, and service classes.  Another common practice is to create “addListeners” and “removeListeners” functions that will do just that; add event listeners to an object/objects, and remove them.

Often, though, both classes fall into the trap of those methods never staying in sync with the rest of the class.  If you create a new object, you forget to clean it up in destroy.  That, or you add a new event listener amongst the other 4, but forget to add it to the removeListeners function.  Worse, and most often the case, these methods aren’t ever called.

The developer meant well, but ended up creating more code, code that isn’t used, and code that isn’t even finished with no obvious indications thereof.  The road to hell is paved with good intentions.

When View’s Go Away

When creating applications, whether Flash or Flex, you’ll have View’s that are shown and hidden.  You’ll have to make the decision how they are hidden.  If you need the View quickly, you’ll set visible to false and true.  If you don’t need to View quickly, you’ll simply addChild/removeChild.  If you only need the View rarely, you can destroy it entirely and rebuild it when you need it.

When you know you need to destroy a View, then you can write code to remove it from memory.  This is the only YAGNI approach I’ve found that works.  Meaning, as soon as you recognize you’ll need to remove a View, only then do you write code to clean up the mess.  If you do it prematurely, you just wrote a bunch of code you may never need, and now have to maintain.  Adding it later actually isn’t so bad, but that assumed you designed for it.  Meaning, if you don’t make it easy to remove a View, it’ll be hard because you first have to re-factor the code that uses it.

From a GUI perspective, this is sometimes ok.  Take the case of a View used in a Flex ViewStack.  All a ViewStack does when you leave a View is remove the View from the DisplayList.  If she’s a big View, that’s nice… but not enough.  You’ll need to refactor this code to support the child in the ViewStack from being destroyed.  You can either wrap the View you want destroyed in say, a Canvas that has code to destroy/create the single child View based on Event.ADDED/Event.REMOVED.  That, or you could make the child a ModuleLoader, and based on Event.ADDED/Event.REMOVED, load/unload the module.  Or, you could just write a function that creates/destroys views based on where it’s supposed to go.  You lose the flexibility & ease of using a ViewStack, but you get more control with more efficient code.

Registering & Unregistering View’s in PureMVC

…but this assumes you can, in fact, get the View out of memory.  In the case of frameworks, larger Views will often use the Presentation pattern, a Mediator, or something like it.  Sometimes it’s directly in the View itself.  How ever it’s done, the “Master View” will integrate with a framework.  In PureMVC, this is usually via registering itself with PureMVC.  There are varying ways to do this.  The way I do it is via ApplicationFacade.getInstance().registerMyView(this), where “this” is the instance reference passed in.  The ApplicationFacade will take the instance, create & register a Mediator passing the instance into the constructor of the Mediator.

public function registerMyView(view:MyView):void
{
        facade.registerMediator(new MyViewMediator(view));
}

Others will dispatch an event, and have a Command class handle those details to prevent their ApplicationFacade from growing large.  Either way, time and time again, I see View’s that should be removed, but have no “unregister” function, nor a Command to remove the Mediator.  Basically, there is a lot of code to register the View the the framework and create associated Application domain classes, but nothing to clean up everything when you’re done with it.

The way I do it is via ApplicationFacade.getInstance().unregisterMyView(this).  This is the opposite of the last method; it’ll unregister the Mediator.  The Mediator’s onRemove function will fire, which gives the Mediator an opportunity to clean itself up.

public function unregisterMyView():void
{
        facade.removeMediator(MyViewMediator.NAME);
}

Weak vs. Strong Listeners

Let me take this opportunity to protest against using weak listeners.  They are for pussies.  Seriously, the only time they are valid is when you are creating code modules that other developers could use, and there is no conventional destruction point.  Since you have no control over when your code will have the opportunity to clean itself up, nor do you know if the developer using your code is smart enough to call your destroy method, you try to code as defensively as possible.  If you must snag a reference to a global object such as “stage” for example, which’ll surely lock your code into memory, you at least make it a weak listener AS WELL AS listening for Event.REMOVED_FROM_STAGE.

It’s been suggested to always use weak listeners when using Timers for example.  Since these are some the major culprits of memory leaks in applications, if you use a weak listener, you’re set right?  Same with Dictionaries with weak keys, right?  *face palm*  Sure man… with that attitude, I’m sure you’re code REALLY makes an effort to help clean itself up.  Besides, Flash Player 9 has some issues garbage collecting weak listeners, so you can’t even depend on it working as advertised.  They’ve fixed a lot of this in 10, but you need to make an effort to de-allocate as much memory as you can.  Just because a large object has no references to itself doesn’t mean the Garbage Collector will immediately free up everything.  While it is a black box, you do have ways of helping it out.  In my experience, it’s more likely to free up smaller objects more often than large objects since running Garbage Collection itself is a processor intensive operation.

If your code is explicit, it’s clear where you’re freeing memory.  If you’re not, it’s challenging to figure out where leaks occur, nor is it easy to write proactive code to free up memory.  Granted, deadlines may dictate otherwise, and I concede the point on those instances.  Regardless, a lot of people who use weak references for listeners forget that unless they keep a reference to the object itself, it’ll randomly get garbage collected, which leads to strange bugs for asynchronous classes that are hard to reproduce reliably.  If you don’t know what you’re doing or have doubt, just use a strong reference and provide a clear way of removing it later.

Any help you can provide… would be… well… helpful!

Make no mistake: Helping the Flash Player garbage collect your objects has major, positive impacts.  Even a half-assed approach is better than none.

Let’s take an Array or ArrayCollection full of Strings for example.  He’s within a View, and you removeChild the View, and null the reference in his parent.  The GC will eat it eventually right?  Wrong… you accidentally have a stage listener buried deep within the bowels of a base class of a Button you’ve written.  If that’s all you’ve done, simply removing the child from the DisplayList and nulling the reference, you’ve at least giving back memory the View was using to display itself to the screen.  Glass half full baby!

Taking it a step farther, what if you’re View implemented a destroy method, and cleaned up its internals to help the GC out, and free up some memory of things it’s no longer using?  Remember, the View itself is still locked in RAM because you don’t yet know about the rogue stage listener.  BUT, you at least remove all Strings in the ArrayCollection and null its reference in your destroy method.  Running the Flex profiler, you’ll see some [reap] and [sweep] using more CPU time + memory go down slightly + less String instances.  Yum yum… while your View couldn’t get it’s fat arse outta RAM, you at least managed to free up all of those Strings it had a reference too.  They no longer have any references to them, and therefore they are cleaned up, and the memory THEY use is freed up.  Rad!  GLASS HALF FULL BABY!  Even with a memory leak, we’re lessening the damage by freeing as much memory as we can.

I know this doesn’t fix the problem, but if you’re under a deadline you gotta do what you can to survive.  The only reason I don’t think weak listeners are justified in the deadline scenario is because it’s like you’re not even trying to do the above: thinking about what View’s & classes need to be removed, writing destructor methods, and validating they worked.  Sometimes you just don’t have time to do those things, though…

What do you do in a Mediator::onRemove method?

Assuming you’re PureMVC is setup correctly, then the onRemove function of your Mediator’s will fire when it’s removed from the PureMVC ApplicationFacade.  Here, you can remove strong event listeners you’ve made on the View, remove the Mediator’s view reference by setting it to null, and clearing any other data.  This is where things get hard.

Removing the View’s listeners and it’s reference is easy; there is a 1 to 1 relationship with with Mediator’s and a View.  But what about Commands and Proxies?  If the Mediator registered one or more Commands, it’s pretty clear he should unregister them.  Often I’ve seen Commands that are relevant to a particular View being registered in the ApplicationFacade by overriding the initializeController method.  For smaller applications this is ok.  For larger ones, it’s still ok in that they don’t necessarily take a lot of RAM.  The problem is this doesn’t scale.  Your Facade will get really really large.  Some people compensate for this by creating multiple Facades for the relevant Application Domain (called Contexts in early RobotLegs builds).  While this spreads the damage, it removes relevance from the code; it isn’t immediately obvious what problem domain these Commands are associated with.  When the Mediator registers them, it’s usually pretty obvious.  That, and it makes it easier to remove them with confidence without worrying about breaking things.

…Proxies, however, are another story.  Proxies are the #1 cause of race conditions in PureMVC.  Regardless of how they are written (holders of data as your Model or Services or both), they are often encapsulated enough that they could be relevant to many different Views, and thus many different Mediators.  This causes a problem in knowing when they should die… if ever.  The benefits of using a stateful client like Flash Player RIA’s, as opposed to a paged based, non-AJAX HTML application, is that you get the benefits of keeping data in RAM.  This in-RAM data allows the application to be quick and responsive.  As long as your data is ActionScript objects (excluding XML, Strings, Bitmaps, and large ByteArrays) your memory usage is often very low.

Therefore, there isn’t a lot of harm leaving Proxies around.  The one side effect is when your profiling your application via System.totalMemory, you’ll notice it doesn’t go down as much when you’re View is destroyed.  Using the Flex profiler, when taking memory snapshots, you’ll notice you’ll have a lot of potential loitering objects (things that are still around in memory even after your View is destroyed and Garbage Collection has run).  If you’re trying to track down a leak, some of those loitering objects, your Proxy and his data, are valid… you want them to loiter in the first place so you can immediately access them again later.  Additionally, you’ll need to write more defensive & proactive code in your PureMVC code that accesses the Proxy.  For example, instead of registering the Proxy every time, you need to ensure it’s not already registered.

private function getSearchProxy():SearchProxy
{
        var searchProxy:SearchProxy;
        if(facade.hasProxy(SearchProxy.NAME))
        {
                searchProxy = facade.retrieveProxy(SearchProxy.NAME) as SearchProxy;
        }
        else
        {
                searchProxy = new SearchProxy();
                facade.registerProxy(searchProxy);
        }
        return searchProxy;
}

Another tactic if the amount of data a Proxy has uses a lot of memory, or you just need every byte you can, is to cache on the local machine.  If you’re running in a web browser, you can use a local SharedObject and save your serialized data.  If you’re using AIR, you can read from the local disk via File, or use the local SQL database support. When the Proxy is re-created, he can check the local cache first, validate the data is good, and use it instead of fetching it again from the back-end.

Bottom line, if you DO remove a Proxy, that’s your cue to go implement his onRemove method as well.  Here’s an example of a Mediator cleaning up its View reference & listeners as well as proxies.

protected override function onRemove():void
{
        if(community)
        {
                community.removeEventListener(LoginEvent.LOGIN, onLogin);
                community.removeEventListener(LoginEvent.LOGOUT, onLogout);
                community.removeEventListener(LoginEvent.CHANGE_PASSWORD, onChangePassword);
                community.removeEventListener(SearchEvent.SEARCH, onSearch);
                community.removeEventListener(PowerEvent.OPEN_POWER, onOpenPower);
                community.removeEventListener(PowerEvent.DELETE_SEARCH_POWER, onDeletePower);
                community = null;
        }

        facade.removeProxy(AuthenticationProxy.NAME);
        facade.removeProxy(SearchProxy.NAME);
        facade.removeProxy(GetPowerProxy.NAME);

        sendNotification(ApplicationFacade.COMMUNITY_SHUTDOWN);
}

And here’s an example of a SearchProxy nulling out internal data, an ArrayCollection and a String, as well as cleaning up an internal web service.

protected override function onRemove():void
{
        powers 		= null;
        _searchText = null;

        if(searchPowersByNameService)
        {
                searchPowersByNameService.removeEventListener(CommunityServiceEvent.SEARCH_POWERS_BY_NAME_ERROR, onSearchPowersByNameError);
                searchPowersByNameService.removeEventListener(CommunityServiceEvent.SEARCH_POWERS_BY_NAME_SUCCESS, onSearchPowersByNameSuccess);
                searchPowersByNameService = null;
        }
}

Remember, once you start implementing clean up methods, you’ll need to keep them in sync with any changes in the class.  Tread with confidence.

Conclusions

Managing memory in Flash & Flex applications is hard.  Validating your hard work actually is paying off is time consuming.  Most programmers who are used to managing memory keep it in the forefront of their minds while their coding, so they often don’t perceive it as an issue/challenge.  A lot of times the frameworks we use handle a lot of the mundane details of these operations for us (like Tweening engines).

If you’re application does need to run for any long length of time and you’re using PureMVC, having a good, up front understanding of what Views need to stick around and which ones don’t will help you better plan for which Views, Mediators, and Proxies you’ll need to write code for to free up memory.  If you do your best to make each View follow it’s own Application Domain problem it’s solving for, then it’s easier to have that entire silo follow the standard removal scenarios, and thus code to it.

For example, if you are creating a FAQ View, and it has a FAQ Mediator with a FAQ Proxy, then it’s pretty simple to clean up thing when you’re done: implement a destroy method in your View, then implement onRemove methods in your Mediator and Proxy.  When in doubt about the Proxies, just keep ‘em around since they usually aren’t the biggest memory offenders… but be sure to validate this using the Flex Profiler and/or System.totalMemory as even rogue XML can take 200 megs or more.  It’s easier to make View’s more efficient than it is to solve for race conditions in PureMVC.

A good tool to help visualize your memory usage in real-time is Mr. Doob’s “Stats”.  Just be sure your system is only running 1 SWF at a time as System.totalMemory, what Stats uses, applies to all SWFs running on your box.  I usually wrap him with a UIComponent, add a right click menu to my app to open him via ContextMenu, and launch him via PopUpManager.  Another tool you can use is Flex PMD.  It’ll help identify which code isn’t used… and perhaps should, like destroy methods that aren’t ever called.  Finally, if you’re desperate on Flash Player 9, you can also force Garbage Collection via the LocalConnection hack.  This saved us on a project where we were loading SWF’s.

Remember, if you destroy & recreate a View and his PureMVC equivalent classes a lot, you may fragment the memory over time, so don’t get overzealous.  Use your best judgement, and don’t be afraid to get the designers help on which sections of the app need to be the most responsive, and which ones it’s ok to lazy load.

Short Version

I’ve built an AIR application in my spare time using Flex that helps you play Dungeons & Dragons 4th edition faster. It took 6 months in my spare time and cost around $50k to build, and $40 a month to maintain.

See and download it here.

It was created in response to my new players, who are tech saavy, having a slow time getting a game going. I copied the design metaphor used in MMORPG games, as well as creating an online database via Django of user submitted content to help the process go faster, as well as supporting house rules. In doing market research, I found my product is for a younger generation, and this makes it challenging to market. I used an Iterative/Agile development process to ensure it got completed.

The app had 3 goals: make money, make our games quicker & easier, and prove to myself I could execute. I succeeded in the latter 2, and failed in the making money… so far.

• Preface
• Introduction: Why the need for Powerz?
• The Power of Power Bars
• Why AIR?
• Why Django?
• Self-Inflicted Iterations
• Quick Note on Scrum
• Schedule
• Technical Details
• Flex 3 vs 4
• Libraries
• Monkey Patches
• Development Challenges
• Feature Casualties
• Marketing & Research
• Target Audience
• Trip to Dragon*Con
• Failures
• Successes
• Conclusions

Preface

I’ve spent my spare time the last 6 months building an AIR application called “Powerz“. Simply, it is an application that allows you to play Dungeons & Dragons 4th edition a lot quicker. It was built in response to newer players hastily flipping back and forth through rule book pages, all the while lamenting how much easier it is to do this in World of Warcraft.

It also became a test for myself. I wanted to see if I could release a product on my own, perhaps even turn a profit. I’ve been attempting for 5 years to create an online version of D&D, fueled by the wishful emails of those around the world who both managed to get their hands on alpha builds and validated my target market. That, and I had recently become an advisor for a startup, so needed creditability that I had produced results in the product sphere as opposed to the service industry which I’m currently in (Flex & Flash consulting/contracting). I had just recently completed my first official Agile/Scrum project getting a startup client to (*ahem* near) Alpha, and upon reflecting, knew I was now capable of releasing a real product on my own.

Introduction: Why the need for Powerz?

D&D 4th edition was created, in my opinion, to cater to, what I call, the “WoW Generation”. Wizards of the Coast, the company also responsible for Magic the Gathering, owns the D&D game. They recognized that to gain broader appeal of a game traditionally challenging to learn and losing market share to video games, they needed to not only make it easier to learn and play, but also target this gaming demographic. The new rules do just that. They are easier to learn for those not inclined to “read a bunch of rules”, and the new power concept allows a gaming group to quickly ascertain how they can contribute to the group.

The problem with the WoW gen’s, though, is that they grew up texting, and not only want, but EXPECT digital tools to be there, as well as cross device. D&D, at least currently, doesn’t totally work like that. They have made great strides in years to provide online digital content, such as Character Builders, rules compendiums, and online magazines with exclusive content. What’s funny to me is that the main goal for most of this content is to be printed to paper and used during gameplay. If you want to use these tools strictly digitally, you can, but that wasn’t why they were made. Ultimately, your still going to be using “real” books during gameplay.

When you have a background in MMORPG’s, or even just gaming on consoles, you look at rule books as primitive, and cumbersome, and ultimately, not as fun. They, like me, know this problem has already been solved digitally.

What kept coming up as a hassle during gameplay with this new generation of gamers was referencing their character’s powers, the main crux of 4th edition rules. I figured if I could solve this problem digitally, I’d be onto something special. I knew I couldn’t solve everything; I’d tried multiple times in the past to digitize the entire D&D experience, and I simply didn’t have enough resources to pull that off. For a digital power bar, however, I didn’t need a lot of resources…

The Power of Power Bars

If you’ve ever played World of Warcraft, Star Wars Galaxies, DDO, or any other Massively Multiplayer Online Role Playing Game, they have the concept of a power bar. There are a series of slots on the bottom, like a horizontal list in Flex, and in each you can drop a power into. It’ll be represented as a little image icon that infers what the power does. Upon rolling over them, they provide details of what they do. Upon clicking them, you use them. While veteran players use keyboard shortcuts and macros, this GUI concept is the core to most MMORPG’s today. Players familiar with such games “get it”, just like you and I understand to click & drag on a scroll bar to scroll text.

World of Warcraft Power Bar

This interface element is important because in MMO’s, those powers, while forming the crux of the 2 most important aspects in those games, chatting & fighting, they also serve another subtle purpose. RPG games in general are technical. They basically take a lot of math, specifically ratio’s & percentages, and personify the results through the use of chance. While neat from a geek perspective, newbie players, especially those not tech saavy, can get overwhelmed by information overload.

Just like how Word hides non-frequently accessed information in menu’s that reveal, so to do MMORPG’s “reveal” information when the user requests it. Additionally, since these interface elements are actually over top of a 3D virtual world, they need to be as non-intrusive as possible.

In short, they provide the user with the ability to learn what their character can do, when she/he can do it, all using a mnemonic device of visual icons that relate to the action in a non-intrusive interface.

Why AIR?

I’ve bashed her in the past and got shot down by desktop developers who came out of the wood work.  To this day I still don’t get clients asking me to build AIR apps.  Regardless, it was a pretty clear choice as to why I needed AIR: offline capability to run Flex apps.

You can play D&D ANYWHERE. I’ve played at friends houses, at a camp site deep in the woods, on a plane, a bus, etc. All of these places don’t always have internet. Nowadays with wireless and cell phones that have decent coverage, you can tether on your device to ensure you have an internet connection just about everywhere. “Just about” isn’t good enough; if you can’t use it, or it’s a spotty connection, you view the “cloud” as holding your data hostage vs. some benevolent caretaker.

So, I didn’t want an internet connection to be a requirement, and thus, targeted AIR vs. a web browser. If you know me, using the Flex SDK was a no-brainer. If you don’t, here’s the run down:

1. Needs to work on as many platforms as possible. AIR works on most PC, Mac, and enough Linux distros. I use a Mac mostly, my gaming group uses Linux and Windows. The Character Builder for DDi works on a PC with .NET. Using CrossOver or Parallels isn’t cheap. Buying an OS + emulation software when you already have a working computer is lamesauce.
2. Writing my own component framework is time consuming. It took me 6 months to build Shuriken for Flash Lite 2, I’m still working on CorePlay for pure AS3… so if I ended up doing that, I’d spend all my time on developing code to develop code vs. finishing my product. Flex has all the components & styling features I need, Flex 4 more so.
3. I make my living developing in Flex Builder. This is what I’m most comfortable, and thus productive in.

Why Django?

Once I personally started using Powerz in my game sessions, taking her for a test run with my gaming group, a new problem arose: setup time. While battles finally went a lot faster, and focused more on strategy and chaos vs. wtf can I do, actually STARTING to play took awhile. Players had to manually input the powers from the Players Handbook into Powerz. Without OCR, this is a lot of meticulous hand typing.  This included those who played with made up house rules.

I thought, why couldn’t we do something like the CDDB days of old? CDDB was an online database where people would submit CD track lists. That way, when you turned a CD into MP3’s, it automatically filled out the song names and album name for you.  By the time it got popular, most CD’s were in the database so you didn’t need to submit them yourself. This was the first Web 2.0 (aka the read-write web) phenomenon that I remember. Basically you are leveraging the community, much like Wikipedia leverages the knowledge from every contributor in the world. This is a double-edged sword, as sometimes CDDB would have incorrect data without an easy way to change it. Wikipedia has had its fair share of drama about article authenticity, and who defines the truth when multiple authors disagree about the same topic.

Regardless, both are using the forces of good, and showcase how leveraging community input can greatly benefit the masses. Two heads are better than one, and thousands are just f’ing awesome.

I set out to build an online database of powers so when one person submitted it, everyone else could use it.  The most common ones, say all 1st level ones for all classes, would be ready to go. That way, even new players could quickly get up to speed. Powerz is supposed to speed up gameplay, and this would be the final lynchpin.

I hate, and suck, and server-side anything. I’d argue the majority of arguments, philosophical debates, and technical podcasts about software are regarding server-side development (iPhone apps aside). When people talk about Inversion of Control, server clustering, or Aspect Orientated Programming, my eyes glaze over, I face palm, and am like, “Wow… that’ll solve like 10% of my problems. That 10% is on the server… WHERE MY CODE ISN’T RUNNING!” To be fair, the majority of all cool software is running on the server. Most Flex apps I’ve written are merely pretty faces for pimp server-side action. The problem with server-side stuff is that you can’t see it. I’m a visual person, I went to art school, and if I can’t see it, to me it’s boring. I like seeing visual results from my coding efforts, and no, logging statements don’t count.

So… I knew I was in trouble with my idea. Every time in the past I had such an idea, I merely turned to PHP, since finding good server devs looking to work for free is really hard. PHP is available on just about every web host, for free, and there are a lot of resources out on the internet for it. It’s also very similar in syntax to ActionScript, so learning it was easy. The problem is, I’m not “in” the PHP community.  Creating a user account system may seem like a reasonable programming task to me, but an ignorant waste of time to a veteran PHP programmer who has used Zend or Drupal on many projects. Most of my endeavors into larger PHP systems have failed because the workload was just to large for me to tackle both the front and back-end. I knew this time I HAD to find whatever framework others were using for a user account system. I’ve failed to find like-minded server-side developers in the past for such ideas. I actually had a friend offer his services for this very project, but I was concerned I might actually make money, and I make a rule of not mixing friends and business.

However, I had just got off a Django project. Having spent 5 years or so evangelizing AMFPHP, yet never actually successfully getting native object passing working, I decided I’d see how easy it was in Python. Additionally, my experiences using Django’s user administration interface along with PyAMF Flex integration was a very pleasant experience. Since I’d already taking a look at Ruby on Rails awhile ago, I figured I’d give Python a chance.

Best decision ever.

I don’t know Terminal, my SQL knowledge equates to SELECT * FROM *, and I’ve never coded Python before. Django is the shit. It’s so easy, I learned my way around Terminal enough to not only “run a local server” (I know, right?), sync my database, and even play with Python code, just like Director’s message window. I even learned how to remote ssh into my production server, install PyAMF remotely, and deploy my code. Django is merely a framework atop of Python, but you have to know all that insane crud just to do it. Oh yeah, and I ended up buying TextMate (in part to Kristofer’s recommendation) to use for coding Python.

Speaking of Python, it’s a LOT like ActionScript 1. Really strange going back to loose typing. Unfortunately, the only real benefit I found was more succinct code. The Aspect Orienated/Functional stuff that uses the Decorator pattern with metadata tags for designating what methods require the user to be logged in only works for more recent builds of Python. Still… I got it and think its hawt. Additionally, the fact that you can import just a method of a class and use it is neat. There’s been a request on an unnamed list requesting ActionScript 3 get this ability so when you import 1 method of a library, you don’t end up importing the whole library, and thus incur the file size penalty. Since SWF is for web, this makes a lot of sense in investing in such a feature. Python’s for loops? Holy fish, advanced, and really cool stuff, way easier than doing similiar things in ActionScript 3 (faster? doubt it). I still don’t get their slicing syntax, but I’m sure if I did it every day I would.

So why Django? Within 3 days I had learned Python, Django, and PyAMF integration to have a working user authentication system working in my Flex app. Blah blah blah… open source has no value… blah blah blah.

…I suck at debugging it, though, a friend told me to look into Python logging, but I just ran out of time… print print print OT OT OT. That, and my admin only works on my local box, not the remote server… *ahem*.

Self-Inflicted Iterations

All my personal projects suffered from the most common symptom that affects a lot of us software developers: starting new projects is ALWAYS more fun than finishing them. You accrue a long list of awesome ideas that never make it to fruition. Of those, a few you blog about. Knowing this industry, a bunch are probably worthy of actually finishing and marketing because they’ll make bling… but… we’re a busy bunch. That, or you see no point because you have no clue how to monetize them.

I’ve had this happen 3 times with my major D&D online project in 5 years, 3 times with some games, and once with every other project that wasn’t specifically created FOR a blog entry since my career started. I almost swore a blood oath before starting Powerz. I didn’t really have too, though, because of circumstances.

First, I had just got off the best gig I’ve had in my entire career. It was working for a startup as a service consultant through Enablus. So, I make bling, but see $0 bling when they insanely succeed. Quite the soul reflection project. Second, I read a book called “Rich Dad, Poor Dad” upon a recommendation form a friend, and was then pissed for 3 weeks straight. Third, I now knew how to do Agile development which was better than my made up iterative process of the past. Fourth, D&D 4th edition really is slow if you don’t know your powers by heart; it’s even worse for DM’s who have 50 billion other things to keep track of. I needed this tool!

I made a simple rule. If I was going to work on Powers, I needed to release SOMETHING a week from when I started. If I grabbed a beer and started coding on a Friday night, there better be a new build, with associated forum post, WORKING up online the following Friday. If I had something come up, like say real, paying work, or my kid(s) got sick, or whatever…. then 2 weeks. If that didn’t work because, say I only had 4 hours in a 2 week period, then I’d lock myself into finishing just one feature before I was allowed to work on anything else.

Failure was not an option. To fail is to fail myself, and everything I stood for, and for those who listened to my blog entries full of “advice”. If I did fail, it meant Agile didn’t work, and all 10 of those blog entries I wrote espousing it were utter bullshit, and I needed to die in a fire. Seriously. After being so frustrated for so long about “figuring out” software development for the past 9 years, this was the line in the sand. Pick your platform, pick your tool, pick your framework, and either put up, or shut up.  Get something done vs. proselytizing about “how to program” with nothing to show for it.

In short, finish something I started.

6 months later, I’ve released version 1.0. That feels f’ing good. Really good. That, and D&D is easier to play now for me, and a lot of others.

Quick Note on Scrum

Remember, if you ever hear about this Agile or Scrum stuff, and are like, “Wtf?”, don’t stress. All you need to know about using Scrum correctly is release working software with a new feature every week (or every 2 weeks). That’s it, that’s all you need to do to be JXL Scrum compliant. If you don’t do that, you should really take a look at what you are really trying to accomplish.

If you’re a manager who’s trying to implement it, and your developers keep going off on paths they weren’t assigned, play to their ego’s. For example, if someone says, “I could write a login form in less than a day!”. If it then takes them 8 days for the form to actually work, call them on it. I guarantee you they’ll spout off about “OOP/Framework setup”, or some other lack-of-DRY-adherence excuse. That’s ok; the point here is to show them that what they thought they could do, they didn’t actually execute on. They’ll either re-assess what it really takes to develop even simple things, or you’ll come to a mutual understanding of what “working” really means.

I had to do that. In 2001, I created a stock charting application in Flash 5, AS1 pre-#initclip days, knowing very little OOP, zero best practices, and no frameworks, in 1 month. Today, that same application would take me 4 to 6 months, and cost 177% more per hour. Granted, the devil is in the details (quality of user experience, less bugs, more features, I actually know what I’m doing now + have experience, etc), but that, to me, says something about my ability to execute. Those in software know faster doesn’t mean better… but when you start splitting hairs about even simple features, you find some surprising things about yourself. I call this, programmer self-awareness through Scrum.

Schedule

For those who are interested, here is how my schedule worked out. When I mention a week, I’m speaking about a week of development time. So, if I started coding on Friday night, but only put in 4 hours for an entire week… I certainly had very little to offer in a true build. So, my first rule was, if I could do a week, do a week, and release. If I couldn’t because I had real work to do, then I’d instead get 1 feature DONE and release. This was still hard, because even if I could only devote like 2 hours a week to my personal project, it HAD to be on that one feature, and I had to force myself to finish it so I could get a build up, and move onto another. It was far from perfect, but it’s as Scrum as I could get with my free time… and it guaranteed I’d release multiple builds with enough room for user feedback.

• First 3 weeks, spent designing & doing initial prototyping (aka design phase in Waterfall). I had no gig lined up, so worked 3 weeks straight on my own stuff. I worked for about 7 days straight in Photoshop CS3.
• Work 1 hour one night, 4 hours another, 8 maybe on a weekend, or perhaps none for 4 days in a row. Bottom line, I still followed my weekly/bi-weekly/single feature release schedule.
• One build took 6 weeks (Alpha 9 to Beta 1) because I had to learn Django and do real work at the same time. Some weeks during that time, I only worked 1 to 2 hours per week, others, I’d work 3 days straight.
• Some weeks were coding. Some weeks were pure Photoshop. Three weeks was learning/implementing Django. Four weeks was setting up the website, porting all code & assets to work on it, and installing all forum and blog software. All was mixed amongst each other. Since I was doing everything myself.
• I did Analytics once in the beginning, and then changed it 2 weeks later when I couldn’t understand my own metrics.
• Start Date: Around April 5th, 2009.
• End Date: Hopefully September 26th, 2009.
• Total Time: 6 months

It’s hard for me to predict how long Powerz would of taken had I been able to spend 100% of my time on it. A lot of things came up, positively & negatively, during development that adversely affected the timeline. Users asked for features, work flows I originally tried didn’t pan out, Photoshop designs took a few tries, and learning Django staggered overall development on the Community portion. All I know is, Flex Builder 3 and Flex SDK 3.3 is foogin’ solid for what I needed.

Technical Details

The web site is pretty standard stuff. Forum is PHPBB, and blog is vanilla install of Wordpress. Hopefully post 1.0 launch, I can modify the blog to match the rest of the site look and feel (excluding forum) via the Dreamer, an easier way to design Wordpress sites via Dreamweaver. Her majesty used it for her site and it’s pretty cool.  I also need to redo my dated screen cast, and tons of other things.

While I did the original web site designs in Photoshop, the site itself was done by her majesty and I in Fireworks.  Fireworks’ text handling is superior to Photoshop’s, and now that CS4 shares the same font engine, you can get good looking interfaces.  If Photoshop were to improve their text handling capabilities, I’d probably still use Fireworks for it’s easier object editing + grouping + moving functionality.  For the web with lots of content, it was just quicker for me and her majesty.

For the video, shot using a Sony DCR-SR200, and edited in Premiere CS3 & After Effects CS3.  ScreenFlow was used for demoing the software.

Flex 3 vs 4

Yes, I very much wanted to use Flex 4. Unlike Flex 2 Alpha, however, she’s unstable and changes all the time. You could build some pretty Enterprise stuff on Flex 2, Beta 1. I would never do such a thing with Flex 4 just because she’s in such flux, and the IDE (+ Catalyst builds) is somewhat flaky. Obviously 6 months later things are more stable… but they still changed a lot (fx prefix, Halo interop, etc.).

I used Flex Builder 3 & Flex SDK 3.2… then 3.3… then 3.4. I use AIR 1.5.2. AIR still has a bug where you can’t seem to use compc (aka Library Projects) because of some weird Vector bug in playerglobals.swc or something. So I just gave up on using a library project.

Libraries

I have 1 Flex project, and a ton of libraries. Ones of note are:

• Cliff Hall’s PureMVC
• Ely’s SuperImage (that I patched)
• Claus Wahlers’ and Max Herkender’s FZip for reading & writing my custom file formats
• Christian Cantrell’s NativeAlert ported to Flex
• Google Analytics
• Google AdSense
• as3corelib for JSON, used in reading & writing custom file format
• Adobe AIR Update framework (which refuses to GTFO of RAM) with custom badges written in Flash (zomg!)
• my DebugMax window which is a lame attempt to copy that older Flash debug window (the Portuguese one)
• …and Grden’s localToLocal function
• Server is running Django on Python, using PyAMF to send AMF objects back and forth. I’m only using one class; getting nested ValueObjects to work with inherited and one to many relationships in Django’s Models was nigh impossible to debug.  For now, I just pass simple Objects which are uber-easy to parse.

Blah blah blah… open source provides no value… blah blah blah.

Monkey Patches

1. Commented out a bunch of InterManagerRequest code in AIR’s NativeDragManagerImpl class. Doing drag operations would cause the app to lock up for 7 to 11 seconds. Not sure if this was fixed in 3.4, but I don’t load multiple Flex apps with different SDK versions, so don’t need this code.
2. Added bitmap smoothing in SWFLoader, which Flex SDK 3.4 (or 3.3 I forget which, added officially)
3. Added a numLines getter to Label for more accurate text height measurement… which I then later didn’t use.  Proper invalidation fixed my measurement bug; text can be finicky, but if you wait a frame……

Development Challenges

There were 3 development challenges I faced.

The first was developing in AIR. If you are using Flex in AIR, it’s nearly the same thing. The hard parts were learning how Windows work, and their quirks (such as when does Window REALLY get its reference to nativeWindow?). How do you design for this as a designer, and what is an appropriate design metaphor? Multiple windows, or 1 window with many states? Also, event bubbling doesn’t work with windows, since each has it’s own stage, so there was a learning curve of when to manually re-dispatch, and when to just hand off to PureMVC Mediators.

The second was learning to work with reading and writing files. The FileReference class is EASY, but there are actually 3 classes (really you just work with 2) for AIR. They too have intricacies to learn, and how you can abstract them, and what the costs are in doing so. What was fun to figure out was how to associate a file with a document window in the PureMVC framework. I really had a tough time trying to tightly couple something when you’re technically not supposed to tightly couple anything in programming.

The third was creating a file format. I believe most traditional developers would of used some sort of serialization via ByteArray and the IExternalizable interface for your objects. To me, that stuff takes a lot of work, and isn’t easy for me to debug, nor verify if it worked. Maybe to those who think in 0’s and 1’s, but not me. It took some tinkering; I copied how SWC works.  SWC is a ZIP file with some XML and binary assets all inside. My file formats are basically the same thing; a ZIP file with a JSON text file holding the serialized object, images for any icons, and a text file determining what type the file format is (since you can’t rely on file extension). This allowed me to open up the file to see it’s contents and confirm my code worked as well as perusing the JSON to see if my objects serialized correctly.

What was great about being the only developer was that I was also the designer and back-end developer.  I knew how to design assets for Flex and could predict what I’d need designed. As the back-end developer, I could create data that the client would need, thus making my job easier in both places.

Feature Casualties

I had 2 features that caused major scope creep, and thus extended how long it took. They weren’t a waste of time, but as with every project, I learned something.

The first was the whole “document editing process” that people are used to. You open a file, edit it, save it if it’s dirty, and close the “document”. If you “Save As…”, you create a new file, yet use same window. I didn’t use this model at first. Instead, I just opened files, kept a reference to the file handle for each AIR window, and removed it when the window closed. However, this is not how every other desktop application works. Additionally, for those on network drives, you couldn’t really work with the files easily when moving things since it always had an opened file handle. So, it was a pretty big re-write/re-factoring effort to put this functionality & work flow in. I know how to do it now, but I’m just glad no one asked for undo yet. *whew*

The second was pimping out Christian’s NativeAlert window. I spent 3 days in AfterEffects when I was between gigs creating a transition in and transition out for the window.  Instead of just popping in, it’d “materialize” in, all magical like. To me, and one of my users, it looked awesome. It got old after 2 uses because you had to wait for it and there are a few alerts when saving files. If I speed it up any, it loses fidelity… so I just removed it. It required a lot of custom code to make the alert do it’s modality magic in time with the transition. I didn’t get to use any of it.

Marketing & Research

This was the most interesting for me. While I’m not a good salesman, I feel that I get online marketing and branding. I also feel like I understand my target audience, since I’m part of that audience. I actually did market research (aka learning my target audience and their needs) both at the start and during development.

I looked online and read about what WotC was offering, and then I looked at all the 3rd party offerings. I took a few for a spin. I read user reviews on forums and blogs, especially the passionate ones. I wanted to gain a better understanding of what they liked about the software, and how they expected software to work. There wasn’t really a pattern. The bitching about lack of Linux/Mac support was low, and a lot of the 3rd parties utilized Java. For those that didn’t, there we already well documented ways of installing a Windows emulator.

One thing that reading the forums did for me was really educate me on the current flux of the industry. Technology really has changed, and continues to change, table top gaming. WotC is in an interesting time of the market, and frankly I’ve think they’ve done a good job on positioning their D&D brand & rules for 4th edition to perform well with these changes. Where they haven’t done entirely well is executing both their community aspects online, as well as delivering their associated digital tools.

The short version of WotC’s transition to the digital age is, as magazine sales declined, WotC first sold their magazines to Piazo, and later brought them back and incorporated them into their new online subscription service; they no longer print magazines. You get the monthly magazines digitally, exclusive content, as well as access to their desktop and web tools. There are good and bad things about this. Some of their software isn’t released years later, but the ones that are are definitely nice.

Target Audience

I wrote something similar to this over a the Powerz blog.  Bottom line, there is a lot of confusion, at least for me, on who really is the target demographic. The high level view is that the older players prefer tangible rule books, whereas the younger ones are more apt to digital content. That’s where the black and white views end. Some from both groups are fine with digital content, but intend to transfer that content to paper in use in their games. Others would prefer to wholly digital, yet still play in person with their friends; like a LAN party.

I intended to confirm this suspicion beyond just talking online…

Trip to Dragon*Con

…so I took a trip to Dragon*Con 2009 this year. I go every so often to stock up on new dice as well as to see the 501st. My goal this year was to promote Powerz, and meet potential customers. I knew it would be challenging for a few reasons. First off, finding D&D players is actually hard. It’s a huge Sci-fi & fantasy conference; it’s not solely for D&D’ers. Additionally, most D&D’ers who do go play with their friends at predefined tables in a reserved hotel ballroom. These are usually niche groups, and while friendly, you can’t just barge in on them in a middle of game. If someone did that to me during mine, I’d be really ticked off. Additionally, the conference itself is confusing because it’s spaced around 4 semi-adjoining hotels (Hilton, Marriott, Sheraton, etc.), and signage isn’t large/verbose enough for you to get a clear picture of where you are, nor what events are happening where. I knew it’d be hard. Her majesty helped set my expectations to a reasonable level.

My first attempt was while waiting in dice line (like a food large buffet, except there are different colored dice you can choose from). One particular customer was taking awhile to get his transaction finished, and was in a TF2 spy costume (I think), so I figured he was a good target.

“Hey chief, you play D&D?”

“Yeah!”

“Cool, you play 4th edition?”

“Naw, 3.5.”

*sigh*

I then asked the attendant who was waiting on customers, but he didn’t play, just pointed to one of his co-workers; but they were UBER busy. Even Solid Snake briefly came out of his box to browse their wares. I didn’t want to annoy every customer, so moved to another vendor booth, full of D&D books.

A girl was just sitting down in a chair, and lamenting about how good it was to sit down. Prime target.

“Hey, you play D&D?”

“Yeah…”

“4th edition?”

Irritatedly, “Only because my group made me…”

“Rad, do you mind if you ask you a few questions about some software I’m building?”

Cautious, and body language suggesting she was about run at the slightest sign I was a vampire, “Ooookkkayy….”

I sat down next to her on the floor to make her feel more comfortable with my laptop, and briefly walked her through why I created Powerz, and what it does.

She immediately got excited, “Oh we already use that through our DM. He uses WotC’s subscription service… DD something…”

“DDO or DDi?”

“Yeah.”

“Which one?”

She shrugged but excitedly explained, “I don’t know, but it’s great. You can input all of your character data, and it prints out all your powers. Our whole group does it.”

I then proceeded to ask her questions about how her group plays, and take notes. Another gent from her gaming group joined in on the conversation, and motioned to a 3rd member who was also browsing books nearby. After I had asked my questions, I thanked them for their time… and went looking for a bar. I suddenly felt insecure, ‘How could DDi do all that? There is no way, I’ve seen the screen shots, read the reviews….’

While waiting for the bar tender who was overworked, and was already getting heckled by some disgruntled women next to me, I reflected on those I had talked to. I knew a lot of the older generation was very pro-3.5, me included. I really like, still, the 3.5 rules and felt they were akin to Macromedia’s Flash 8; the pinnacle of the company. I started getting really depressed, and quickly started going over all of my past research in my head.

…then I remembered something the girl had said. “Print them out.” Print? To paper? wtf? A smile came to my face, I gave up waiting on the bartender, and quickly headed to where I remembered the D&D gaming tables being the previous year. I stopped by the bellman’s desk to obtain some change so I could buy a few people beers in exchange for me pestering them with questions.

Paper. That was really what I kept seeing. The generation gap in that one prefers tangible assets to play with and the younger ones do not. Again, it’s not that black and white sometimes, but it’s still true. While waiting 1 hour and 40 minutes in line, I met a father who had brought his 11 year old daughter to “meet” the creators of the online game she played. It was some alternative to Club Penguin, which was apparently ruined under new management. She was really excited, and wanted to be a programmer when she grew up. The father, clearly out of his element, saw me in my business professional getup, clearly over dressed for such a casual, and often costumed event, and seemed happy to have some one to talk to. He found it strange that his daughter would spend his money on these online games for things that didn’t exist. Instead of a new barbie doll, she’d get a some new, exclusive outfit for her online character. We discussed her online habits, and those of her friends. I explained what I did, and he was intrigued.

His daughter is my target audience. The girls and boy in front of her, definitely early teens, one dressed up as Naruto, and others as Sango from InuYasha… those are my target audience. They grew up texting, the internet always existed. Those who were my age, dressed up as star trek characters… those probably aren’t my target audience. These kids revel in online content, but still have social interactions with each other in person.

People like me, and the kids who are used to digital tools don’t mind keeping things digital. Others see digital tools as a means to a paper end.

Upon leaving, I was convinced my target audience was mostly online and not at Dragon*Con… or at Dragon*Con WITH their PARENTS. It was a great learning experience, even talking to the few people I did was helpful.

Failures

The project as a whole, while successful, failed to meet one of its primary goals: make money. It does have some successes, which I document below. I had two main goals: releasing a product to prove I could execute, and to make money, enough to cover costs. Currently, there is not enough community features to justify a subscription pricing model. Additionally, I highly doubt I’ll get enough users to cover the costs via the ad supported model I use currently.

If you add up all the costs:

• Around 6 to 10 weeks of man hours (design, development, marketing + expenses)
• hosting costs ($20 hosting + $20 Django container @ Mediatemple)

The total comes to around $50,000. Each month I go $40 more into the hole as the ads currently aren’t providing enough revenue to cover costs, and I haven’t figured out how to monetize the anonymous usage data.

I have a new idea that will actually generate revenue, and Powerz could actually be a companion application, so that’s a positive.

I’m not sure about this statement, but… here goes: I’m glad I know I’m capable of executing vs. knowing how to make money and not execute. This, as opposed to someone who knows how to make money, but can’t execute. The sad fact is, most rich people I read about know how to make money, and get other more qualified individuals to execute for them.  Doh!

As a developer, however, I view this as yet another thing to learn, so that’s a good thing! If I start learning how to monetize things better, I can focus my development efforts on what matter monetarily, and eventually find easier ways to leverage help vs. doing everything myself. If at first you don’t succeed, try, try again.

Successes

My favorite part, the wins.

1. I now have a tool to make my D&D gaming easier, and more fun.
2. Others now have access to the same tool, with the ability to help guide it’s direction.
3. I’ve proved to myself for the 2nd time that Iterative development works.
4. I’ve proved to myself that I can execute on my ideas.
5. I learned Django & Python, and also learned not to hate the server-side.
6. I’ve learned, again, how much work it really takes beyond just coding to release a product.
7. I learned a little bit about engaging potential customers.
8. I learned a little bit about market research.
9. I learned how NOT to migrate data.
10. I now have something new to learn: How to make money with software.

Conclusions

While I’m disappointed I failed to monetize my work, it’s still early in the game, so maybe I’ll think of something. Overall, I’m really happy to have finally finished one of personal projects into an actual product. That, and I learned a ton. Soon, onto idea #2!

If you play D&D 4th edition, check it out and let me know what you think. If you don’t, but know people who do, let ‘em know; I’d be most appreciative!

1. Part 1 – Stressful
2. Part 2 – Code Refactoring
3. Part 3 – Branch Workflow
4. Part 4 – POC, Strategy, and Design Challenges
5. Part 5 – Acceptance Criteria & Punting
6. Part 6 – Tools, Extra Merge Day, and Postponed Transitions
7. Part 7 – Bugs, Unit Testing, and Throughput
8. Part 8 – Demo, Burnout, and Feature Juggling
9. Part 9 – Scope Creep
10. Part 10 – Conclusions

This entry is about utilizing branches for each developer in Subversion, Merge Day, and how while cool, it’s an ivory tower process.

Note: This isn’t a tenet of the Agile methodology itself, it’s just something that works well when you have a bunch of developers collaborating together rapidly, and a specific workflow our client requested we follow.

Branch Workflow

We’re utilizing a Branch Workflow on my current project.  What this means is that each developer creates their own branch in Subversion.  If you’re utilizing Tortoise SVN on PC or Versions on Mac, this is effectively a folder.  As you may know, Subversion has 3 default folders that you typically utilize in a repository, and hopefully each project gets its own repository.  These are branches, tags, and trunk.

Some people choose to ignore these.  Some put multiple projects into the same repo.  Both are fine because the mere fact people are utilizing source control in the first place, even if its just for disaster recovery, is great.

These folders, however, really do have a few reasons for their existence that have been well thought out.  Each source control system tries to out-do the next.  In Subversions case, the simple definitions are trunk is for the current version of your project, tags are for multiple older versions of your project, and branches are for experimental features or code that will possibly be merged back into trunk later.

Those are EXTREMELY simple definitions.  If you read the SVN book as well as other people who incorporate SVN into their workflow, the definitions and purposes can get quite complex.  For the project I’m currently on, branches take on a new meaning.  Not only are they where experimental code goes, but they each represent a user story.  Meaning, if I’m working on completing the user story where the user can modify their hardware device settings via a Flex form, that’ll go in it’s own branch.  Using source control to do this is pretty straight forward.  Instead of checking in here:

https://svn.server.com/repos/project/trunk/

You instead check-in here:

https://svn.server.com/repos/project/branches/sprint2/deviceform/

And the other developer with me would be working from:

https://svn.server.com/repos/project/branches/sprint2/registration/

Pretty easy right?  It gets even easier to check in.  No more, “Dude, have you checked your shiz in?” to another developer, or the cardinal rule of doing an update before checking in.  Now you can check in to your hearts content knowing it’s your own repo to abuse how you wish.

This has the side benefit of NOT breaking trunk… usually.  Again, the point of having frozen code you KNOW for a fact works is to use a tag, or a series of tagged builds.  A lot of developers get seriously bent out of shape if you break the trunk.  At one job, whoever broke the trunk had to use this monkey icon for their IM icon for one week.  It was funny until the 50th person asked why I had a silly monkey icon, and by that point, I was swearing I’d never break ANY trunk ever again.  This can possibly have the effect of keeping ‘em happy.  At the very least, this reduces the possibility of it happening because no one is working from trunk.

Instead, you check into trunk at the end of every sprint, or for my team, from Merge Day onwards.

Merge Day

Part of Agile is to have a day a few days before your UAT where everyone merges their working user stories into the same code base and “works out the kinks”.  Since everyone was coding in a different direction, this can be good and bad.  Good because they are working on completely unrelated things, but in GUI coding, we all know some things are extremely related.  Specifically, these are your data model in the form of ValueObjects, calls to a framework such as the use of Event classes in Cairngorm and PureMVC Mediators, and modifying settings in higher level view’s or CSS.

Since this can potentially have major consequences for some portions of a user story, or set of stories, it’s important to devote a pre-scheduled time where the team knows their code will be merged with everyone else’s code.  This is called Merge Day on my project and it happens every other Wednesday.  I take my branch and my co-workers, and merge his code into mine.  Then, I fix things that explode.  This could be as simple as just merging code over, or a complex verbal discussion to work towards resolving 2 different implementations.  No code is trivial.  If another developer added a style to MXML for example, setting the background color may seem small to you, but could be the difference between working or non-working code for the other developer’s code, so its important you do these merges together.

Once that is done, I merge into trunk, test again, fix explosions, and finally check in the working build.  I then upload a new build, or in the case of my current project, deploy a new build in a new sandbox utilizing Django’s web interface.  It can deploy to the web the latest bin-release that’s checked into SVN by merely clicking a button.  I’ll send an email to all members of the team identifying what’s new in the build.  This includes user stories, fixed bugs, and other things of note that are different from previous builds.

For fellow developers, this is a courtesy.  For Project Managers, it’s a necessity for them to set client expectations for Friday’s UAT.  For everyone it’s a Sprint milestone as well as reality check.  You can identify what user stories are done, which ones aren’t, what older ones you may have broken, and which current ones have issues.  You can then plan with your team what to spend the next day and a half on.  This is also where the rules can potentially break down.  What I’ll usually do is check my local copy into a new branch, and either fix partially completed or broken user stories, or just finish what I can.  The temptation here is to abandon branches since you only have a day and a half and it isn’t worth the trouble; just don’t break trunk in the meantime.

…riiiight.  The last thing you want to have happen the day before UAT is get all stressed out because trunk is broken.  To me, it’s worth the extra time in using the process to ensure your UAT prep goes smoothly.  I’m conservative.

I say that all high and mighty like, but on Sprint #2, I did just that.  Did I get lucky or was it just mad skillz?  Luck.

Conclusions

Now, all of the above processes could have been done with just utilizing trunk.  As you can hopefully see utilizing a branch for each developer, or multiple since creating a new folder is really easy, helps things go a lot smoother without surprises.  You can also plan for the chaos on Merge Day, which while extremely stressful for whoever is doing the merging, is at least expected and a concerted team effort.  I really have enjoyed the Branch Workflow so far.

…however, I don’t see it catching on.  Every Flash & Flex developer I’ve ever talked to doesn’t use branches.  Sam Robbins mentioned over Twitter they might start adopting this workflow at their place of work, and that’s great, but again, it seems to me most developers feel trunk is good enough and branches are just an unnecessary complication.

Stay tuned for #4 in the Agile Chronicles series where I talk about the POC (Proof of Concept), business strategy, and design challenges.

1. Part 1 – Stressful
2. Part 2 – Code Refactoring
3. Part 3 – Branch Workflow
4. Part 4 – POC, Strategy, and Design Challenges
5. Part 5 – Acceptance Criteria & Punting
6. Part 6 – Tools, Extra Merge Day, and Postponed Transitions
7. Part 7 – Bugs, Unit Testing, and Throughput
8. Part 8 – Demo, Burnout, and Feature Juggling
9. Part 9 – Scope Creep
10. Part 10 – Conclusions

This entry is about the joy of coding quickly, finding the balance between getting something done quickly vs. architecting for the future, and dealing with the massive amount of re-factoring that’s entailed in iterative Scrum development.

Coding Quickly

I’m coding like I’m in Flash again. Instead of spending 3 weeks setting up Cairngorm or PureMVC with all your use cases, agreeing on the framework implementation details with coworkers, and getting enough of a foundation together that you can actually compile the application and start seeing screens, you instead make a mad dash to get app working in just a day or less. Rather than discussing with your team what the best ValueObject structure is and how your service layer should work, you instead get a login service working in under 40 minutes. If something changes massively, such as the data structure of the user object returned, you just modify or delete & rewrite the entire ValueObject. You didn’t spend a lot of time on it anyway, so it’s not like your “architecture masterpiece” is getting deleted; it’s just some scaffolding code to get you up and running.

Coding For the Future

…yet, it’s not scaffolding; it’s real code that needs to work, and work the entire project. Deciding how much to write well & encapsulated vs. just getting it done is extremely challenging, and fun. When do you git-r-done and when do you over-architect? How much and where? Hard questions to answer, fun times. Part foresight, part gambling, all calculated risk taking.

You know your service layer, the code that talks to the back-end probably WON’T change.  It’s extremely unlikely that in the middle of your project, you’ll switch from .NET and XML to PHP and AMF.  Therefore, spending more time architecting that portion can be done so with confidence in using the extra time it takes.

Anyone from a design agency should already find that very familiar.  You have a series of impossible deadlines, and arrogant programmers (like me) exclaiming you must utilize OOP, design patterns, and frameworks.  You’re challenged with meeting your deadline(s), trying to do right where you can, and learning throughout the process.

This is slightly different in Agile for product development (or even service development) in that once launched, your application doesn’t have a limited shelf life.  It’s an actual product.  Traditionally software is used 3 to 5 times its original intended lifespan, although, I’d argue with web software that is lessening.  Even before launch, you’ll be extending certain areas, and expecting it to perform solidly.  Deciding what to hack together for deadline’s sake, and what to invest well thought out architecture time in is really hard.  REALLY hard.  And fun!

UAT’s As Checkpoints

During sprint UAT’s (every other Friday for my team), or even just posting the latest working build for the team to see, you’ll inevitably question certain functionality and performance.  ”Why is that screen so slow to load?”, “Getting to this screen is more tedious than it should be…”, or “My RAM and CPU usage are through the roof!”.  The designer may see their designed creation in action, and totally change their mind on how it should look or work.  The stakeholders, after using it, may realize that it totally doesn’t solve their original goal(s) like they thought it would.  You may even notice a bunch of positive enhancements to make on already working sections.

This may sound frustrating, but it’s good for a bunch of reasons.  First off, this is the main reason Waterfall fails as a process.  None of these things can happen until the project is COMPLETE, in the Validation phase where you validate the software is on spec.  A lot of you may already have had those things happen during a project; now imagine none happening until the entire product is complete.  It’s a lot harder to change that much code that late in the game.  You now have the opportunity to fix bad decisions, improve design implementations, and add enhancements… early!  This is when they can have the most positive impact, reduce risk, and get battle tested more.

Secondly, when you go to fix something, you can code with more confidence since the functionality has at least been used.  Programmers second-guess themselves all the time.  They have to; early decisions made incorrectly can have disastrous consequences later (quoted from one of the Pragmatic Programmer authors in an interview).  It’s really frustrating to be insecure about how a user story actually works.  After getting it “working” in a reasonable timeframe, and using & discussing it, you can have confidence in what you code is more “correct”.  Well… almost.

Third, your design gets more real.  After banging on the implemented version of the design comps, your designer/UX person can make better decisions if their design is actually working, and the programmers can collaboratively discuss how to change/improve it.  This assumes your designer/UX person hasn’t moved onto another project by this point; keeping them on retainer for at least 4 hours a week is helpful for the project.

Fourth, you get confirmation certain problems are in fact real problems.  You may think something is slow, but if no one notices but you, does it really matter?  Naturally, your ego as a programmer is inclined to fix it anyway, but remember, your goal is get things done, not fix something that isn’t broken.  Same goes for problems you know of and other people see; it is just an iron-clad check mark that something is in fact a problem and needs to be addressed.  If you have performance problems for full-screen video on your Mac in Safari and Firefox, and so does your project manager in Windows in IE, Firefox, and Safari, then you can confidently infer that the majority of other people will too.  Granted, testing with more than 2 people is preferred, but the point here is that you get a helpful checkpoint with a 2nd set of eyes.  Coding this quickly without too much care to architecture, juggling a lot of moving pieces is a lot to handle.  Having a helpful team member confirm an issue early is better than finding it months later in QA, even if you knew about it and forgot.

Bottom line, using a UAT as an early checkpoint for completed user stories ensures they truly are complete and good and points out problems or potential enhancements early.

Refactoring

The above leads to refactoring; re-writing or modifying existing code.  A lot of times refactoring is a pipe dream. Usually you’re so focused on getting things done, having time to make something work better or faster, even the possibility, is the carrot that can keep you going.

Not in Agile. Based on the past 5 weeks and talking to Darrell (my project manager at Enablus), you re-factor on average 30% of your code per Sprint. You’re coding so fast and so furiously, that not everything is encapsulated as much as it could be (except for my service layer, it’s tight baby!). Not only that, but as you see the software in action, you can then start making valid changes. Maybe the functionality didn’t work as good as you originally thought it would, or perhaps you suddenly realize, now that you see it, that it needs something added. While this is easy from a user story perspective, just modifying an existing user story or adding a new one, it may not be so straightforward in code. A lot of times, there was no way you could foresee the change you are now tasked at making. If we COULD see those changes ahead of time, there’d be no need for the Agile process in the first place. This means that some of your code needs to be majorely re-worked, or even just thrown away and done from scratch.

While you’re technically working on a user story, you’re potentially breaking another. It’s not necessarily spaghetti code, but it’s certainly not Orgathoganal by the Pragmatic Programmer’s definition… unless you’ve architected that section out already, you’re a bad ass, or lucky. I’d argue the 30% is a loose average. In the first sprint, I didn’t re-factor anything, nor the 2nd week into Sprint #2. In the 2nd and 3rd sprint, I was re-factoring up to 40%. In Sprint #4, it’ll definitely be at least 40% again. The 40% arose from taking 3 tries to get a piece of functionality the designer wanted correct. The 40% next Sprint accounts for my bitmap caching engine suddenly needing to save not just 1, but types of ValueObjects, and all the existing View’s that now need to support both. Not to mention the fact we were working with the server-side team for the first time and still figuring things out.  The percentages are not indicative of the entire code-base, but rather, my time spent the entire sprint (2 weeks).  All this while working on new user stories…

For example, while you originally stated a user story would only be a “2 – mostly easy”, it ended up taking you a total of 5 days to complete because you were re-factoring and fixing other existing user stories that it related to. This can lead to the perception that your original point estimations are inaccurate when in reality, your point estimations are accurate, it’s just there is no adjustments made for re-factoring. This isn’t always necessarily taken into consideration when forming a point average for what your team can complete each sprint. Some sprints, you hit your “20 average” and another, you only hit 15, but you could have possibility re-factored 7 points worth of existing user stories, thus skewing the results.

Refactoring really confirms how much you wish you could predict the future.  As I’ve stated before, sometimes it’s easier to just start from scratch again on a certain component now that you know better how it’s supposed to work.  The original piece of code may have been really small and not well thought out in the first place for the sake of time.  That’s totally fine, as the mere fact you’re deleting it and starting from scratch atests to it being a good decision at the time.  Other times, however, you’ll notice you’ll have to do some major changes to a bunch of different classes, of which because not everything is encapsulated, you may suddenly feel like it’s spaghetti code; changing one thing breaks another, totally unrelated piece.

I will say with ActionScript 3, strong-typing and runtime exceptions have really helped me refactor A LOT faster than in the past.  I can “break with confidence”, even if I know my code is crap (it isn’t, I’m just going for dramatic effect here… *ahem*).  This has really helped remove the “fear” factor you can get with touching code.  It’s one thing to have your code build trust with you.  You really thought about its architecture, beat on it some, and it held up.  Cool, your code has built some trust with you.  In coding quickly in Scrum, however, how much trust do you really have when only parts are uber-solid?  Knowing that your code is going into a real world product people are paying for doesn’t lessen the pressure and stress.

Again, AS3 has really helped me here.  If there is a problem, I’m more likely to find it now, and find it quickly.  Additionally, KNOWING that fact allows me to, again, code with more confidence, try more ideas, and end up with better code.  Now, you might think you should start coding for every eventuality, at least from assuming errors such as checking for null and isNaN like crazy, but quite the opposite.  A lot of the runtime errors can point out problems pretty quickly, and the catch here is they point them out in both quickly written code AND well architected code.  The point here is that even well architected code will have problems you don’t forsee.  What I end up doing is using my best guess at the time, using foresight based on our past UAT and other project detail discussions, and moving on with life.  Stressing too much about one section is a waste of time; if it works, rad, move forward.  You may rewrite it again later anyway…

What Doesn’t Change and What Does

Experience has really taught me what to code quickly, what to architect well, and all the in betweens.  I haven’t got it all figured out yet, but I DO know of some sections that usually never change, and ones that change all the time.

The ones that never change are the service layer.  This is your Business Delegates in Cairngorm, or your Remote Proxies in PureMVC (or if you’re like me, your Business Delegates that your PureMVC proxies call).  If they DO change, it’s because the server-side developer changed the the name of the service, or the location.  Whoop-pu-dee-doo… 1 line of code in either the class or your ServiceLocator.  If you’re delegates/proxies use Factories to actually parse the server’s returned data (XML, JSON, AMF, etc.) then you’re even more insulated.  Again, middle tier technology doesn’t really change in the middle of a project.

A data model change usually affects your entire application.  For example, if you change the data structure of a Person object (PersonVO), suddenly your Factory changes, your VO’s change, any Controller classes modifying PersonVO’s (such as Commands in Cairngorm or Proxies in PureMVC, and potentially Commands as well), and any View’s that represent or edit them.

If you’re creating complicated View’s, whether based on a design comp with little detail, or it’s not a conventional GUI control, it will definitely change over time once someone uses it and gives feedback.  Any View based on a list of dynamic data that needs to draw a bunch of children that represent a ValueObject, such as a repeater or a custom Chart will go through extreme refactoring; both modifications of item renderers and drawing performance improvements if you don’t extend List and do your own drawing routines.

View’s such as your main Application file, an optional MainView, a Login, and Menu’s do not change assuming you use 1 CSS file and straight forward skinning.

Most Event and Utility classes just get added to; you don’t really change them, rather you add or remove class properties and/or methods, but their names and package structure stay the same.

For Cairngorm Commands, they just grow in scope as the development age of your application increases.  Since PureMVC Commands delegate a lot of this Model modification off to Proxies, those Proxies tend to grow in scope as the complexity of your data interactions increase.  They only get waxed or massively changed if your data model does.  This doesn’t really happen later in the project.

The above is totally a case by case basis, but has been consistent on a lot of my projects.  Your mileage may, and most likely will, vary.

The Con to Refactoring

There are a few cons to soo much refactoring.  The first is, some clients don’t understand why you’re coding the same thing twice… or more, especially when Scrum is supposed to be about getting it done quickly vs. over-thinking it.  In my experience, if you can speak intelligently at a high level, you can explain each refactoring part away.  I can’t, so usually explain it to a project manager who’s capable of translating it in lamens terms to a client.

The second thing is that it makes merging on Merge Day a TON harder.  You may have already refactored like twice the week before, and totally forgotten all the details of why you did.  Suddenly, 4 days later (every other Wednesday in our case), you’re having an insanely hard time merging code from your branch(es) into trunk.  This may require a long conversation with your team members, and you are struggling to remember why you made such massive code changes.  Even if you do remember, the other developer may feel a little frustrated if you didn’t invite them into the code refactoring change discussion for something you may at the time have felt was trivial.  It probably was trivial, it’s just blown out of proportion now since merging is always stressful.  Either that, or you just spend a few hours getting trunk working again.  If I totally wax something, I’ll usually put a large drawn out code comment to explain why.  Additionally, I’ll do the same thing in SVN check in comments.

The third thing is it’s a project manager’s nightmare.  If she doesn’t have enough forewarning of these and their possible affect on not getting a single or set of user stories done by the end of the sprint, it can be a bad surprise.  Communicating these during the daily standup meeting with potential ramifications is best.  It can also make planning future sprints challenging as well.  If your team has been chugging along at an average 12 points per sprint for 3 consecutive sprints, and suddenly in sprint #4 you spend 60% of your time refactoring, you’re clearly going to finish with a lot less points in user stories completed.  This sets the project manager up for failure.  They cannot effectively communicate projected progress to the client, nor visibility into the current progress of the app since something that worked for awhile may suddenly break in the next UAT.  You’re supposed to be completing user stories, not creating new ones that break old ones.  Again, forewarning is the only thing I know immediately to do.  I’m not sure what doing too much refactoring is a symptom of yet.  Most so far on my current project, and past ones, have been for random reasons.

Conclusions

I really like how fast I can code some things in Agile.  Other things have stayed the same, but the overriding goal of “get it working, but don’t write crap code” is such a high bar… and I love it.  It’s the same speed of agency coding, only you know you’ll have to live with the code (aka potentially eating your own mess) so you end up producing better code than you would in an agency setting.

I also like either drawing on experience, or just making challenging inferences, on what to architect well, and on other parts to just get something working without too much thought.  It’s nice to have the variety.

Finally, I’m not sure what to think of the refactoring.  I like that it’s “ok” and an expected part of the process, but I feel that my project is unique in the amount I’m personally doing.  My coworker for example isn’t doing as much as I’m doing at all; he’s chugging along on other user stories and is set to beat me, again, in point values for user stories completed at the end of this sprint.  We’re really pushing the limits of Flash Player here, and only one section in this large app is really this challenging; the rest are your run of the mill Flex screens.  So, it sounds to me like the “on average 30% of your time is spent refactoring per sprint” still applies.  There is no way I’ll be refactoring this much on some of the easier sections in future sprints.

Stay tuned for #3 in the Agile Chronicles series where I talk about every developer using their own Branch in Subversion.

1. Part 1 – Stressful
2. Part 2 – Code Refactoring
3. Part 3 – Branch Workflow
4. Part 4 – POC, Strategy, and Design Challenges
5. Part 5 – Acceptance Criteria & Punting
6. Part 6 – Tools, Extra Merge Day, and Postponed Transitions
7. Part 7 – Bugs, Unit Testing, and Throughput
8. Part 8 – Demo, Burnout, and Feature Juggling
9. Part 9 – Scope Creep
10. Part 10 – Conclusions

This entry is about what processes I came from, my definition of the Agile/Scrum process, and how stress has been spread out throughout the project vs. at the end.

Skip Intro

Preface

Five weeks ago, I started a project with Enablus, a firm that works with mainly startups to build products. I loathe service work, and love product work, so am really glad to be working with them. Even better, they are only 30 minutes from my house. I don’t really need to go more than 2 to 3 times a week, but I like how close they are when I do. One thing that struck me in the initial project discussions was that Enablus was really big on process and lingo. Darrell Ross, the project manager and partner I’m working with, although hard to read initially, still showed signs of uneasiness when I didn’t give him the answers he was wanting to hear in the initial discussions. For example:

“Do you utilize Continuous Integration?”

“Uh… what the hell is that?”

“You know, everyone on the team checking in code into a source control system every day, utilizing Test Driven Development practices, automating your builds, etc.”

“Er, I use Tortoise SVN and BeyondCompare all the time! Although, most people I work with don’t check in every day, don’t use TDD, and do not automate their builds.”

“I see…”

We’d constantly go through this. He’d ask a serious question like, “Do you utilize OOP and an software framework(s)?” and I’d respond with some cynical response questioning the merits of when OOP and frameworks are applied, or what I did and did not like about a particular framework, and how I don’t always officially follow their implementations… never really answering his question and thus not giving him confidence. I was constantly amazed at how structured Darrell made things sound, and he probably wondering how the hell a positive impression of “Jesse Warden” was formed in his head in the first place based on the answers I was giving him.

In all instances, we’d discuss the details of my responses, and he seemed to get the details he needed, feel better, and move on. Too ease his stress, I opened up and said that I’ve never really worked in places that were above a Level 3, and even those I felt never followed the processes he was mentioning, specifically Waterfall or Agile development, to the letter. Later in my consulting and contracting career, I had moved to using some concepts of Agile such as iterative development, and having builds of the software regularly on a schedule (usually weekly) even if the client didn’t need one for weeks or months. It had nothing but positive ramifications in the long run, so I stuck with what worked and saw other successful people doing similar things. It’s really a simple process (my version): Have a build working by Friday, every Friday.

I’m not sure Darrell believed me that everywhere I’ve worked, whether full-time or consulting/contracting, did not use a standardize process such as Agile. Maybe it did and I just wasn’t privy to the details?  Again, I never cared till this year.  Either way, nothing sounded as strict and defined as Darrell made it sound as he described the Agile process to me. Hence the point of this series of posts.

Introduction

I’ve never been interested software development processes. To me, it’s always been more important to be really good at the 3 main things that count: knowing your platform, knowing your tools, and knowing the language you code in. The better you know those 3 things, the more capable you’ll be at developing software, helping those on your team develop software, and thus ensure your team will be successful. As Dave Wolfe has said, it takes a village; a great team can ensure success. Whether the project actually succeeds isn’t really up to the software team at that point; hence the second rule to the “8 of 10 software projects fail” is “9 out of 10 are perceived as failures”. Even if you create an awesome piece of software, management, stakeholders, and/or clients can still perceive it as failing.

As I’ve found throughout my career, there have been more pressing issues than “process”, such as:

• getting the dang design comps
• making a workflow where fonts will compile on both machines (Mac and PC)
• coming to a conesnsus with team members on how the framework you are using should be implemented
• finding out what the heck we are building really is
• finding out how the heck what we are building really works
• resolving source control issues

There are tons of other things, but each has the capacity to reach fire drill status, and thus become a major time sink and productivity killer. As you know, losing time is the LAST thing you want to have happen in software development.

It also depends on the type of work. Working with Design Agencies, there was no process. Not because Agencies are dumb, but rather because the project timelines Agencies have are typically significantly shorter than traditional software or product development. On average, 2 weeks vs. 4 months. While a deadline can stretch to 4 months, you only work 10 man days on it, and of those 10, 5 are done initially, and the other 5 are done over the course of 2 or 3 months on client changes. In short, if you have 5 days to complete a complicated coding project, you’re more focused on getting things done than you are worrying about encapsulation, continuous integration, and frameworks. Writing code on day 1, and wondering who in the heck has time for UML and test cases. Agencies are deadline driven, whereas in software development, nothing is ever on time.

In consulting, you’re typically working within the constraints of your client’s company. It can go down two different ways. Either you get treated like a service shop, and requirements are thrown over the wall, and you complete how you see fit. Otherwise, you integrate with an existing team, and work within their processes, trying to improve them where you can. Contracting is usually the same way, although, you usually have little to no say in the process.

Even including regular work (W2 at a software shop), all instances can significantly have their chances of success improved through skill. If you’re skillful in the aforementioned 3 areas (platform, tools, language), it doesn’t matter how screwed up things are; you can always depend on skill to make or break a project.

…or so I thought. Going back to the 2nd rule of software development, “9 out of 10 software projects are perceived as failures”, I started to have more of the second rule happening vs. the first.  In those projects, more of my time would be spent doing non-software things, like pitching a tent in the project manager’s office, discussing details with stakeholders, drilling sales on what they were really trying to accomplish, etc. In the end, trying to define success to ensure my team and I weren’t setup to fail from the beginning. Being setup to fail, yet having a team with mad skill, results in you months later outside the office smiling on your umpteenth beer regaling all in the pimp architecture you created that never did, nor will, see the light of day. Creating awesomeness with no result is extremely more frustrating than flat out failing. The positive to failing is you can learn from it. Failing, even though technically you succeeded, is just straight ridiculous, and it was at those points I started to become more interested in the software development process. I realized I needed to find a team who knew all these processes I’d read about on the internet blogs.

Enablus is one of those teams. They clearly know Agile, as far as I know it 5 weeks in.

Agile As I Know It

I got Darrell to explain how Agile, specifically the Scrum part, works. This is my paraphrased, and re-interpreted version of our discussions, and my working with the process for a few weeks.  If you have a good idea, do market research to ensure it’ll sell, then you just need a good team to execute it. The execution on the software side for product development utilizes an Agile approach.

User Stories

The Agile approach, for the engineer, starts by defining “User Stories”. User stories are a set of sentences that describe what the user experiences while using the software. These can be derived from the wireframes; design comps aren’t a requirement. A user story can be expressed like so:

Login: User inputs Username and Password and clicks “Start Application”. Login is validated. For all other login instances, the Dashboard is displayed.

Notice there is no implied technology or design construction details. The user story merely describes what the user see’s, what interactions the do, and what the results of those interactions are. That same user story could apply to Flash, Silverlight, AJAX, PHP, etc. It’s technology agnostic; the point here is to document a user experience to ensure we’ve captured it well enough to be understood, and thus executed upon. The same goes for design. It doesn’t say “2 text input fields for username and password, with bold headings to the left, aligned center”. It’s left up to the interpretation of the designer to execute that user story, and hopefully provide direction for the GUI programmer through design comps assuming the user stories were derived from the wireframes, and not the design comps.

Notice also, it’s not a “feature”. It doesn’t say, “The user can login”. That leaves things massively open to interpretation, and solves no business value even if the feature is “completed”. As we all know, not all logins are the same. Documenting features, and executing them is one of the number one ways to be setup to fail, and is also one the main reasons Waterfall doesn’t work. If a software spec says, “The user must be able to log into the system”, but doesn’t describe registration, single sign-on, error scenarios, and how it’s supposed to look, you’ll end up with software that the client doesn’t like.

Some user stories are extremely high level. These are sometimes called meta-stories because they actually encompass a wide array of smaller user stories. Describing a login scenario can have a multitude of smaller user stories within it such as “user is presented with an error dialogue stating that they must first activate their account via the confirmation email before gaining access to the entire application’s feature set”. The level of detail only matters in so much as to make sense to the designer and engineer executing it. If a user story encompasses weeks of coding, it’s probably a meta-story and should be broken down into more manageable user stories that are easily defined, and more easily tackled in a reasonable time frame. What’s reasonable and doable is different for each team.

You create user stories for your entire application. What does, rather what SHOULD, the user be able to see, do, and experience?

Backlog

These user stories go into a “Backlog”. This backlog is a list of all the user stories, say 50… or a hundred. How many user stories you have are irrelevant. What’s important is that they are all documented in one place called the Backlog. You need to ensure all of the user stories are valid with the stake holders. It also helps to go over them with your entire team to ensure they make sense to everyone involved.

As you move through your project, you may find you’ll need more user stories to account for things you didn’t think about, or for additions/modifications to existing user stories. You then just add them to the back log. It’s normal for a backlog to grow during a project life cycle. However, you do not have to complete the entire backlog to complete the project. A lot of times, the stakeholders will make a business decision that 40 of the 60 user stories is enough to have a great product, and say “ship it”. That’s one of the key benefits to Agile is that you have working software throughout the project, and at anytime can make the determination of when it’s “done”… or is just a version 1.

Point System

Once your done with your backlog, you then have your team go through and assign points to each user story. To do that, you need a Point System. A point system is defining a metric for “how challenging something is”. In simple terms, a 1 is easy and a 5 is hard. It doesn’t have to be 1 through 5; it could be 1 through 10 or whatever. It’s best if they are numeric values because you use the points to measure a variety of statistics about your project. Additionally, each team may have their own definitions of what is challenging. While it’s best to use the same 1-whatever metric for the client and server team, the server team will most likely think creating a new server-side method a “1-easy” whereas a client developer a “5-hard”.

Once you’ve defined your point system and gained consensus amongst the team, you then go and assign a point value to each user story. This in essense gives each user story a “value”; how many points a user story worth. If you have multiple team members working on different parts of the system, say server-side and client side, you then have a point value added to each, such as “2 – mostly easy” for the server-side portion and “3 – doable” for the client side portion. The combined value gives the value of the user story, in this case 5.

From these point values, the project manager can then go back to the client, and ask, “With 20 points, what features do you wish to have completed in the first Sprint?”

The client can “purchase” user stories using those 20 points. A Sprint, which I elaborate on below, is an uninteruppted time period in which your team works on completing their assigned user stories. You make a guess in Sprint #1 on how many points your team can complete per Sprint. As time goes on, you start to see a pattern emerge of how many points per Sprint is the team completing. The client and managers can use this metric for planning and resource purposes. You can forecast, somewhat, how many Sprints you have to do to complete 60 points worth of features, and when you’ll have them completed. The chosen user stories that the client purchased are what your team is working on in Sprint #1.

Assigning points is a very important step for the engineering team. You typically don’t change point values mid-project, only assinging points to modified user stories or new user stories.

Sprints

As I mentioned, Sprints are a period of time that your team works on their assigned user stories. A sprint is however long your team decides it should be to get a working build with the user stories assigned. This could be one week, two, or a month. My current team is using 2 week cycles. It’s important to note the “uniteruppted” part. The client and project manager won’t interject user stories mid-sprint, nor modify your work load. Basically, they don’t bother you. This allows your team to focus on the task at hand, prevent fire drills, and stay productive.

In the examples provided, your team has 2 weeks to complete 20 points worth of user stories. Now, a sprint isn’t just, say 10 days of straight working. You usually have 3 special days. The first day is the kick off. You determine the user stories you and your team members will tackle. A couple days before the last is your “merge day”. It’s when everyone merges the code together from their different branches (or you just make sure trunk works if not using branches, more on that later) to get an idea of where the build is at, and where your team is at with their user stories. This is a final checkpoint before…. the UAT. UAT stands for user acceptance testing and in this case, it’s going over the final working build. Your team collectively goes over each user story, and identify if the build satisfies the user story or not.

As I said, our sprints are 2 weeks in length. The first Monday is spent confirming what user stories are in the current sprint and who’s working on them. We try to work in our own personal branches so the 2nd Wednesday is “Merge Day”, where I take any branches I and my teammate have, merge into trunk, and work out the kinks. We make a mad dash the rest of Wednesday and all of Thursday to finish our remaining user stories, or fix issues with existing ones, or just the build in general. Again, over the course of a sprint, you can start to get a group average of how many points your team is capable of doing in a sprint.

We’ve been using a shared Google Spreadsheet with each Sprint’s user stories in its own tab.  We collaboratively use this all the time for reference.

Daily Standup

We have a call everyday at 1 pm EST / 10 am PST. Our server-side time + client is out west in Cali, and our Flex team + UX + designer + project manager + rest of the team for other areas is east in Atlanta. We each go around talking about what we completed yesterday, this morning, and what we plan on going the rest of the day. We also cite any roadblocks or issues, and if that involves another team member, we just schedule a separate call, or time to talk over IM, with that specific person. This results in our calls usually lasting 5 minutes each day. I’ve heard of other projects at Enablus having their daily standup lasting longer than 5 minutes, but our team is pretty focused; we know what we need to do.

The majority of my road blocks involve a web service not working like expected suddenly, or questions for the designer on how a part of the design comp is supposed to work. Most of these I just solve on my own by calling the team member outside of the meeting time.

Pro’s & Con’s

There you have it, what Agile is based on what I’ve been told and experienced so far. So how’s it working out so far? Here are the pro’s and con’s for this first entry in the series.  I’ll elaborate in future articles, but here’s the quick rundown.

Pro’s

• I’ve never been this stressed in the first two weeks of a project before.
• coding with a “git-r-done” mentality; fast and furious.  Not everything has to be uber-architected.
• each developer has their own branches; this makes checkins & merges a lot easier and less stressful
• It’s nice to have something real after just 2 weeks.  Gives a great feeling of validation very quickly

Cons

• a lot of re-factoring
• creating new user stories later can break older ones
• merge day is stressful
• “need to see it working”; the desire to see working functionality before important functionality decisions can be made isn’t solved

Conclusions

In conclusion, I really like Agile methodology that we’re using so far as it allows me to code some things extremely fast and not have to worry about uber-architecting it. I just have to get it working which I’m good at doing quickly. For the things that matter, I can spend an allocated time which I feel good about doing because I can more easily see over time which areas deserve architecture and which ones do not.  The merge day is extremely stressful, but overall I like having an allocated day for it and using separate branches which prevent daily SVN drama.

I’ve never been this stressed in the first two weeks of a project before. The typical scenario is you get excited, and can’t wait to dive in. It’s a happy feeling filled with anticipation. Then, as the deadline approaches, you steadily get stressed out and things become less fun, albeit extremely frustrating. This isn’t always the case, but I’ve continually expected fun in the beginning, stress at the end.

It’s been really nice to be stressed at the beginning; this has implied to me the project won’t be so traditionally bipolar. The same thing happened to my mood once I capped myself at 2 cups of coffee a day. Think of it, you have 2 weeks to create working software; ready, GO! That’s insane, and really cool at the same time. What could you do in 2 weeks? It’s really nice to know that even after just 2 weeks of work on a 4 month (ish) long project, even the first 2 weeks will produce something valid and working.

Stay tuned for #2 in the Agile Chronicles series where I elaborate on the re-factoring challenges.