Consulting Chronicles #2: Fixing a Pile of Rubbish – Part 2

In Part 1, I covered how you earn trust from the client, and set the stage for how you start to fix a large pile of rubbish; a seriously broken piece of Flex software.   While there are 5 aspects to fixing it using the 5 P’s: Programming, Prowess, Politics, Positivity, and Patience; the Programming one is large, so here is the second part.

Introduction

You’ve gotten the build system under control.  You can compile and debug.  You have a good logger in place, and have started to understand the flow of the application.  You have a few simple bug fixes/improvements checked in and documented.  The client has seen you can positively contribute.  You’ve taken a deep breath and are armed with coffee/tea/water/morning beverage of choice.  What’s next?

There are 3 main things you need to accomplish within 2 to 3 weeks:

  • Assess what the major problems with the code base are AFTER you’ve gotten a feel for it.
  • Learn the data model.
  • Understand the client’s goals.

Deep Dive

You need to dig deep into the code base, touching all areas, and get a general understanding of how things work.  The deeper the better.  You don’t want to spend too much time in one area attempting to fix things.  Even if it’s horrible to work within, you need to tough it out for a few days.  The goal is to have a broad understanding of the entire code base.  Once you’ve checked in some working code, you’ll see what happens when you change it, how it performs, how the deployment process works, and how the client reacts to changes, both pro & con.

Learn the Data Model

The second thing to do is learn the data model.  This is often the most telling of how a code base is put together, and helps define the code’s perception of what the data model is.  The Value Object’s on the front-end rarely exactly match the back-end database schema, and a lot of Application & Business logic is often put into them, and their associates who work with them.  These things are very telling of how a code base is put together, how it intends to work with the data, and how the original developer(s) perceived the data model in ActionScript.

More importantly, it gives you 100% validation of what the intent of the software is.  Yes, the wireframes, design comps, and requirements you may get your hands should technically give you that, but the data model usually defines the business.  Large clients tend to spend a lot of time on their database models, and once these get aged, they don’t often change without a lot of drama.  Flex developers, being somewhat agile, will tend to put their own helpful changes on the client side VO’s not just for coding purposes, but for “meaning” purposes.  The VO’s will often have special meanings, things specific to the business the client is working in.  This makes the VO’s easier to work with, and easier to understand.  English is often easier to read than a database table name.

Also, you can see how this interpretation with meaning is consumed.  Changing a data model affects every part of your application, and as such, you can see the soul of an application on how it handles the data.  This insight into the plumbing really gives you a good understanding of how the developer(s) implemented their version of MVC (everyone has their own version).

You learn the data model, you learn the client’s business.  You also learn the previous developer(s) perception of that business.

Understand the Client’s Goals

One of the first questions you should ask your client is “What will make you happy when I leave?”.  This’ll give, usually, a clear milestone of what you should be doing during your engagement.  Communication is a flawed process.  The client may think they know what they want.  They may also be telling you what they think you want to hear.  The client may have no clue what’s going on with the code base; they could think things are great, and thus have other priorities revolving around future projects.

Whatever the case, your research into the code base’s overall status, as well as the data model, will allow you to be a more effective communicator.  Once you can give a clear assessment to the client on the status of the code base as well as being able to speak their lingo via knowing the data model, you can start to have a more important, and better communicated, conversation.  You need to revisit their goals, their goals with you, and assure all are actually attainable given the current status of things.  Other times, they are actually waiting on your assessment to proceed, hence the work you do being EXTREMELY important.  You’re influencing product/service strategy direction, so that’s why you don’t want to skimp; hence why you should take at least 2 weeks.

Yes, I’ve had clients ask me to take days, one large only giving me 3.  Naturally I warned them my assessment wouldn’t be thorough, professional, and could suffer in accuracy.  They still trusted my gut instincts and recommendations, though.  A lot of times these engagements lasted months anyway, so I had no qualms about changing my assessment and updating it as time went on.  Regardless of engagement length, you should do this too.  Keep your client in the loop; the earlier they know about the horrible surprises, the better.

Once you have a clear understanding of the code, and have effectively communicated this to the client, they’ll have more accurate goals in mind, some of which may change your direction.  You, too, can have goals in mind and present them to the client.  Sometimes they are selfish, in making your life easier to work with the code.  Other times they are in parity with what the client needs.  You need to be careful about choosing goals that aren’t in line with what the client wants.  They are paying you to do a job; and working on the most important thing can a lot of times work in your favor.  I don’t mean just in earning trust, but also earning time.  Once your boss has enough wins under his/her belt, they can communicate to their boss easier to negotiate more time in case you need to do some major refactoring for example.  On the other hand, you are the expert.  If you need to accomplish some goals with the code base to actually satisfy the client’s overall goals, so be it.  Just ensure you communicate that.

Communicate the reality to the client, so they can communicate their true goals to you.  You can then work on the most important things.

Programming Tip #7: Revisit Your “To Fix” List

Programming vs. Programmers

Whether you takes notes in Notepad/TextEdit, or you fill with code with TODO’s and FIXME’s, you need to get a preliminary list of things wrong with the code.  You can’t fix problems unless you’ve identified them.  Some problems in code aren’t immediately apparent, or indicate larger problems over all.  Some problems require further investigation to figure out the true cause.  Also, a laundry list makes you look valuable to your client.

Your challenge is to decide between what you deem is most important and what your client deems as the main goal (for the week/month/sprint).  A lot of times, if you’re client gets what you are doing, you can usually guide the goals as long as you have tangible results.  The client can then communicate/show those to their boss/stakeholders.

For example, on one project, I had spent about a week getting the lay of the land in a Flash code base for an Enterprise sized website.  Not many Flash sites reach that size, but the few that do require the same level of architecture that any Enterprise sized project does.  The client wanted to significantly update the design of the site.  After a week in the code, I realized it was un-usable.  While the site actually had the first good MVC architecture with support for multiple SWF’s I’ve ever seen, apparently the same developer did NOT write the View code.  The FLA’s, while running atop a mostly solid AS2 architecture, were using Flash 5 style coding techniques.  Code on objects, violating DRY everywhere, and basically using code as cement for the current design.

The client previously had someone else from my firm do an assessment and state the code was good enough.  2 months later, with little to no change in the code they have me come on-site, and state they needed a complete re-write of the front end, that didn’t go over too well.  It didn’t matter; I knew I was right, and had to find a way to earn their trust, and then start incrementally fixing it.  I needed help, and I realized the first way to do that was the on-site employee I was working with.

So, I spent week #2 and #3 making friends with him, teaching him Flash, and showing him how NOT to code Flash.  He was a PHD, whip smart, and previously doing server-side .NET.  Once he saw, and agreed that there was a wrong way to do things, and a right way, we started to work together to incrementally fix things.  Since we had 2 weeks before the project officially began, we had some time to do some recon and R&D.  He also taught me a lot about the existing code base, and the business as a whole the entire 3 months I was there.  Once he did a couple visual components in Flash and 1 screen my way, he started to agree it was the right way to do things.  He also started realizing the massive amount of work before us.  Together, we worked together to formulate a plan, and keep management in the loop.

As you can see, my first order of business wasn’t to start re-architecting the site.  It wasn’t to start writing code.  It was instead to make a new friend, empower him, and garner a staunch ally in my battles ahead.  While it’s great to ensure you have firm plan of attack in your programming tasks, it’s another to have people helping you.  Especially if they are employees and not fellow consultants.  Remember, as a consultant and an outside party, you are immediately suspect, sometimes hated and feared.  Before you wage a war on code, you sometimes need allies to help you.

The point here is, while you may have a long list of programming tasks you need to do, sometimes the more important thing to do is get more programmers.  Another point to is that those working at the company can give you better insight into how the company works, what the true goals behind the project are, and how you can best operate in the current environment.

Yet another responsibility as a consultant is to advise the client the true nature of the situation.  While it’s great you can be brought in on short notice and fix the problem, clients will eventually resent you if all you do is fix their problems without helping them succeed.  Empowering the client, in this case the client’s own employee, to be a major/main contributor to the project is a wonderful win.  Additionally, it ensures when you leave, the client doesn’t need you, nor feel like you abandoned them.

Short Term vs. Long Term

Some tasks you decide to do may have longer term gains vs. shorter term ones.  It’s important to discern what the current situation dictates you do.  A fix list is a lot like an Agile/Scrum Backlog.  You don’t have to make a list, prioritize it, and have it be set in stone.  You can change priorities as the situation dictates.  Your client’s goals may change as time goes on, and you need to be able to adjust your current goals to match.  You still need to write it down so it doesn’t get lost.  If you’re spending your time improving the GUI to be faster and more efficient, and suddenly the testers start finding a significant amount of problems with the data they are seeing, you should then adjust your focus.  This doesn’t necessarily mean that you personally have to drop everything you are doing.  Rather, if you have team members, you can empower one of them to take ownership of it so you can focus on larger issues.

Again, the constant challenge you’ll be dealing with as the project progresses is if you’re producing fixes that have visible, tangible, and positive results in the eyes of the client.  While you’re doing that, you need to be focusing on the larger architecture problems.  It’s a challenging balance, but is why many consultants enjoy what they do; it’s really hard, and thus a challenge.  When in doubt, go for the shorter term goal.  A large ship may turn slow… but it IS turning.  Eventually you’ll get things on the right track.  That, and you’ll ensure you’re keeping your client happy as best you can.

Yes, there comes a point where you have break a ton of things to fix larger issues.  One of the ways to do this is by using a branch in SVN.  You can fix it there without affecting the rest of the team, and once fixed, merged.  Make sure you merge with plenty of time to back out of the merge.  Also, you have to make sure that the rest of your team is aware of what you are going to do, and what could go wrong.  It helps to make an official merge day.  That way, if you need more time finalize things, you have all day.  Additionally, you can then get your entire team’s help to iron out the kinks.  Finally, if things go horribly wrong, you can just revert, and merge all the new changes the team has done into the newer, reverted trunk.  The challenge here is to make sure the branch you, or a team member is working on, doesn’t get too far out of sync with trunk.  If it does, merging will be extremely painful, and finding all the broken things and fixing them to ensure the rest of the team stays productive will be really hard.

My gut check has usually been if it takes longer than 4 days for me to check in working, improved code, it’s taking too long.  It’s better to not check it in.  Instead document it and put it on ice in a branch instead.  That way, if you have time, you can come back to it, or at the very least borrow code from it in the future.  I know it’s hard as an architect, but shorter term, quick fixes are better than long term ones.  Yes, it may actually take you longer to do shorter term fixes because of longer term problems, but this is where your persistence pays off.  And again, you’re still “getting things done” in the eyes of the client.  You can only go up from here.

Make that list, and keep it flexible; you’ll add things and change the order as time progresses and the client’s mind changes.  Delegate to your team where you can to allow you to focus on the larger architecture issues.

Programming Tip #8: Goals, Performance, Visuals, and Explosions

When in doubt about how to prioritize your massive list, here’s the priority order you should use:

  1. Client Goals
  2. Performance
  3. Visuals
  4. Explosions
  5. Diagnostics
  6. Documentation
  7. What’s Next?

Client Goals

The first thing you should always do is accomplish/fix whatever goal perceives as priority.  If it’s a meta-goal (aka large milestone), make sure your tasks continually work towards that.

Performance (not Memory Leaks)

If the client is mostly happy, move onto performance.  The number #1 thing I see in Flex apps is performance problems.  This could be memory related, but it’s often related to doing too much at one time.  This is because most people don’t know about invalidation, deferred instantiation, modules, how to work with externalized assets, when to embed instead, how to use green threads, how to use memory and disk data caching, DisplayList tactics, proper memory management… or just how Flash Player works.  Therefore, if you start to fix performance problems, and get enough in that the app starts to “feel” better, you can deploy two builds to show the client after 3 weeks: the old one and yours.  The differences will be night and day.  You can be a bad ass and state, “We’re not done yet…”.  This is the most tangible way a programmer can make a difference.  Yes, there will come a point where you are splitting hairs for inconsequential gains.  In that case, move on… unless your team is idle, in which case, TUNE THAT SHIZZZ!~!!!

Visuals / Look & Feel / Skins / Styles

Visuals are usually the 2nd most important thing you can do.  However, programmers typically aren’t good at it, so I put it behind performance, and thus make it 3rd, since coding is typically easier for programmers.  However, visual improvements have a larger impact in RIA’s simply because the whole point of choosing Flash Player in the first place is for its rich visual capabilities.  That and the Bob’s, those project stakeholders in large companies who have no clue what you do and sometimes don’t even speak English (they may use English words, but you have no clue what they are saying), clearly can see a positive visual change.  Additionally, with very little work, you can imply that you did a lot.  It’s amazing how making 1 screen look better makes it seem like you busted your tail for 3 days, yet spending 2 weeks to solve a memory leak barely gets you a thank you.  Most Flex projects you consult on are either broken programmatically, or visually.  You should focus on the things you can fix performance wise first, and visual second, unless the whole reason you were brought on the project is because the existing team “doesn’t know a thing about skinning”.

Often, the perception of skinning, whether Flex 3 or 4, is that skinning is something you can just tack on top of the existing code base.  As you know, this is simply not true.  Sometimes great architecture changes are needed just to accomodate a design.  Thus, sometimes visuals can take a significant part of your time, yet still give you a good bang for you buck; justification for the immense time it takes to re-factor Views to accommodate the design.

A lot of times, especially in Flex application development, you’ll find that the designer had little to no idea of Flex capabilities, or had no clue what they were designing was going to built in Flex… or perhaps they thought Flash instead.  Other times, the design is just bad to begin with, yet everyone on the team thinks the 9 developers are just having a hard time implementing it.  In these situations, you can either attempt to get the design back on track, make minor modifications, or tough it out.  If you’re firm has UX and designer resources, that’s one angle.  Other times the client may have good internal resources who simply have been setup to fail.  A lot of times it’s an outside vendor/design agency who has no experience actually delivering what they design, yet have bad ass sales staff, hence getting a gig they can’t technically pull off.

Using your guidance, you can bring them back on track.  Other times, upper management see’s no problem with the design, “it’s signed off on”, and you’re stuck with it.  This is often the most morale killing since you know no one in the right mind will use what you are building, and yet you’re struggling day and night to build something that won’t see the light of day.  That, or it just has a ton of usability problems, and these usability problems you’re merely solidifying as you attempt to create un-conventional GUI elements.  This is the point where you need to find inner strength, whether religious, or some other spiritual purpose to make it through.  I’m serious, not being sarcastic for once.  You can dig internally, externally for pay… or just walk away.  This is a viable option too; more on that later.

One thing that can help is to change your perception.  Instead of building something useless, you’re instead challenged with building something hard.  If you accomplish, you’ll have learned a lot, and finished something challenging.  Your talents will be recognized, and when it comes to a head that the software design won’t work, you’ll be the first they ask to do it right… unless you’re merely the development arm, in which case they’ll hire another agency.  Hopefully by then if you’ve pointed out enough design problems, they’ll invite you in on the next design review meeting.  Better than nothing.  Better, obviously, is someone from your own firm, or hiring you someone you know through you as a 1099/W9 (whatever your CPA/CFO says).

Conclusion: Make it look good, people from every background will immediately notice.  A lot of gain for little work compared to performance.  Prioritize visual fixes ABOVE performance if you’re good at it.

Explosions / Runtime Exceptions / Crashes

Things that are nice to hear from your client:

  • “She made the app stop exploding.”
  • “I no longer get error popups”
  • “The app is MUCH more stable, thank you!”

When a project is sinking, like a ship, multiple explosions are usually a clear sign of the cause.  If you remove the explosions, people think the ship isn’t sinking.  While performance can be felt, and visuals can be immediately seen… explosions, while sometimes predictable, still produce insecurity.  If you remove the explosions, you create security in your ability to kick ass.

Spending time making the app more stable is an ongoing process, and as such comes behind performance and visuals.  If you hit a wall with the above 2, adding stability always helps.  This doesn’t mean wrap everything with a try/catch; it just means find the holes, and plug ’em (find an error, catch it, and log it).  This doesn’t include memory issues.

Diagnostics

Projects as large as Enterprise ones require diagnostic tools.  To keep something that large running, you need insight into their innards.  The standard Flex tools such as the profiler, network manager, and even Firebug/Charles are not enough. This is as simple as knowing how much RAM is being used, to as fine grained as how many service calls to a 3rd party API are still out-standing, and which ones haven’t fired because of your 2 to 4 per domain restriction.  A lot of tools are built to monitor specific things with the application in question.

Diagnostic tools allow 3 important things.

  1. Give you a swath of information on what is going on inside the app (logging, memory, etc.).
  2. Allow you to proactively know what’s wrong with the system (errors, warnings, etc.)
  3. Empower back-end, testers, and management a clear picture of what is wrong.

They are built to give insight into a particular business/application portion of your app.  They aren’t generic like loggers, but more of a specific gauge/hud/dashboard into specific areas of your app.

One example was a simple logger to ensure all Ominture tracking calls were going out on our website.  There was some disparity between what we on the Flash client were sending, and what the back-end was receiving.  After checking the code, I blamed the JavaScript and/or back-end, and our simple tool proved we were right.  Even a simple, drag-able window that prints out text can save you weeks of debugging and back and forth.  Additionally, it helps the other developers, those doing the JavaScript and the back-end.  They don’t have easy insight into what’s going on in your SWF.  Anything you can do to help them helps the project.  Often, the easier you make it for them, the easier it becomes for yourself and your team.  Remember, time they are spent doing their job on their own without your time is time you can spend working on what you need to be working on.

Another example was building a simple DataGrid window in Flex to determine what images were cached on the client on local SharedObjects, their size, and ID.  This helped ensure what I was seeing on screen really was cached vs. live data, and gave me insight into how often I was caching, and how much.

When building games, AIR apps, or anything that needs to run for a long time, Stats has helped me a lot.  While the Flex/Flash Builder profiler & the classes in the flash.sampler will give you finer grained information, Stats quickly tells you changes in memory.  Simple, quick, small, un-obstrusive… diagnostic tools don’t need to be complex to give you great insight into your application.  Collectively, however, they make you all powerful and all knowing into the workings of the code.

Another common trick developers use is to put the Flash Player version in the right click menu; whatever the value of Capabilities.version is.  Another good one I’ve found to save trips across the country (I’m not kidding, it seriously prevented another flight) is adding Security.sandboxType.

Unit Tests

Even if you or your client don’t follow TDD practices, wrapping unit tests around code that doesn’t behave consistently is a great diagnostic tool in it’s own right.  You’ll start to notice that once you earn respect from the client, they’ll start asking you what’s wrong during a crisis/fire-drill.  If the app doesn’t work, you’ll get IM/Emails instead of other employee’s who have been there for years.  Having unit tests that quickly ascertain where the problem is helps you both identify what the true problem is with confidence (which says a lot to the client), and secondly allows you to direct the problem to whose it is; even if that means it’s you.  Most importantly?  It saves your time.  Less time putting out fires, less distractions, more time coding.

The lowest hanging fruit here is wrapping your service layer; that code that gets data from the back-end since that’s often the most badly coded in my experience.  Secondary is testing the Factory layer (or whatever code parses the data into VO’s).  This confirms whether or not you’re getting good data from the back-end, and thus determines if your system is getting polluted and getting confused.  Third, testing application domain logic is ok, although, usually harder since sometimes Models/Proxies/Mediators tend to be in a specific state when things go wrong.

Documentation

I have pretty strong feelings on documentation since the projects I’ve been on that are screwed up, documentation for/in the code has been the last thing that’s been helpful to me.  That said, code comments that describe hacky sections, specifically the why, ARE helpful.  The “how’s” and code organization comments have been worthless to me to stop ships from sinking.

Documents that describe the architectural approach or the object model are extremely helpful.  While understanding the code’s “story” during the course of your deep dive is helpful, having an already prepared document that goes over this does wonders to get up to speed quickly.  If you have time to document these 3 things, that’s a great use of time.

What’s Next?

Sometimes your client is so stuck in the now because of the immense task list and deadlines they have, they aren’t really thinking about what’s over the horizon.  If you enjoy working with the client, it’s partly good business to question what’s on the horizon to see if you can extend your contract.  Mostly though, it’s to ensure they don’t go through the same pain they just went through.  Just because you’re primary job is a consultant doesn’t mean you can’t be a software developer & architect in the near future.  This is also a way to get them on track as a consultant; doing it right the first time (which is the 2nd time technically, but that’s ok).

Conclusions

Hopefully that gives you enough criteria and general guidelines to move forward on any disaster you encounter from a programming perspective.  Remember, you don’t have to go it alone.  If you need more help, ask for it.  Whether from your own firm, hiring yourself, or hiring directly through your client.  Yes, you can do the interviews; most clients will defer to what you need anyway.  Doing the recruiting is often best done by you if you are networked.  Using email, twitter, Facebook, IM, local industry meetings, etc… whatever works.  Why?  When’s the last time you read a job description that was accurate towards the job?  Thought so.  10 years Flex Experience with 5 years of Photoshop and MySQL, indeed…

Also remember it’s easier in the long run to fix things in short term.  You may feel your time is better spent working on the larger issues at play to prep the application for launch.  Perhaps your right.  However, when push comes to shove, you’re better off checking in little wins every week than 1 big architectural change every 3.  If you can delegate this out to your fellow developers to keep the client happy so you can focus on larger issues, great.  Divide and conquer.

Always keep abreast of what your client’s current goals are.  They directly correlate to what you should be spending your current time coding.  From there, if you’re left in doubt, chip away at performance, visual, explosions, diagnostic tools including unit tests, documentation, and finally preparing for whatever is coming down the pipe.

My next article will cover Politics which directly affects your ability to even program.  If you don’t play the politics game, you won’t be able to even do what your job is; code your way out of disaster.

One Reply to “Consulting Chronicles #2: Fixing a Pile of Rubbish – Part 2”

  1. Again an excellent article Jesse. I really like reading your advice, will help me alot on my future projects.

Comments are closed.