Agile Chronicles #11: The Backlog and Product Development Challenges

I wanted to talk about the Backlog again with regards to Scrum, an iterative Agile Software Development process. I’m working on 2 products simultaneously in my spare time, and have noticed a few patterns the Product Backlog has helped me with, as well as pointing out priority problems. I thought it important to bring this up again on what a Backlog is, how it solves the scope creep problem, and how you can use it incorrectly.

Caveats

I continually abuse 2 terms below, Backlog and Sprint.  Sprints don’t have to be 2 weeks, but for the sake of simplicity, I assume they are.  Also, Backlogs contain user stories, not requirements or features.  However, I assume for the sake of this article your features are already formed into user stories.  Finally, I assume that once I capture a requirement, it’s thrown into the Backlog.  I actually have a document which holds the validated requirements, which I then later formalize into user stories, and then put those officially into the real Backlog.  In this article’s case, that process is assumed to have already happened.

Review: What is a Backlog?

First, a review. A Backlog is a place you store features, or user stories. There are often 2 Backlogs for a software project, a Product Backlog and a Sprint Backlog. The Product Backlog contains all features for your software. At it’s purest form, it’s an spreadsheet that lists all the things your software is supposed to do, with a rated business value. While it can be seen somewhat as a requirements document, it is more defined, and singularly focused. It merely lists features/user stories.

While a requirements document may say “The application needs to allow users to register and login”, the Product Backlog would probably have that as 4 line items:

• Upon entering [product URL], and the user isn’t logged in, they’ll be presented with a login screen. They can enter their username and password, click the “Login” button. If the correct credentials are provided, the user is shown the landing page.
• If the user inputs the incorrect username and/or password, the Login screen is shown with the error message “Your username and/or password was incorrect”.
• If the user is already logged in when entering [product URL], they are automatically taken to the landing page.
• If the user does not have an account, and arrives at the login page, they can click the “Register” hyperlink to be taken to the New User Registration page.

There are a lot more user stories you can write, or even refining the ones above. There are a few points here in the difference. The client wants users to register and login. This requirement can easily be documented in a requirements document. That requirement belays a lot of work for developers, and a lot of things for the PM to track. Requirements are for capturing the client’s needs, but are not setup for clearly laying out what a developer should do, nor what the acceptance criteria is. The developers want clear instructions on what they are supposed to code with a clear acceptance criteria. Most importantly, the Project Manager needs to have trackable items, clearly documented, that are accepted by the client. Additionally, she needs to have weights and priorities attached to these items so she can track over time and give the client transparency into the project.

These units of work, called user stories, are stored in the Backlog. They are mutually agreed upon by the client, developer(s), and the PM.

Review: Sprint Backlog

A Sprint Backlog is where you put features/user stories that team needs to complete each Sprint. You put user stories into the Sprint Backlog from the Product Backlog. Once the Sprint starts, the Sprint Backlog becomes frozen. While you can add & modify things in the Product Backlog while the Sprint goes on, you cannot add/modify things in the Sprint Backlog. Additionally, the Developers are the only ones who actually mess with the Sprint Backlog. They’ll assign user stories to themselves, complete it, and move to the next.

What Are We Building?

One of the reasons for the high rate of software project failures is due to the software not being what the client wanted. Software is usually built to a specification or “spec”; a document that outlines what the customer/client wants. Clients are busy. Thus, back in the Waterfall days, you’d get a sales and/or PM to document as much of the client’s requirements as possible, formalize it into a contract, usually a Statement of Work or similar, and have both parties sign it. This contract was supposed to be what you were building, by when.

There were 3 huge problems with that. First, it assumes you accurately captured requirements. This is a fallacy. Communication is a flawed process based on the filters we humans use to understand other humans (my mind filters the thought into speech I believe you need to here, you hear this speech and interpret the meaning of my words, and filter it to what you think I said). Thus, you need to, rightfully, assume any documentation, even mutually agreed upon, is flawed.

Second, developing software is slow compared to making dinner. The business and/or customers who needed the software can change.  Software needs to embrace change to be relevant.  Iterative development embraces that concept whereas Waterfall does not.  Waterfall assumes you got it right “months ago”, and there is no room for the client to change their mind.

Third, it’s a myth that people can tell you what you want.  Clients think they know, even if they are actually involved in making software, but they don’t.  Usability testing is the only proven way in software to validate if your software works or not.  To do usability testing, you need to have a working build of software.  To have a working build of software, developers have to make a build vs. work on the build.

Iterative development forces you to make working builds every 2 weeks, and gives the client the opportunity to validate that your implementation of user stories are actually valid.  This also allows them to validate the software is solving their current needs, and if not, they can shuffle the Product Backlog around, and re-prioritize for the next Sprint.

The Product Backlog contains all of the features your software has, and the Sprint Backlog has the frozen features your team is implementing this Sprint.  The key here is that the Product Backlog can change.  Whether your client wants to re-prioritize, or they have new things they want to add.

Scope Creep

Scope Creep is defined as the scope, or total requirements for a project, growing over time.  The word “creep” is used to imply that it’s growing beyond the originally agreed upon scope, and it’s doing it slowly enough not to raise the ire of anyone beyond those actually designing & developing the project.  It’s the bane of web & application developers everywhere.

Unless you use a Backlog in Scrum.

If the client wants to add new functionality, they can.  Adding new functionality to a project is often a good thing.  Whether a good idea that could of only arisen AFTER a build of the software was used by the client, or a glaring hole in available functionality was noticed weeks later.  Sometimes the new functionality is just a piece of functionality re-stated, but with additional work behind it.  Developers usually hate this because while the client perceives it as “the same scope, just re-worded”, developers know that it’s 3 times the work.  First, they already built it; second, they have to re-work their existing work assuming it can be re-factored at all; and third, they still have to code all the NEW functionality added…

…within the same timeline.

Unless you use a Backlog in Scrum.

Remember, the Sprint Backlog, the things developers are working on during a given Sprint, is frozen once your team starts.  It CANNOT be changed once you start.  The client is welcome to modify, change, and re-prioritize the Product Backlog whenever they want.  If the functionality your team delivers at the end of every Sprint meets UAT (User Acceptance Testing… also known as “we built exactly what you asked”), and the client wants to change it, it’s added as a new feature to the Product Backlog.  This is huge.

This means that anytime the client “adds” or “changes” something, it’s clearly documented in the Product Backlog.  The key here is if your team already delivered a user story from the Sprint Backlog, and the client wants to change things, they can’t magically undo the completed user story your team completed.  It stays completed, and a new one is created for insertion into the Product Backlog.  This is a track record for the PM and your team to clearly show they client they ARE adding scope the project.  Your team IS delivering on what they were asked to do.

I don’t want to get into teaching the client, managing their expectations, etc. I just want to point out here that’s how it’s supposed to work, and that’s fine.  Most Product Backlogs grow as the project moves forward.  This is a good thing.  It shows things are evolving and getting refined.  Your client is getting what they want.  Your team is shielded from scope creep sabotaging your work and the client’s expectations.

Product Development Helper

I’m involved in 2 products currently.  One is an Enterprise Dashboard Framework for my company, and one is an image app I’m working on with Joel Hooks.  Both projects were well under way when I was brought on board.  Both project owners specifically asked me to join to help them accomplish 2 things.

First, they wanted to launch.  Second, they wanted an opportunity to do Scrum development.

While I have gleefully accepted helping them reach these goals, this put me in a few challenging situations.

Economics of Design Phase

From my experience in software, I’ve found (and read online to corroborate) that the most impact with least cost is spent in the Design Phase.  This is when the User Experience (UX)/Interaction Designer (ID)/Information Architect (IA)/Designer people/person figures out what we’re building.  To give you context, let’s pretend I charge $50/hr (<– lol!) and conveinantely leave out the design part (where wire frames are made to look hot via Photoshop/Illustrator/your mom):

• 2 hours spent on wire framing a Login = $100
• 16 hours spent coding the Login to work = $800

Now, the ratio varies, but typically development time greatly exceeds time spent wire framing.  There is also a dependent relationship here that the above math doesn’t show. Attention to detail & well thought out user stories implemented in the wireframes often significantly reduces the amount of time developers have to spend.

• 2 hours spent refining registration = $100
• 6 hours saved by devs not having to stop, lose momentum, ask questions, get clarity, re-work, wtf = $300

Another advantage is “cheap flexibility”.  If you want change your mind about something in Design Phase, it’s really cheap.  I’ll often utilize a pen and paper for creating software, whether that’s in the beginning, or after we hit a design/functionality challenge in the design comps and we have to go back to the drawing board as it were.  Drawing boxes and arrows, and then summarily ripping out the piece of paper for 2 hours until we get it right.  This, as opposed to having developers figure out mid-sprint, costing them valuable momentum, and setting them up to fail.  …and costing a lot more money.

• 4 hours playing around with the Landing Page = $200
• 2 days using developers to play around with the Landing Page = $800

Keep mind, your Designer doesn’t contribute, nor is beholden to, Story Points.  If you utilize your Developers to experiment, you’ve detracted from their ability to deliver on their committed stories.  This costs you twice: Their lost productivity they’ll have to make up, and by spending more money using developers vs. designers to play around with a design.

False Starts

I’ve tried & failed enough times at launching my own product ideas that I’ve learned what not to do.  The #1 problem I ran into was lack of design direction.  As a single man team, I had to do design, front end & back-end development, and product management.  That’s hard.  If you know the idea, the market, and are passionate about it, you can actually do pretty well simultaneously performing all of those roles.

However, as a coder, I naturally spend way less time on design.  That was consistently a mistake I paid for. It absolutely annihilated my momentum.  More importantly, as I got frustrated I didn’t have design direction, knowing I’d have to stop coding and do more upfront design work, I lost the will to continue.

Once you lose the will, you’re screwed.  You won’t launch.

Design Phase vs. We’re Almost Done

Thus, for both projects, I’ve spent a lot of time on the design phase.  This means everything from documenting what we’re doing, drawings and wireframes, and keeping track of our Backlog.

The problem is, both teams are rearing to go… they want user stories to munch on with Sprints.  The old adage here, “garbage in garbage out” still applies to the Backlog as well, thus I’m sticking to my guns.  While we collectively could extract what user stories there are, and sort by priority, without a good design to guide those user stories, we’re just going down a misguided path with awesome equipment.

I haven’t figured out how to manage this.  My only saving grace is that I can easily remember back to when I was “just starting” or “re-starting for the 4th time” that my will was waaaayyyy harder to break in the beginning; it was only once I had been working for 3 days straight, and hit a design brick wall that it crumbled.  However, if I was just thinking and playing around, my will could stay strong for months.

We Need To Do This Also: Work Intimidation

Both projects have their challenges.  For the Dashboard, I know the audience, so it’s pretty easy to wireframe for it, and outsource the design.  For the image app, I have no clue who they are, and the descriptions I get from her majesty and Joel, I can’t relate to.  So, I had to create Persona’s, validate these persona’s, and then create goals from them.  Even simple goals lead to a lot of software work.

Another problem I found with trying to launch a lot of my own products and failing was breadth of work.  Not scope, but the amount of goals the software was trying the accomplish.  I’ve found the amount goals you’re trying to solve is directly proportional to the amount of work you have to do.  I was only successful when I targeted a reasonable amount of goals to the created a somewhat reasonable amount of work.

At first, you think “man… that’s ALL the software does?  This’ll be easy… and disappointing.  There’s not much of a challenge, and I’m not sure my users will like software that doesn’t do a lot.”

Bullshit.  If you believe that, you’ve never launched software.  I believed that, then I failed 50 billion times trying, and eventually gave in, and tried to launch even something mediocre.

The incorrect perception I had to overcome was that launching, with all the hard work it takes, is the end all, be all.  It’s not.  A “launch” just has to be a working product normal people can pay to use.  Like any web based, or self-updating desktop software, you can iterate well after version 1.

I’ve constantly had to remind myself of this as I document & wireframe functionality for both applications.  I get really disheartened when I recognize a Persona goal isn’t met, implement it on paper or the wireframes, and recognize the breadth of work I’m creating for my team.  The fun challenge I’ve found is figuring out how much I can take away, yet still launch.  The key point here is “take away” merely means tossed in the Backlog.  Good, valid idea, documented, but not making the 1.0 launch cut.

I don’t have to feel bad at all, in fact, I should feel good twice over.  First, because I’ve significantly reduced scope, and thus increased our chances of successfully launching; and second, I’ve done my part and documented the valid feature, and stored it in it’s appropriate place.

Painful process, but I seem to be getting better at it.  I can’t imagine building the Google.com homepage.

Projects Backlog

I do a lot. I’m busy.  I work a lot.  Thus, prioritizing my time in helping with these projects actually creates a “Project Backlog” in my real life… one that I am not following very well.  I can only do so much in a day, and if I don’t prioritize my tasks into a linear list, nothing will get done.  So far, I’ve tried 3 day sprints where I focus on 1 project for 3 days, and then hop back to the other.

Some days, that doesn’t work, and I get confused on which one was I was working on.  At the very least, I at least try to do my “5 minutes a day” rule: spend at least 5 minutes a day working on the task.

While the ivory tower solution is to focus on 1 project first, and get it done, it doesn’t work like that because there are dependencies.  So, I’ve tried to balance it out by working on 1 project while the other is in a lull/waiting on someone else.  Sadly, it seems a Project Backlog doesn’t really work because I can’t freeze it like a Sprint Backlog, and there are too many dependencies out of my control.  I refuse to waste time, thus, I’ll just hop to some other project to keep productive.

Conculsions

If I had the choice, I’d probably just focus on project at a time.  However, this is way different than coding.  With coding, assuming I have a ton of user stories and design comps waiting for me, the onus is on me, and the time is full.  With designing & managing… sometimes I physically can’t do anything other than wait.  Also, some of the tasks I need to do are a lot shorter.  Writing a document takes 30 minutes vs. 4 hours to code a component.  Therefore, it’s way easier to shuffle a schedule around, and make it full of multiple projects vs. just 1.  While hard, I feel like I’m on the right track… I just need another 10 years of practice.

Bottom line, I’ve found the Backlog really helpful in both capturing all we want to do, and all that we don’t have time to do.  It shields my from my clients inadvertently sabotaging my team’s development on their project, while ensuring we capture what they need.  It ensures everyone knows what they need to do, are setup to succeed, and is trackable.  It kills scope creep dead, and helps me reduce software complexity.