Agile Chronicles #9: Scope Creep

The Agile Chronicles is a set of articles documenting my experiences using an Agile process (Scrum) in software development on my current Flex project.

1. Part 1 – Stressful
2. Part 2 – Code Refactoring
3. Part 3 – Branch Workflow
4. Part 4 – POC, Strategy, and Design Challenges
5. Part 5 – Acceptance Criteria & Punting
6. Part 6 – Tools, Extra Merge Day, and Postponed Transitions
7. Part 7 – Bugs, Unit Testing, and Throughput
8. Part 8 – Demo, Burnout, and Feature Juggling
9. Part 9 – Scope Creep
10. Part 10 – Conclusions

In this article, I discuss how Scrum for the most part brings Scope Creep under control, the areas where it doesn’t, and what you can do about it.

Introduction

One of the most interesting things about Scrum that took me 4 months to realize is that Scope Creep hasn’t been a major issue. It’s not totally 100% under control mind you, but I’m really impressed that it hasn’t come up in any defensive discussions like it usually does, aka, “It’s not my fault we’re not done, I’ve been working insane hours, blame it on scope creep”.

Scope creep refers to the ever increasing requirements of your software as a project progresses beyond what was initially agreed. Scope Creep, to me, is one of the main reasons why Waterfall, while still a valid way to make money, tends to produce non-useful software. The view of most developers is that scope creep is bad because it’s new work that has to be done in the same time frame, a time frame that’s probably already aggressive.

The term is used a lot more in agency work because developers are more aggressive about ensuring they know the scope of what they are developing. Since the time frames are a lot shorter, there is little to no room for screw ups and/or re-factoring. Any misinterpretation of what the client wants and how it is supposed to work can be disastrous. The developer(s) may have to re-code a lot of things from scratch, work longer hours when she/he/they are already exhausted, and be extremely frustrated the entire time doing it. Naturally, this is the worst possible time to not only add additional work or modified work, but adding it without thinking about the consequences of doing so. This is usually a recipe for bad software not being ready on time, with the project manager(s) left in the position of explaining why things weren’t done on time, and the developer(s) completely at the end of their rope.

It’s not usually this bad… it’s always this bad. Scope creep sucks.

Being held accountable for additional work added after the fact is just cruel to designers and developers and a way to ensure things won’t get done, nor done even remotely correct.

Yet the sad truth is that scope creep probably doesn’t have malicious merits behind it. Business moves faster than software development. Rather, it is probably important (or perceived as such), a valid functionality addition/change, or a new business requirement because things have changed since the project/Sprint originally changed. The target demographic for the site may have changed, the users had issues with a portion of the design after simple user testing, or perhaps a business to business application needs to solve a new need since the rules of engagement have changed. While you may not WANT to change things, it could mean the difference between the software being useless vs. useful.  Most importantly, like all software, the more you use it, the more you start to learn what will make it better.  You don’t know this until features start working, hence getting new scope in the middle of a project.

The Backlog Holds Scope

Scrum defines the list of user stories (features you wish to your software to have) and defects (bugs) be documented in a “Backlog”. This is a list of 60 or more things that you need to get done. At the beginning of each Sprint, the project stakeholder(s) & project manager decide what user stories and/or defects from the Backlog will be attempted each Sprint. Each User Story will have an assigned point value (not the law, just some metric to identify the user story’s overall value & challenge to implement) while defects are worth 1 point. Based on the average user story points completed per sprint, you then add a few more to ensure developers are overloaded just a little as well as having tasks to fall back to in case a user story cannot be done or becomes invalid.

As you can see, Scope Creep is deflected (mostly) for 2 reasons. First, you do not alter the user stories & defects chosen for a sprint after the fact. If you do, this sabotages Scrum. You don’t know what your team can accomplish each sprint because you keep changing their tasks. You never know what they completed in a normal instance, and therefore, have no metrics to use for projecting when things will get done.

Second, the additional scope is probably valid, so any additional scope creep is documented instead, and put on ice in the backlog. Rad!

This has a few other benefits as well. The client, who is already involved in the Scrum process and thus has transparency into what is going on can clearly see their additional scope adding to the projected project release date. How? Do the math:

1. 3 developers are completing 25 user story points on average per sprint
2. The Backlog contains 125 user story points worth of work
3. If each Sprint is 2 weeks, the project will take around 5 sprints, and thus 10 weeks

The scope creep then adds 10 additional points worth of user stories.  That’s at least another week’s worth of work.

This is powerful because you can totally stand by your point of, “Yes, adding scope does in fact push the deadline.”  To this day, that “fact” falls on deaf ears.  $10 says if it were a hammer, it still wouldn’t work.  At least now you have factual data; best you can do.

Again, you don’t need to complete the entire backlog to finish the project.  A lot of times you get the backlog 90% complete, and the stakeholders go, “Ship it!  We’ll do it in version 2”.  The point here is that a nice place is created for scope creep, the backlog, and the user stories can be re-prioritized each sprint so they do in fact become priority when it’s appropriate.

Invalidating or Modifying Existing User Stories

This works for for new scope, but not modified scope.  Modified scope is functionality that’s either modifies what you are currently making, or invalidates it.  For instance, I was coding a scrubbable timeline graph in my current project.  We found that the level of fidelity we were drawing at made scrubbing an ineffective way of perusing the data because it wasn’t scrubbing at a high enough fidelity.  You can only move something 1 pixel by hand, although Flash can draw at the sub-pixel level.  We attempted to solve this by drawing the graph in 2 view’s with the 2nd view only showing 1/6th of the data, thus giving you a more zoomed in view.  This still didn’t work because it negatively affected the importance of another design.

After trying this a few times, our project stakeholder and the designer called it quits and decided on using a separate view we had as the primary way of finding what the user was looking for.  So, within 2 or 3 clicks, the user could find what they were looking for, knowing full well the capable fidelity of what they were currently using.

While effective, I coded the same stuff 3 different times 3 different ways.  Very frustrating at first but I was really glad to get closer because we have a design that works; meaning, it’s actually been coded and tested and does what it’s been designed to do.  If I look at the original design comps I was given 4 months ago, and compare to what we have now, it’s totally different.  This increase of scope in coding new views, and modifying old ones were based on good design decisions.  I made that sound like it went smoothly, but it didn’t always.  We all were very frustrated at times with our picture of how something should work not always bearing fruit of that vision in reality, ie, our bi-weekly UAT builds.

When I’d do daily builds to test things out, the temptation was to modify on the fly… and we did some of that.  That is why a lot of my original time estimations on my user stories for the sprint were grossly under-estimated.  Either the designer and I, or the client, designer and I would try different things each day or couple of days to see if they worked.  While good for the app, the 3 main challenges were ensuring our decisions were well thought out, ensuring our decisions weren’t reactionary to a build-in-progress, and ensuring the extra scope was documented.

This got really fuzzy.  Was it new scope?  A new user story?  A modified user story?  Additional acceptance criteria?  A defect?  I felt like a lot fell to refined acceptance criteria, which naturally pissed me off because it sabotaged my time estimations.  However, I’ve never been negatively held accountable for my time estimations, only my regressions (broken functionality that had been working in previous builds), so I got over it.

Another type of scope creep beyond the invalidation & modifications of the above was when something compelling arises during development.  Either the designer will take personal liberties that go beyond the initial requirements and create something that is just really frikin’ cool, or we’ll use a stable build, and think, “Wouldn’t it be cool if…”.  I will say the former has drastically changed the course of our project multiple times.  For example, our application is not about video streaming, but reviewing already recorded video.  Yet, when the designers created a Quicktime movie showing their vision of how it should visually work, the streaming really was a compelling part of the app.  Suddenly the CEO of our client, who had been pretty adamant about about not supporting streaming at all, was taken aback, as all of us were, about how cool it looked.

To this day, streaming is still a tincy business value of the application we’re building, but originally none of us perceived spending any time on it until the designer took us to the next level.  This was easy scope creep to manage because we’d just add a bunch of new tasks into the backlog, and modify existing ones that weren’t being worked on yet.

The worse ones, though, are the “Wouldn’t be cool if…”.  Those are REALLY hard to track, and really hard not to immediately implement.  At a bare minimum, I document most of those new tasks into VersionOne, our tracking tool.  Keyword is “most” here because some are so small you wouldn’t think they’d negatively affect your time tracking metrics, yet they do because they are so numerous.  This gets into the same theory people use a large upfront requirements gathering session during Waterfall in that they don’t want nit-picking on already gathered functionality.  If it’s not in the spec, you aren’t doing it.  Yet, perhaps there is a valid reason you should try to add functionality?  Then for Scrum, this should technically be in your Acceptance Criteria… yet it can’t be there either, because you didn’t know it’d be cool till you actually saw it in action, and had that “ah hah!” moment.  This would be the only area where I’ve seen scope creep negatively affect my time estimations, and strain my ability to be a developer vs. a client manager.  When this happens, I struggle between being a developer that loves pleasing the client creating cool new stuff, and a consultant who doesn’t like the client disrespecting the process that they signed up for since that’s a reflection on you also not running your process correctly.  It’s hard to balance, at least for someone who has a strong need for “approval” of those I work for.

Conclusions

Bottom line, I’m really impressed with Scrum’s innate handling of Scope Creep.  It clearly recognizes the importance of new scope that’s late the game, scope that may hold the key to making the software relevant or not when it’s actually shipped.  I think any new business requirement that drastically changes the needs your application needs to address won’t prevent your project from being sabotaged; business moves wicked fast compared to software development.  Regardless, at least now the #1 reason I’ve seen projects be crap when they are actually released as well as creating massive amount of loathing in developers towards the client & project managers they work for is brought under control.  For the rest that slips by, as long as you document it, it won’t sabotage your tracking metrics used for projections.