Agile Chronicles #1: Stressful

The Agile Chronicles is a set of articles documenting my experiences using an Agile process (Scrum) in software development on my current Flex project.

  1. Part 1 – Stressful
  2. Part 2 – Code Refactoring
  3. Part 3 – Branch Workflow
  4. Part 4 – POC, Strategy, and Design Challenges
  5. Part 5 – Acceptance Criteria & Punting
  6. Part 6 – Tools, Extra Merge Day, and Postponed Transitions
  7. Part 7 – Bugs, Unit Testing, and Throughput
  8. Part 8 – Demo, Burnout, and Feature Juggling
  9. Part 9 – Scope Creep
  10. Part 10 – Conclusions

This entry is about what processes I came from, my definition of the Agile/Scrum process, and how stress has been spread out throughout the project vs. at the end.

Skip Intro

Preface

Five weeks ago, I started a project with Enablus, a firm that works with mainly startups to build products. I loathe service work, and love product work, so am really glad to be working with them. Even better, they are only 30 minutes from my house. I don’t really need to go more than 2 to 3 times a week, but I like how close they are when I do. One thing that struck me in the initial project discussions was that Enablus was really big on process and lingo. Darrell Ross, the project manager and partner I’m working with, although hard to read initially, still showed signs of uneasiness when I didn’t give him the answers he was wanting to hear in the initial discussions. For example:

“Do you utilize Continuous Integration?”

“Uh… what the hell is that?”

“You know, everyone on the team checking in code into a source control system every day, utilizing Test Driven Development practices, automating your builds, etc.”

“Er, I use Tortoise SVN and BeyondCompare all the time! Although, most people I work with don’t check in every day, don’t use TDD, and do not automate their builds.”

“I see…”

We’d constantly go through this. He’d ask a serious question like, “Do you utilize OOP and an software framework(s)?” and I’d respond with some cynical response questioning the merits of when OOP and frameworks are applied, or what I did and did not like about a particular framework, and how I don’t always officially follow their implementations… never really answering his question and thus not giving him confidence. I was constantly amazed at how structured Darrell made things sound, and he probably wondering how the hell a positive impression of “Jesse Warden” was formed in his head in the first place based on the answers I was giving him.

In all instances, we’d discuss the details of my responses, and he seemed to get the details he needed, feel better, and move on. Too ease his stress, I opened up and said that I’ve never really worked in places that were above a Level 3, and even those I felt never followed the processes he was mentioning, specifically Waterfall or Agile development, to the letter. Later in my consulting and contracting career, I had moved to using some concepts of Agile such as iterative development, and having builds of the software regularly on a schedule (usually weekly) even if the client didn’t need one for weeks or months. It had nothing but positive ramifications in the long run, so I stuck with what worked and saw other successful people doing similar things. It’s really a simple process (my version): Have a build working by Friday, every Friday.

I’m not sure Darrell believed me that everywhere I’ve worked, whether full-time or consulting/contracting, did not use a standardize process such as Agile. Maybe it did and I just wasn’t privy to the details?  Again, I never cared till this year.  Either way, nothing sounded as strict and defined as Darrell made it sound as he described the Agile process to me. Hence the point of this series of posts.

Introduction

I’ve never been interested software development processes. To me, it’s always been more important to be really good at the 3 main things that count: knowing your platform, knowing your tools, and knowing the language you code in. The better you know those 3 things, the more capable you’ll be at developing software, helping those on your team develop software, and thus ensure your team will be successful. As Dave Wolfe has said, it takes a village; a great team can ensure success. Whether the project actually succeeds isn’t really up to the software team at that point; hence the second rule to the “8 of 10 software projects fail” is “9 out of 10 are perceived as failures”. Even if you create an awesome piece of software, management, stakeholders, and/or clients can still perceive it as failing.

As I’ve found throughout my career, there have been more pressing issues than “process”, such as:

  • getting the dang design comps
  • making a workflow where fonts will compile on both machines (Mac and PC)
  • coming to a conesnsus with team members on how the framework you are using should be implemented
  • finding out what the heck we are building really is
  • finding out how the heck what we are building really works
  • resolving source control issues

There are tons of other things, but each has the capacity to reach fire drill status, and thus become a major time sink and productivity killer. As you know, losing time is the LAST thing you want to have happen in software development.

It also depends on the type of work. Working with Design Agencies, there was no process. Not because Agencies are dumb, but rather because the project timelines Agencies have are typically significantly shorter than traditional software or product development. On average, 2 weeks vs. 4 months. While a deadline can stretch to 4 months, you only work 10 man days on it, and of those 10, 5 are done initially, and the other 5 are done over the course of 2 or 3 months on client changes. In short, if you have 5 days to complete a complicated coding project, you’re more focused on getting things done than you are worrying about encapsulation, continuous integration, and frameworks. Writing code on day 1, and wondering who in the heck has time for UML and test cases. Agencies are deadline driven, whereas in software development, nothing is ever on time.

In consulting, you’re typically working within the constraints of your client’s company. It can go down two different ways. Either you get treated like a service shop, and requirements are thrown over the wall, and you complete how you see fit. Otherwise, you integrate with an existing team, and work within their processes, trying to improve them where you can. Contracting is usually the same way, although, you usually have little to no say in the process.

Even including regular work (W2 at a software shop), all instances can significantly have their chances of success improved through skill. If you’re skillful in the aforementioned 3 areas (platform, tools, language), it doesn’t matter how screwed up things are; you can always depend on skill to make or break a project.

…or so I thought. Going back to the 2nd rule of software development, “9 out of 10 software projects are perceived as failures”, I started to have more of the second rule happening vs. the first.  In those projects, more of my time would be spent doing non-software things, like pitching a tent in the project manager’s office, discussing details with stakeholders, drilling sales on what they were really trying to accomplish, etc. In the end, trying to define success to ensure my team and I weren’t setup to fail from the beginning. Being setup to fail, yet having a team with mad skill, results in you months later outside the office smiling on your umpteenth beer regaling all in the pimp architecture you created that never did, nor will, see the light of day. Creating awesomeness with no result is extremely more frustrating than flat out failing. The positive to failing is you can learn from it. Failing, even though technically you succeeded, is just straight ridiculous, and it was at those points I started to become more interested in the software development process. I realized I needed to find a team who knew all these processes I’d read about on the internet blogs.

Enablus is one of those teams. They clearly know Agile, as far as I know it 5 weeks in.

Agile As I Know It

I got Darrell to explain how Agile, specifically the Scrum part, works. This is my paraphrased, and re-interpreted version of our discussions, and my working with the process for a few weeks.  If you have a good idea, do market research to ensure it’ll sell, then you just need a good team to execute it. The execution on the software side for product development utilizes an Agile approach.

User Stories

The Agile approach, for the engineer, starts by defining “User Stories”. User stories are a set of sentences that describe what the user experiences while using the software. These can be derived from the wireframes; design comps aren’t a requirement. A user story can be expressed like so:

Login: User inputs Username and Password and clicks “Start Application”. Login is validated. For all other login instances, the Dashboard is displayed.

Notice there is no implied technology or design construction details. The user story merely describes what the user see’s, what interactions the do, and what the results of those interactions are. That same user story could apply to Flash, Silverlight, AJAX, PHP, etc. It’s technology agnostic; the point here is to document a user experience to ensure we’ve captured it well enough to be understood, and thus executed upon. The same goes for design. It doesn’t say “2 text input fields for username and password, with bold headings to the left, aligned center”. It’s left up to the interpretation of the designer to execute that user story, and hopefully provide direction for the GUI programmer through design comps assuming the user stories were derived from the wireframes, and not the design comps.

Notice also, it’s not a “feature”. It doesn’t say, “The user can login”. That leaves things massively open to interpretation, and solves no business value even if the feature is “completed”. As we all know, not all logins are the same. Documenting features, and executing them is one of the number one ways to be setup to fail, and is also one the main reasons Waterfall doesn’t work. If a software spec says, “The user must be able to log into the system”, but doesn’t describe registration, single sign-on, error scenarios, and how it’s supposed to look, you’ll end up with software that the client doesn’t like.

Some user stories are extremely high level. These are sometimes called meta-stories because they actually encompass a wide array of smaller user stories. Describing a login scenario can have a multitude of smaller user stories within it such as “user is presented with an error dialogue stating that they must first activate their account via the confirmation email before gaining access to the entire application’s feature set”. The level of detail only matters in so much as to make sense to the designer and engineer executing it. If a user story encompasses weeks of coding, it’s probably a meta-story and should be broken down into more manageable user stories that are easily defined, and more easily tackled in a reasonable time frame. What’s reasonable and doable is different for each team.

You create user stories for your entire application. What does, rather what SHOULD, the user be able to see, do, and experience?

Backlog

These user stories go into a “Backlog”. This backlog is a list of all the user stories, say 50… or a hundred. How many user stories you have are irrelevant. What’s important is that they are all documented in one place called the Backlog. You need to ensure all of the user stories are valid with the stake holders. It also helps to go over them with your entire team to ensure they make sense to everyone involved.

As you move through your project, you may find you’ll need more user stories to account for things you didn’t think about, or for additions/modifications to existing user stories. You then just add them to the back log. It’s normal for a backlog to grow during a project life cycle. However, you do not have to complete the entire backlog to complete the project. A lot of times, the stakeholders will make a business decision that 40 of the 60 user stories is enough to have a great product, and say “ship it”. That’s one of the key benefits to Agile is that you have working software throughout the project, and at anytime can make the determination of when it’s “done”… or is just a version 1.

Point System

Once your done with your backlog, you then have your team go through and assign points to each user story. To do that, you need a Point System. A point system is defining a metric for “how challenging something is”. In simple terms, a 1 is easy and a 5 is hard. It doesn’t have to be 1 through 5; it could be 1 through 10 or whatever. It’s best if they are numeric values because you use the points to measure a variety of statistics about your project. Additionally, each team may have their own definitions of what is challenging. While it’s best to use the same 1-whatever metric for the client and server team, the server team will most likely think creating a new server-side method a “1-easy” whereas a client developer a “5-hard”.

Once you’ve defined your point system and gained consensus amongst the team, you then go and assign a point value to each user story. This in essense gives each user story a “value”; how many points a user story worth. If you have multiple team members working on different parts of the system, say server-side and client side, you then have a point value added to each, such as “2 – mostly easy” for the server-side portion and “3 – doable” for the client side portion. The combined value gives the value of the user story, in this case 5.

From these point values, the project manager can then go back to the client, and ask, “With 20 points, what features do you wish to have completed in the first Sprint?”

The client can “purchase” user stories using those 20 points. A Sprint, which I elaborate on below, is an uninteruppted time period in which your team works on completing their assigned user stories. You make a guess in Sprint #1 on how many points your team can complete per Sprint. As time goes on, you start to see a pattern emerge of how many points per Sprint is the team completing. The client and managers can use this metric for planning and resource purposes. You can forecast, somewhat, how many Sprints you have to do to complete 60 points worth of features, and when you’ll have them completed. The chosen user stories that the client purchased are what your team is working on in Sprint #1.

Assigning points is a very important step for the engineering team. You typically don’t change point values mid-project, only assinging points to modified user stories or new user stories.

Sprints

As I mentioned, Sprints are a period of time that your team works on their assigned user stories. A sprint is however long your team decides it should be to get a working build with the user stories assigned. This could be one week, two, or a month. My current team is using 2 week cycles. It’s important to note the “uniteruppted” part. The client and project manager won’t interject user stories mid-sprint, nor modify your work load. Basically, they don’t bother you. This allows your team to focus on the task at hand, prevent fire drills, and stay productive.

In the examples provided, your team has 2 weeks to complete 20 points worth of user stories. Now, a sprint isn’t just, say 10 days of straight working. You usually have 3 special days. The first day is the kick off. You determine the user stories you and your team members will tackle. A couple days before the last is your “merge day”. It’s when everyone merges the code together from their different branches (or you just make sure trunk works if not using branches, more on that later) to get an idea of where the build is at, and where your team is at with their user stories. This is a final checkpoint before…. the UAT. UAT stands for user acceptance testing and in this case, it’s going over the final working build. Your team collectively goes over each user story, and identify if the build satisfies the user story or not.

As I said, our sprints are 2 weeks in length. The first Monday is spent confirming what user stories are in the current sprint and who’s working on them. We try to work in our own personal branches so the 2nd Wednesday is “Merge Day”, where I take any branches I and my teammate have, merge into trunk, and work out the kinks. We make a mad dash the rest of Wednesday and all of Thursday to finish our remaining user stories, or fix issues with existing ones, or just the build in general. Again, over the course of a sprint, you can start to get a group average of how many points your team is capable of doing in a sprint.

We’ve been using a shared Google Spreadsheet with each Sprint’s user stories in its own tab.  We collaboratively use this all the time for reference.

Daily Standup

We have a call everyday at 1 pm EST / 10 am PST. Our server-side time + client is out west in Cali, and our Flex team + UX + designer + project manager + rest of the team for other areas is east in Atlanta. We each go around talking about what we completed yesterday, this morning, and what we plan on going the rest of the day. We also cite any roadblocks or issues, and if that involves another team member, we just schedule a separate call, or time to talk over IM, with that specific person. This results in our calls usually lasting 5 minutes each day. I’ve heard of other projects at Enablus having their daily standup lasting longer than 5 minutes, but our team is pretty focused; we know what we need to do.

The majority of my road blocks involve a web service not working like expected suddenly, or questions for the designer on how a part of the design comp is supposed to work. Most of these I just solve on my own by calling the team member outside of the meeting time.

Pro’s & Con’s

There you have it, what Agile is based on what I’ve been told and experienced so far. So how’s it working out so far? Here are the pro’s and con’s for this first entry in the series.  I’ll elaborate in future articles, but here’s the quick rundown.

Pro’s

  • I’ve never been this stressed in the first two weeks of a project before.
  • coding with a “git-r-done” mentality; fast and furious.  Not everything has to be uber-architected.
  • each developer has their own branches; this makes checkins & merges a lot easier and less stressful
  • It’s nice to have something real after just 2 weeks.  Gives a great feeling of validation very quickly

Cons

  • a lot of re-factoring
  • creating new user stories later can break older ones
  • merge day is stressful
  • “need to see it working”; the desire to see working functionality before important functionality decisions can be made isn’t solved

Conclusions

In conclusion, I really like Agile methodology that we’re using so far as it allows me to code some things extremely fast and not have to worry about uber-architecting it. I just have to get it working which I’m good at doing quickly. For the things that matter, I can spend an allocated time which I feel good about doing because I can more easily see over time which areas deserve architecture and which ones do not.  The merge day is extremely stressful, but overall I like having an allocated day for it and using separate branches which prevent daily SVN drama.

I’ve never been this stressed in the first two weeks of a project before. The typical scenario is you get excited, and can’t wait to dive in. It’s a happy feeling filled with anticipation. Then, as the deadline approaches, you steadily get stressed out and things become less fun, albeit extremely frustrating. This isn’t always the case, but I’ve continually expected fun in the beginning, stress at the end.

It’s been really nice to be stressed at the beginning; this has implied to me the project won’t be so traditionally bipolar. The same thing happened to my mood once I capped myself at 2 cups of coffee a day. Think of it, you have 2 weeks to create working software; ready, GO! That’s insane, and really cool at the same time. What could you do in 2 weeks? It’s really nice to know that even after just 2 weeks of work on a 4 month (ish) long project, even the first 2 weeks will produce something valid and working.

Stay tuned for #2 in the Agile Chronicles series where I elaborate on the re-factoring challenges.

21 Replies to “Agile Chronicles #1: Stressful”

  1. Hey Jesse,
    Great post and your timing could not be better. Gotta agree, product dev is where it’s at. I to am just starting up with a company (SproutBuilder) that focus on product dev and uses SCRUM. I have been doing Flash/Flex dev for close to ten years at a wide variety of places but have never really been exposed to formal development processes like SCRUM. It’s been more like a little of this, some of that, okay yeah that works, good process!

    So far it’s a little overwhelming but on the whole looks like a good thing. I can certainly see the benefits for both the team and individual.

  2. Nice post, Jesse,
    The thing is regardless how agile a process could be, it always requires discipline and consistency, plus persistence as well. That makes the “stress” part of the deal. I know sometimes getting developers doing TDD is harder than make them learning a new language (or two).

  3. I’ve come to more or less the same conclusion: that I need to know more about software development processes, which until recently I regarded as within the realm of project managers and the like, even though I’ve been exposed to Subversion, Trac, Buzilla and the like. And like you I thought it was enough to Know The Things Developers Should Know: your platform, your tools, your language, your community. (Hey, that’s plenty! Can I have a life too please Sir? :)

    But a recent experience has taught me that, as I get involved in projects that are more like software and less like ‘branded applications’, it’s not enough to manage client expectations just for me, the contractor, I need to make sure the team wins as well, in a way I had not given a lot of thought to until now. And your post certainly echoes my recent conclusion that learning more about software development is one way to help achieve that.

  4. This is great – read it twice.

    The best thing that I’m taking away from this is getting my clients to write user stories. I usually just make them up myself but I can see a benefit to getting them to have more input.

    For almost my entire career I’ve been the solo developer on every project. The upside is that I never have to check my code in with anyone else or justify my choice in architecture. The downside is that I haven’t been able to mooch off other more experienced developers and learn from them. Thanks for taking the time to write about your experience here. I definitely got some good mooching in. :-)

  5. Excellent post, jesse.

    As polyGeek said, im taking the part with the user stories as well and will try to get this working within our shop. Its way better than the traditional features set that never get done in time by the managment team.

    I also like the part with points instead of “estimated time” we put on each feature here.

    Cheer and keep us updated with this serie of articles.

  6. Great post Jesse. I had the opposite experience on going to Scrum, going from a really long development cycle to one month sprints. It’s a bit jarring at first, but I enjoy it now.

    One thing that Ken Schwaber says is that a sprint should not a 30 day reoccurring death march. That’s an important piece of Scrum. The team should be allowed to determine what they can get done in a certain period, which makes it less stressful. It’s up to the product owner to decide which items are most important to get done.

    The comments about the first two weeks reminds me of this: http://martinfowler.com/bliki/EarlyPain.html

  7. great explaination from a developer’s standpoint.

    The development process is always important as lomg as it doesnt break the rythem of programming and it really delievers what you planned for sprints.

    Thanks again for sharing your experience.

  8. The discussion about Features vs. Stories was really important for me – it makes me think about articulating these subtle differences in my software process.

    What amazes me the most is that you’re writing about a remote and distributed Scrum process – I’m still dreaming about a process, team and tools mature enough to handle such remoteness and distribution. Can you share more about the way you handle remoteness and what set of tools you use to achieve really effective collaboration between team members and stake holders?

  9. Just came up for air and decided to catch up on your agile posts. This is very good and accurate I might add. I really enjoy hearing you compare and contrast Software Development and the reality of consulting we live in. I can only dream of landing a gig where they use a process other than “Get to coding Nerd! Oh, and I need it done yesterday, oh and it better be good or I’m not paying you. Cool let me know if you need anything, and by need anything I mean figure it out on your own”

  10. Great reading and always on time with the info. Was feeling as if I had become a retarded developer for a bit. Enjoyed knowing that even our role models are going through “The Change”.

  11. hi Jesse, Thanks for your thoughts on this subject. I’ve been trying to evangelise this to some of our project managers for a while, your articles have helped shed light ;) is there any chance you can release an example of your google docs template for managing Sprints? I was about to create one but it would be great to match it against one that’s actually been road tested! thxs paddy ;)

Comments are closed.