Agile Chronicles #7: Bugs, Unit Testing, and Throughput

The Agile Chronicles is a set of articles documenting my experiences using an Agile process (Scrum) in software development on my current Flex project.

1. Part 1 – Stressful
2. Part 2 – Code Refactoring
3. Part 3 – Branch Workflow
4. Part 4 – POC, Strategy, and Design Challenges
5. Part 5 – Acceptance Criteria & Punting
6. Part 6 – Tools, Extra Merge Day, and Postponed Transitions
7. Part 7 – Bugs, Unit Testing, and Throughput
8. Part 8 – Demo, Burnout, and Feature Juggling
9. Part 9 – Scope Creep
10. Part 10 – Conclusions

This entry is about the ugly truth of bugs, how you apparently reduce this problem via unit testing, and how we’ve attempted to improve throughput.

Bugs

Utilizing Scrum, you have a Backlog of user stories you wish to get in your application. You chew through this list of say 100 each Sprint to incrementally add to the finished product as well as slowly chip away at the backlog. The goal isn’t to implement every user story in the backlog, but rather to continually implement them well enough that the stakeholders can make the call when to ship. Remember, one of the main business reasons to utilize Agile methodologies is that you get something useful, sooner. As you know, there is significant change in our industry very quickly. These two churn factors, changing business requirements & changing technology landscape, results in software requirements becoming stale over time. Since the precept of Scrum is that you get working software sooner, stakeholders can make more educated inferences on what user stories they should add, when, and when you should ship a version to production. This could be when 40 or 99 user stories are implemented out of the 100 in the Backlog; it just depends on your current business situation. With product development, there are a lot of other factors that could be dependent on eachother. These can include hardware, marketing materials, the product’s website launching, a conference used as the PR launch platform, etc.

As time goes on, your team will start to develop a pattern.  You’ll collectively start getting an average number of user story points completed in a Sprint, and this in turn will allow the Project Manager to identify your average burn rate; the rate at which you complete user stories over a sprint. They can then extrapolate when the backlog will be completed with a decent degree of accuracy by doing the math. This in turn allows the stakeholders to make better decisions during the planning phase of each sprint, choose which user stories to prioritize in development, and plan the other facets of the project accordingly.

An example may be, if he needs a marketing screen cast animation for the website to showcase a cool feature, if that user story will be completed next sprint, he/she can then plan on showing that animation to investors in 3 weeks. The user story should hopefully make it through the UAT on the 2nd Friday. The designer can then do a screen cast of the feature with the working software on the following Monday, spend another day cleaning it up, and implementing it on the staging server by Wednesday. If all goes well, the CEO’s meeting on Thursday with the board of directors should be able to showcase the working feature in the real website.

Sounds great, doesn’t it? It did to me too until we had bugs. The only projects where I’ve never had any bugs were the really simple ones. I don’t just mean simple in terms of how hard it is to do something, I also mean where the stars aligned. We’re talking about the all of the following going off without a hitch:

1. The design was really easy to create in code
2. The designer clearly knew the capabilities of the platform and components
3. The Project Manager had clear requirements
4. The client was really positive, excited, and had 100% delegated authority to us
5. There was no complicated application logic.
6. The code flowed like a steady stream of water, took the shape of it’s encapsulated container, and like an icy tray under a faucet, was ready to be put on ice and delivered to the client.
7. Early demo’s, that worked well, really impressed the client.

That set of events hardly EVER happens. To me, it’s only happened when clients with no expectations want something really simple I’ve done 10 billion times before. I actively seek NOT to do those jobs because their boring. I’m a masochist with a short attention span, architect of my own demise. I like a challenge.

We started getting bugs. In terms of Version One, the web software we’re utilizing to track daily progress on user stories and hours spent, this is called a “Defect”. The bugs are added to the Backlog as defects, and are worth 1 user story point. Apparently this is ok, the but the understanding with Agile seems to be that you just don’t get bugs. My first gut reaction is “Agile is frikin’ dumb”, what the hell type of project doesn’t have bugs!? Oh yeah… the ‘really simple ones'”.

What’s even more frustrating is the way you code in Scrum; at mach-2. You code as fast and as hard as possible to get functionality working in a reasonable time frame. This means that potentially all of your code isn’t as encapsulated as it could be, you haven’t necessarily thought out all eventualities because you simply don’t have time, nor care, and some parts may desperately need some re-factoring later on. When later on comes around, and know a section of your code probably won’t change again, you start to re-factor. While things are easier to work with, you may have created more defects… because even with strong-typing and ActionScript 3 runtime exceptions, you still may have a system that isn’t as loosely coupled as you’d like.

In short, this atmosphere is ripe for creating bugs. Worse, it’s also ripe for regression. Regression is when you add new user stories and/or fix defects, and you break existing user stories. Apparently in Scrum, this is heresy. It’s ok if it happens every once in awhile, but if it happens every Sprint, clearly something is wrong, and people start to question if the code is spaghetti code.

Our code isn’t spaghetti code, but there are portions that have gone through SIGNIFICANT amounts of requirement changes, both in design, front end functionality, and back-end data modeling. Yet, the acceptance criteria hasn’t changed for user stories in that section.  The portions that have settled down, I’ve had time to re-factor those to be more encapsulated.  To me, the gross assumption for Iterative Development like this is that while exposing design and functionality changes in the beginning of the project is great, if it happens in the middle, you apparently aren’t doing it right. To me, this is wrong because just about every change we’ve made, I agree with. I really buy into doing software in Iterations because we getting real builds of it into people’s hands, and getting very valuable feedback, both from ourselves and the client.  Additionally, I believe we really are pushing the limits of Flash Player here for a compelling user experience; if things went off without a hitch, I’d question why everyone in software on the planet wasn’t using Scrum.  Now, churn has died down significantly, specifically this sprint. I feel we’ve reached the (mythical?) point in Agile where you feel like you’ve got a good assessment on how you’re software should work, so you can start battening down SOME of the hatches you know are pretty safe from churn.

Regardless, we still have regression issues. I’ll add the ability to page through dates only to find a redraw bug in another grid that shares the same date variable. To prevent regressions, we’ve implemented a policy of doing “smoke tests”, aka running through existing user stories and ensuring they still work before actually marking a new one done. In total, defects are taking around 30% of our time per sprint. Considering the amount of re-factoring I’ve done on both the guts of some components as well as the Mediators that hold the app together, I’d say that’s pretty darn good. It’s not good enough though. We aren’t chomping through user stories fast enough. This leads me to my next point…

Unit Testing

Based on my research, the only way to prevent regression is to spot it before it happens. The only way to spot bugs before they happen is to have good test coverage, regardless of whether your code is encapsulated or not. Good test coverage is having a large amount, and thorough, set of unit tests. A Unit Test is basically code that’s written for the explicit purpose of testing and trying to break your main code. The majority of what I’ve read online about Agile processes assumes some form of automated testing. Whether this is self-testing code, test harnesses, or a full-blown Continuous Integration with a build server, your code needs to be testable.

I’m still a Test Driven Development n00b, but suffice it to say, there are 3 main things I’m learning about unit tests. First, they assume your code is easy to test. If you are implementing unit testing to an existing code base, there is potentially a good amount of re-factoring you need to do just to get your code testable. There are a variety of techniques you can use to test code without needing to greatly modify it. Regardless, if you’ve no intention of writing easily testable code, chances are, you’ll end up doing a lot of re-factoring just to get it to test. I’ve always written test harnesses, which are basically a Flex Application MXML file that instantiates 1 of my components, and feeds it some test data (whether real or fake ValueObjects). This allows me to quickly test something in isolation, quickly, and determine who’s fault a bug is. Rather than finding a needle in a haystack (testing the entire application), I can combine both the thorough chunk by chunk testing of pieces of the hay in isolation as well as expediting where I think the bug is actually coming from. While helpful in making development and debugging quicker, it’s still doesn’t make the API of the components easier to test.

I’ve started following the TDD process in a side project after work hours to approach testing from another angle; test first vs. adding tests an existing project. It’s really neat in that it’s finding bugs before they happen, and it really does make me think about my component’s API, just like all the blogs and book said it would.

So, I’m in the process now of increasing my test coverage; writing unit tests for those parts that I know won’t get hit with changing requirements much in the next couple of sprints. At the same time, I’m learning how to write better tests, and improving the API’s to my components. This results in more re-factoring which increases the change for bugs and regression, but at least now I’ll have an easier time tracking them down. They are NOT replacing my test harnesses; there is no way a unit test, a non visual entity, can allow me to see my components working, and both play with functionality as well as visually confirm the design is implemented correctly, quickly. In tandem, though, I’m hoping all the blogs, articles, and the book I’m reading are correct about unit tests helping me prevent bugs and regression. If not, I’m screwed.

Throughput

One thing Scrum clearly has failed at is preventing me from working longer hours. Combined with the 2 above problems, I’m definitely in agreement with one of the criticisms against Agile, and that is that it only works with senior level developers. I take that to mean senior level developers do not have bugs, and have enough test coverage they can re-factor & add features with total confidence. Those things happen to me all the time on simple projects; perhaps if I used Agile on less challenging projects, I wouldn’t have these problems? It’s really strange, too, because you can be an awesome “senior level developer” on other projects and have zero need of unit tests. My guess is the definition of senior level developer includes product development, not service development. Bottom line, what this means for me is that I need to not only fix bugs faster, but also need to increase my test coverage quickly without compromising my main goal of implementing new user stories this sprint. The only way to do that is to work longer hours; not to many that I get burnt out and end up writing crap code because I’m tired, but just enough that I somehow work miracles.

I’m a firm believer that the best way to learn something is under an extreme time crunch, and this has been a great opportunity for me to learn TDD and writing unit tests in general. What I don’t know, however, is how to solve our throughput. As I’ve said, the amount of regression, while perceived positively by me, is not shared by our client in the overall scheme of getting enough user stories accomplished by their launch date. They are coordinating a lot of important milestones; the software I’m writing with my team is just one small part in this startup’s success. There are only 3 ways I know how to make our software occur faster:

1. Work longer hours
2. Cut features
3. Reduce our defects
4. Hire Another Developer

While most contractors / consultants consistently do #1 on purpose because you make more money, that’s not why I’m a contractor / consultant. I do it for the lifestyle; I like choosing the projects, like this one for example. Additionally, I like working longer hours because it’s my choice, and hopefully my fault for needing to do so, not some manager who f’ed up and now I have to work harder because of their mistake. So, I am working longer hours under a positive moniker; I know I need to get things done, and I agree wholeheartedly with my own decision.

The second one is pretty easy; based on us reaching a normalcy with user story points per sprint, they can determine what features will be completed by when. To what degree, well, that’s a point of contention. If I were to point out yet another glaring hole with scrum, it’s not well defined acceptance criteria. If a user story works, I don’t consider it done. I consider it done when it works AND it looks like the design. Some technical clients don’t care about the design. I do because of my art background and my belief that the user experience is directly related the many scientifically proven reasons of why good design is important. Additionally, some parts of regression may be minor enough that they detract from the over all experience, and give the impression that less user stories are completed than in fact really are. This is a powerful impression that regression and lack of finalized user stories can impress, and something I’ll remember next time. Bottom line, we have focused features we’re working on, and the rest we aren’t. This does help us both in focus and in morale a lot.

The third I only can trust in the writings of smart people that unit testing really does work. I’ve adjusted my expectations that it’s not a magic bullet, but I do feel the majority of the simple things will get nailed as I’ve seen it happen with my side project. The big things I usually know about anyway, and even add defects manually in Version One before the client and project manager know about them on Merge Day 1 of 2.

Number 4 didn’t work. I don’t think it’s a budget problem, but rather a communication one. My belief is the client is following a lot of the edicts that startup veterans blog about, and that is smaller teams lead to better communication. In Agile processes, it’s all about good communication. When I worked in bigger teams, I found communication became harder, separation of work became harder, and my productivity went down as I mediated between the team members. So, I respect the desire to keep the team small, focused, and communicating well. Regardless, there are parts of our software I know I could put another developer on, and allow them to work independently, making progress, without affecting my co-worker and me. I don’t think the added communication overhead would negatively affect my productivity too bad. The client is smart, though, so I’ll just trust in their decisions.

One things for sure. Every startup I’ve read about says they worked their tales off. So, I guess I’m doing things correctly when I increase my hours to increase throughput.

Conclusions

In my continuing research in online blogs, articles, and books, it’s clear that bugs are not at the forefront of a lot of the Agile methodologies, yet some form of testing is. This leads me to believe that the more test coverage you have, and the more senior you are, the more Agile will work for you. Hopefully the work I’m doing now will pay off in reduced overhead spent on defects in the future, and allow me to hit more user stories with more confidence.