Improving Your Estimates: Part III

This is Part III of a 3 part series detailing common problems with software estimates and some simple ways to handle them. Part I can be found here, and part II can be found here.

Unfortunately, most of us software developers are pretty bad at estimation. Is it because we're incompetent? Of course not. It's because coding is creative work, and creative work is unpredictable. Writing software is not like working on an assembly line in a factory. Workers on an assembly line put together a product based on meticulous specifications and instructions. When it comes to software, that's the compiler's job. A team of programmers on a software project is analogous to the team of engineers that designed the product being put together on the assembly line. We might use a familiar set of tools and idioms, and carry over techniques and design elements from project to project (just like an engineering team does) but make no mistake: every project is different, and we are inventing/designing/specifying a new product each time.

Luckily, despite the unpredictable nature of the work, there are a few tricks we can use to make our estimates more accurate and more useful, and to prevent scheduling disasters.

Today, I'm going to talk about a couple of similar problems: getting a new assignment while in the middle of an important task or project, and realizing midway through a task that there are complexities you hadn't anticipated.

Problem: Gotchas and Reassignments

There are a few classes of time sinks that are harder to predict than those mentioned previously in this series. These can throw a real wrench in your estimates even when you do use empirical estimation and are explicit about what you're estimating, because they occur less frequently, but often eat up a bigger chunk of time when they do occur.

First, we have the “gotchas”. These are the unanticipated but necessary sub-tasks, the technical hurdles hiding in the details, and those subtle complexities in the problem domain that don't become obvious until you're knee-deep in conditional logic. As the saying goes, the devil's in the details; unless the task at hand is trivial, it's impossible to predict these things before you really dig into the code.

Next, we have reassignments. One of the most common delays in completing an important task is being reassigned to work on something even more important (or, at least, something that someone thinks is even more important). A reassignment may come from a different manager than your original task, and it may be disguised as a request that you “multi-task”, but don't be fooled. You can't work on something new without affecting the schedule of your original assignment.

There's not much that can be done to actually prevent these sources of delays, so we have to take a more reactive approach.

Solution: Early Warning System

When disaster strikes, a little time to prepare can make all the difference. That's why we have hurricane early warning systems; we can't stop the hurricane from coming, but we can prepare for the worst and ride out the storm in safety.

The same principle applies in software development. A key team member running behind schedule can wreak havoc on Gaant charts and project schedules. To a project manager, that's a real disaster! But, if you keep an eye on the weather, so to speak, you can mitigate the risk and prevent a catastrophe. (Even better, unlike real weather, sometimes by giving an early enough warning, you can avert the storm altogether!)

“Keeping an eye on the weather” means stepping away from your keyboard every few hours to think about how things are going. Just because you're being fantastically productive doesn't mean things are going well. (And if things are going poorly, it doesn't mean you've done something wrong!) Any of these warning signs should prompt you to alert your manager that the schedule is at risk:

• You find yourself thinking “Well, that part took longer than expected, but I'm sure this next bit will be easy. I'm sure it will all balance out in the end.” You should never depend on a lucky break to meet your schedule.
• You need input/feedback/help from another person and haven't had a chance to verify that they have time to spare.
• You've discovered a sub-task that wasn't accounted for in your original estimate.
• You've run into an “interesting” technical problem. These can be especially insidious, since you'll feel the urge to hack away and not come up for air until you've solved the problem.
• The domain turns out to be more complicated than you had anticipated. If the problem domain is complex, it's likely there are additional subtleties you will discover as you continue working on the task. This makes it unlikely you will make up for lost time later.
• It looks like you will finish just in time. This is usually a sign that you will finish late, but won't realize it until you're almost done.
• You are reassigned to some other work, even if you don't think it will take long. It's easy to get absorbed in your new task and forget how fast the clock is ticking on your original assignment. If the new task was handed to you by someone other than the person who gave you the original task, it's especially important to make sure everyone is aware of the switch in priorities.

There's no law stating that you can't revise an estimate after starting a task. The beginning of the task is the point in time when you have the least information on which to base an estimate. It would be silly not to refine that estimate as you gather more information. It's better to deliver a valid estimate a day late than to stick with an invalid estimate until the end.

---

And so ends the series. I hope my advice helps you avoid the scheduling problems that are so common in our industry. Remember, delivering reliable estimates takes more than just arbitrarily padding your first guess. By using the techniques described in this series, you can avoid nasty surprises for both yourself and your manager. Not only does this ease project scheduling woes and make your manager's life easier, it makes your life easier too.

Improving Your Estimates: Part II

This is Part II of a 3 part series detailing common problems with software estimates and some simple ways to handle them. Part I can be found here.

Unfortunately, most of us software developers are pretty bad at estimation. Is it because we're incompetent? Of course not. It's because coding is creative work, and creative work is unpredictable. Writing software is not like working on an assembly line in a factory. Workers on an assembly line put together a product based on meticulous specifications and instructions. When it comes to software, that's the compiler's job. A team of programmers on a software project is analogous to the team of engineers that designed the product being put together on the assembly line. We might use a familiar set of tools and idioms, and carry over techniques and design elements from project to project (just like an engineering team does) but make no mistake: every project is different, and we are inventing/designing/specifying a new product each time.

Luckily, despite the unpredictable nature of the work, there are a few tricks we can use to make our estimates more accurate and more useful, and to prevent scheduling disasters.

Today, I'm going to talk about two ways that an estimate can be interpreted, and the problems that can arise when you mean one thing and your manager hears another.

Problem: Two Kinds of Estimates

Another factor that can cause confusion between managers and developers is that there are two kinds of estimates. An estimate can be categorized by which of the following two questions it answers:

• How much of your time will it take?
• When will it be done?

The answers to these questions can be wildly different. The answer to the first question is (more or less) under your control, and once you start using empirical estimation, you can be relatively confident in your answer.

The answer to the second question, on the other hand, usually isn't under your control. Most development tasks require co-ordination with other people: users, customers, business analysts, QA staff, DBAs, and/or other programmers. All these people have busy schedules too, so when you're ready to co-ordinate with them, they may not be ready for you. A classic example of this situation is a overworked QA department. You may have some code ready for QA, but they might not get to it for two days, two weeks, or longer! You might only spend two days working on a feature, but that doesn't mean it's ready for production in two days!


When the manager is asking the second question (“when will it be done?”), and you answer the first one (“how much of your time will it take?”), you've got a recipe for trouble.

Solution: Be Explicit

The easiest way to deal with this problem is to be explicit in your communication. (In fact, that's a good way to deal with any kind of communication problem.) Tell your manager exactly what your estimate means (“this task will eat up ___ days of my time”, and what it doesn't mean (“this task will be done in ___ days”).

Now it probably won't be that easy. If your manager wants to know when the code will be ready for prime time (will it be ready in time for the demo to the big prospective customer on Friday?), you'll need to figure out who else will be involved, and go visit them with your manager to put together a picture of the schedule for the task. You may even need to get your manager talking to their manager to align the priorities and schedules of the two work groups.

It may seem like a hassle, but it will allow your manager to walk away with an accurate idea of how long things will take, instead of unintentionally promising that prospective customer a feature that won't be ready in time.

---

Stay tuned for Part III next week.

Improving Your Estimates: Part I

This is Part I of a 3 part series detailing common problems with software estimates and some simple ways to handle them.

Unfortunately, most of us software developers are pretty bad at estimation. Is it because we're incompetent? Of course not. It's because coding is creative work, and creative work is unpredictable. Writing software is not like working on an assembly line in a factory. Workers on an assembly line put together a product based on meticulous specifications and instructions. When it comes to software, that's the compiler's job. A team of programmers on a software project is analogous to the team of engineers that designed the product being put together on the assembly line. We might use a familiar set of tools and idioms, and carry over techniques and design elements from project to project (just like an engineering team does) but make no mistake: every project is different, and we are inventing/designing/specifying a new product each time.

Luckily, despite the unpredictable nature of the work, there are a few techniques we can use to make our estimates more accurate and more useful, and to prevent scheduling disasters.

Today, I'm going to talk about a very common cause of poor estimates: the difference between ideal days and real days.

Problem: Ideal Days vs. Real Days

Early in my programming career, I often found myself surprised, annoyed, or defensive when I estimated some task or feature as “two to three days of work” and someone showed up at my desk asking about it two days later. Eventually I realized the reason for my reaction. When I said “two to three days of work”, that's not what I actually mean. What I actually meant was “this task will take 16 to 24 hours of my uninterrupted time, based on my understanding of the task right now.” The difference between those two phrases caused me, and my managers, many more headaches than were necessary.

There are a lot of differences between the days we're thinking about when we give estimates, and the days that sneak by us so quickly in the run of a work week. When we're thinking about how long a task will take, we consider the details involved in executing that task, but not all the day-to-day distractions that eat up our time but have nothing (or little) to do with that task:

• Meetings.
• Customer support calls.
• Checking email, answering phone calls, and chatting with folks who drop by your desk.
• Helping the new guy set up his development environment.
• Participating in whiteboard design sessions, design reviews, or code reviews.
• Troubleshooting that strange intermittent network or VM issue.
• Helping a junior developer track down an elusive bug.
• Answering questions or pairing with a developer who needs your expertise about that part of the system you know better than anyone else.
• Did I mention checking email?

These kinds of time sinks are easy enough to account for, and most of us get better at factoring them into our estimates as we gain experience in the field. However, Hofstadter's Law (“It always takes longer than you expect, even when you take into account Hofstadter's Law.”) reminds us that we always have room to improve when it comes to providing more accurate estimates.

Solution: Empirical Estimation

Hofstaedter's Law implies that we are bad at estimation, even when we keep in mind that we are bad at estimation, because we are bad at estimating how bad we are at estimation! Fortunately, there's a trick for getting around this sticky problem: basing our estimates on historical data (a.k.a. empirical estimation).

The first step in the process is gathering that data. When assigned a task, record your initial estimate. Then, keep track of the elapsed time you spend on the task, including all the time sinks mentioned above. After a few weeks, you'll have a pretty good data set. Compare your initial estimates with the elapsed times for various tasks. You will probably be surprised by the difference between the two. Ratios of 1:2 and 1:3 are not uncommon.

You'll be tempted to dismiss the results because they make it look like you're a lot less productive than you really are. After all, those elapsed times include time spent on that “emergency” bug fix, the two-hour architecture meeting, and the morning you had to take an hour off to go to the dentist.

Whatever you do, don't give into this temptation to discount the data. All the factors that caused your estimate to differ from the elapsed time for those tasks are likely to crop up again in future tasks. The value in this exercise is forcing yourself to face the facts. All those things contribute to the difference between estimated delivery dates and real delivery dates. In the future, keep this ratio of on-task time to elapsed time in mind, and estimate accordingly.

If you're nervous that these new estimates will make your manager think you're just padding your estimate so you can slack off, explain your thought process. Let him know the task will take 6 hours of intensive work, but empirical measurement shows you that 6 hours of on-task work usually comes with 6 more hours of meetings, assisting other team members, and the like. It may help to record a sample of the types of time sinks you encounter while recording your historical data.

---

Stay tuned for Part II later this week.

Agile Adoption Patterns

This bit of news is a little late, but hopefully still useful to someone out there who's interested in introducing agile practices to their project.

A few months ago, Addison-Wesley released a great resource for teams and coaches relatively new to agile methods: Agile Adoption Patterns by Amr Elssamadisy. Unlike much of the Agile literature, this book doesn't just tell you what the end of the long road to agility looks like. It tells you how to get there, from where you are now. This book won't solve any problems for you, but it'll help you recognize the problems you'll face, and it'll arm you with some tools to deal with them with a minimum of pain and frustration.

I've reviewed the book for InfoQ, so you can read more of my thoughts on it here.

Making Documents Lean

I was going to write a whole post about lean documents, but while doing my research, I quickly came across this article by Scott Ambler over at AgileModeling.com. It's a long read, so if you're short on time, read the Critical Points and Why Do People Document? sections. The former makes a great print-out list for your cubicle wall. The latter is more politically sensitive, so I wouldn't post it in public (since it's probably only relevant if you have a traditional, political workplace), but it is a great resource if you're interested in trying to introduce agile ideas and want to avoid resistance and backlash.

If you're really short on time, here are some highlights:

• The fundamental issue is communication, not documentation.
• Agilists write documentation if that's the best way to achieve the relevant goals, but there often proves to be better ways to achieve those goals than writing static documentation.
• Document stable things, not speculative things.
• Take an evolutionary approach to documentation development, seeking and then acting on feedback on a regular basis.
• Prefer executable work products such as customer tests and developer tests over static work products such as plain old documentation (POD).
• You should understand the total cost of ownership (TCO) for a document, and someone must explicitly choose to make that investment.
• Well-written documentation supports organizational memory effectively, but is a poor way to communicate during a project.
• Documentation should be concise: overviews/roadmaps are generally preferred over detailed documentation.
• With high quality source code and a test suite to back it up you need a lot less system documentation.
• The benefit of having documentation must be greater than the cost of creating and maintaining it.
• Ask whether you NEED the documentation, not whether you want it.
• The investment in system documentation is a business decision, not a technical one.

Risk Management Lessons Learned From Texas Hold 'Em

A couple years ago, I was an avid Texas Hold 'Em player. For about a year, I played 4 tables at a time online for a few hours a day, I read poker books, and I made enough money to pay my rent. (Because you're playing against other players rather than against a casino, it's actually possible to make money at poker.) Eventually it started feeling like a grind instead of a hobby, and I gave up on the game to spend more of my spare time on software-related pursuits.

These days I just play occasionally for fun, but I'm glad I put so much thought and energy into the game for a while. Hold 'em is a true meritocracy; you will only win in the long term if you are better than your opponents. There is no vagueness to the result. You either quickly learn to plug the holes in your game or you lose. Because it's a non-trivial undertaking but the results are so unavoidable, there are several life lessons that the game illustrates quite well.

Here are a few of my favorite lessons from hold 'em as they apply to risk management in software development projects:

Understand expected value, and use it!

Players who don't understand the concept of expected value don't generally do well at hold 'em (or any form of gambling, for that matter) in the long term. If you understand your chances of winning, the cost of losing, and the reward for winning, the decision of whether to call a bet or fold becomes purely mathematical. If you don't, the best you can do is guess. Often, you'll guess wrong.

Often project managers and sponsors usually don't know how much a project will really cost them or how much value they will gain from it. Even if cost & benefit projections are available or relatively easy to calculate, no one calculates the expected value.

I know of one company that began calculating the expected value (or ROI) of projects and realized they had signed a deal with a development cost ten times higher than they would get paid for it. Obviously projects like this have a negative expectation, but they get undertaken surprisingly often if no one checks the numbers.

The same analysis can (and should) be applied at a more granular level. It's old news that the majority of features in an average software product are rarely or never used. I have a suspicion that these features sneak into software products because no one bothers to think about their expected values.

One more thing: expected values aren't just for deciding whether to play a hand or not. You can (and should) also use them to decide whether to stay in a hand. We all know perfect project cost estimation is impossible, so continuously revisit your estimates once you have a few iterations under your belt. If the expected value of a project or feature changes, don't ignore that information! That brings us to lesson number 2:

Don't throw good money after bad.

The biggest mistake most players make is putting more money into a pot when they have very good reason to believe they are going to lose. Why would someone do this? They've crossed their threshold of pain. In other words, it's likely they've already lost $100, but there's some slim chance they haven't. The psychological difference between losing $100 and $200 is minimal, but the psychological difference between losing $100 and winning a few hundred is huge. Unfortunately, when you make decisions based on the emotional distance between different outcomes, rather than based on the statistical likelihood of those outcomes, you'll get yourself in trouble.

The same thing happens on software projects. Even when it is clear a project is a train wreck, if enough money has already gone into it, people will get really good at coming up with reasons to continue putting money into it. Don't be one of them. The fact that you made bad decisions is not a good reason to continue making bad decisions.

- - -

The interesting thing about these two lessons is that they aren't by any means earth-shattering; they're basic common sense. They basically boil down to this: only make decisions that make financial sense. After reading this article, you may find yourself thinking "Well, duh!"

It's tempting to dismiss the above lessons for that reason. We'd all like to think we are above irrational behavior, so we don't need to be reminded of such common-sense lessons. In poker at least, that's clearly not the case. The biggest challenge for would-be poker pros isn't knowing when to call, when to fold, when to raise; it's having the self-discipline to fold when you know you should. Optimism, pride, and desperate hope drive them to keep putting money into a pot they're unlikely to win.

I think the same thing happens in software development. Everyone realizes that they shouldn't pay twice as much to develop a product as they will be paid for it. The problem is that we don't always have to self-discipline to stop, measure both sides of the equation, and make the hard decisions based on the results.

How To Tell If You're Doing Agile Right

Agile software development is about asking a question, taking a best guess at the answer, acting on that assumption, and then back to asking a question about what happened as a result. It's about gathering feedback and acting on it in a continuous cycle. That's it. Iterations and retrospectives are the core of any shift towards agility. They are axiomatic; everything else can be derived.

All the popular agile practices like test driven development, continuous integration, refactoring, short iterations, and daily stand-up meetings are nothing more than some of the answers a lot of people arrived at after asking some interesting questions. (Edit: That might sound like I'm discounting these practices. I'm not. I've found them all to be very useful tools.)

If the only question you're asking is "are we doing Agile right?", then you're probably not. There's no magic formula. Some practices are helpful across a wide variety of contexts, but there's no guarantee a given practice will be right for your situation.

If you're asking questions like these:

• What is our biggest problem? What is the most important question we can ask ourselves right now?
  
• What are the expectations of each stakeholder group? What are our stakeholder groups?
• What are this project's risks? Can we reduce any of them? How can we force ourselves to tackle them early? What mitigation strategies can we put in place? What can we do to help ourselves notice new risks as the project proceeds?
  
• Is this project worth undertaking? What is the expected value? Do we know the TCO of the project? Do we know the opportunity cost?
  
• Why did we miss that deadline? Did anyone realize that would happen and not say anything? Did everyone realize that would and not say anything? What was the cost of missing the deadline, and what was the cost of finding out we would miss it so late? Would we be in a better position if we faced reality sooner? How can we help ourselves do that next time?
  
• How much is employee turnover really costing us? Can we do anything to reduce this cost? Can we do anything to reduce employee turnover? Is it cost-efficient to do so?
  
• How did that bug get into production? What changes can we make to prevent similar bugs from reaching production in the future? What will the side effects of those changes be?
  
• How can we improve the way we work? What do we mean by "improve"?
  
• Are users actually using this feature? How are they using it? What do they think about it?
  
• Why are we adding this feature? What is the feature's TCO? What value will it deliver? What risks are involved with adding this feature?
  
• What have other teams with this problem done? Do any of their solutions make sense in our context? What makes us similar/different?
  
• How do you unit test a user interface? Is it worth it (for us)?
• Why are we producing document x? What is it's TCO? What is it's value? Who's the audience? Do they have any ideas about increasing the value of the document?
  
• Is this particular "best practice" or "agile" practice helping us work more effectively? Are we missing something important, or is it just not useful in our context?
• If we don't know the answer now, how can we find out quickly?

... and if you're asking them collectively, as a team, in a blame-free manner, then it's likely you're on the right track.

Of course I'm not prescribing that you ask exactly and only these questions... they're just the kind of questions that I find lead to positive changes and other interesting questions.

I'm sure you are all thinking of other questions that should be on this list. Please share them in the comments!

An Agile Bookshelf: 10 Must-Read Books

I did my best to select books I believe have something to offer for both those already doing agile software development and those who are curious about agile but remain skeptical. I'd like to think that even if you think "Agile" is a useless buzzword, you'll find something useful in each of these books. Quite simply, they are books that have changed the way I think about software development and how I practice it.

Without further ado (and in no particular order):

Agile Software Development: Principles, Patterns, and Practices
by Robert C. Martin

This book is more about good object-oriented design as it is about agile development practices. In fact, Martin is done with his overview of agile practices by page 84 (out of a total of 524 pages.) The rest of the book focuses on fundamental principles of OO design (using the definition of design from Jack Reeves' classic essay "What Is Software Design?".) This book confirmed my suspicions that there was a strong relationship between good OO design and agile practices. Good OO design gives your software the flexibility it needs to enable rapid response to change. Agile practices (particularly Test Driven Development) give you the feedback you need to keep your design clean... if you have the discipline to do it.

This book is best suited to developers who already have a few years of practical experience and a good understanding of the basics of object orientation. It will solidify your understanding of good OO practice, provide a few new insights, and give you a vocabulary to talk about both. I consider it a must-read for any serious professional working in an OO language.

Working Effectively With Legacy Code
by Michael Feathers

"Is your code easy to change? Can you get nearly instantaneous feedback when you do change it? Do you understand it? If the answer to any of these questions is no, you have legacy code, and it is draining time and money away from your development efforts."

So begins the back cover of this book. Feathers throws out the traditional (and vague) definition of legacy code and replaces it with this one: code without tests. Without tests, you don't know whether your software is getting better or worse when you make changes. Without rapid feedback after changes, quality suffers and time is wasted in tracking down defects further down the line (when it's much more expensive).

I meet lots of people who are curious about unit testing, but don't know where to get started with their current (untested) project. Feathers explains a number of techniques to get you existing code base under tests as safely and painlessly as possible. If you're interested in developer testing but feeling overwhelmed, or just wondering how to tame a big, messy code base, this is the book for you.

Test-Driven Development: By Example
by Kent Beck

When you get into the habit of writing a test before writing the code that will make it pass, some very interesting things start to happen. Beyond the benefits you get from unit testing after coding (a growing suite of regression tests, confidence your code works, etc.) you get several new benefits: better design (because you're forced to pay attention to it earlier in the process), tests that are more readable and less fragile, and a lot less time spent manually testing and debugging.

You also start spending a lot less time writing unnecessary code, since your tests are derived from actual requirements, and you don't write code unless you need it to make a test pass. The tests act as unambiguous, executable requirements. You've connected the dots between vague requirements and working code.

This book is an an hands-on tutorial of how to develop software test-first. If you're curious about test-first development, this is a great book to get you started.

Refactoring: Improving the Design of Existing Code
by Martin Fowler

No one gets design perfect on their first guess. You will always have a firmer understanding of the needs of your design after getting feedback from reality than you did at the beginning of a project. Besides that, the needs placed on the design will almost always change over time, so even if you guessed right, your design is going to change. If you don't make an explicit effort to change it, it will degrade over time as hacks are put in place to work around the current design in order to get new features in place.

The best way to change design is to consciously evolve it over time. Keep it clean; improve it a little bit every time you add a feature, and you won't hit that dead end where it's impossible to add new features and a total rewrite is the only option. This book shows you how to do that. It explains the link between good tests and refactoring, lists the 'code smells' you should keep an eye out for that may be a sign that your design isn't optimal, and gives some advice about which refactorings are most appropriate in each situation. This book is a must-read for anyone who is interested in software maintainability.

Agile Estimating and Planning
by Mike Cohn

This book takes the guesswork out of software estimation. In it, Cohn outlines an empirical estimation process; that is, one based on evidence rather than predictions. Rather than just take a best guess at the beginning of a project (when we have the least information of any point in the project's lifetime), he uses early measurements of the work capacity (or 'velocity', in agile terms) of the team to determine the real pace at which the team can deliver working software. This gives management critical information early (allowing them to adjust scope to meet a given deadline, or adjust the deadline to meet a given scope). This puts an end to the trap customers, management, and developers so often find themselves in: not realizing until very near the deadline that you're way off schedule.

If you've ever been surprised by a schedule slip, or want more predictability in your software development process, read this book.

The Goal
by Eliyahu Goldratt

This is the business novel about The Theory of Constraints, a set of principles for continuously improving operational efficiency. Although the specific example in the book is industrial, the principles also apply to software development.

The book is a fun and easy read, but contains a number of dramatic insights into what productivity really is, how it can be related to everyday tasks, and how it can be improved. I don't want to elaborate and ruin the story for you. I'll just say this: I can practically guarantee you will discover unnoticed inefficiencies in your software development process after reading this book.

Fearless Change: Patterns for Introducing New Ideas
by Mary Lynn Manns and Linda Rising

Very few software professionals work alone. So chances are, if you have a brilliant idea about how to work more effectively (agile or otherwise), you're going to need to convince others to do it too. Usually, the higher-leverage the idea is, the more people you need to convince. Unfortunately, people often instinctively resist change, and most of us don't have a natural gift for making people more comfortable with it. That's where this book comes in.

This book is about introducing new ideas to established organizations. The first half of the book explains the nature of the task, your role as a change agent, and the challenges you'll face. The second half is a catalog of patterns you can use to face those challenges.

Introducing new ideas to an any large organization is never easy. If you've ever thought of a more effective way to work, but had trouble convincing others to adopt it, this book can help.

Waltzing with Bears: Managing Risk on Software Projects
by Tom DeMarco and Timothy Lister

Risk management is possibly the only element of management that cannot be delegated; at the very least you have to manage the risks of delegating risk management. Neglecting sound risk management principles and practices can be an extremely costly mistake. Unfortunately, risk management is done extremely poorly, or not done at all, in many development organizations.

In this book, DeMarco and Lister explain why risk management is so important, why it is often done so poorly, and how to do it effectively. It's short and conversational, but packed with valuable ideas. I recommend this to anyone who doesn't want another late, over-budget, or otherwise failed project on their resume.

Crucial Conversations
by Kerry Patterson, Joseph Grenny, Ron McMillan, Al Switzler

A 'crucial conversation' is any conversation in which opinions vary, stakes are high, and emotions run strong. They have long lasting effects. They can transform relationships, both for the good and the bad. They are the most important conversations we have, and have long-lasting effects. Unfortunately, most of us handle crucial conversations poorly.

In this book, the authors provide tools for getting better results from crucial conversations. They explain how to recognize when a conversation turns crucial, how to focus on your real goals (rather than your transient emotional goals, like your sudden urge to prove the other person wrong), how to curb your own defensive/aggressive reactions, how to share information without making others defensive, and how to create conditions of safety that allow the other party to share information freely.

Unless you work by yourself, you need these skills to work effectively. (And they're handy skills to have for your life outside work, too.)

Peopleware
by Tom DeMarco and Timothy Lister

This book questions the application of standard management theory to knowledge workers. They show how this misapplication causes many of the problems perennially faced by managers of software professionals, and outline some new management principles specific to knowledge workers, backed by years of research from the authors' work with dozens of software organizations. Even better, they point out where management is missing out on opportunities not just to avoid problems, but to actively develop highly effective, productive teams.

Peopleware is a short read full of interesting ideas. Chances are you'll strongly disagree with some of the ideas in this book, consider some of the others common sense, and pick up a half dozen new insights along the way. I'd recommend this not just to managers at all levels, but to anyone who works on a team as well.

How to Deal With Frustration

I just wrapped up a contract with a large corporate client. As is the case with almost all large organizations (even those attempting to transition to agile practices), there was no shortage of wasteful corporate policies and procedures.

Nothing frustrates lean-thinking people quite the way inefficient or wasteful processes do. We all want our expertise and abilities to be leveraged to create and deliver as much value as possible. When wasteful processes prevent this, we get mad. Not because we're inherently negative people, but because striving for greatness starts with refusing to be mediocre. We try to fix the broken process. If we don't immediately succeed, we bitch.

When you get three or four passionate lean-thinkers together, their frustration can feed off each other and become a real problem. But until recently, I didn't realize that. I might occasionally point out we were wasting time and should get back to work. That was about it.

Then I read My Start-Up Life by Ben Cosnocha. In a side bar about time management, Ben says:

"I believe time management is important to make each day rewarding and productive. But instead of focusing on how I can save a minute here, a minute there, I think about a different metric: energy."

He goes on to quote The Power of Full Engagement, by Jim Loehr and Tony Schwartz:

"Energy is the fundamental currency of high performance. Capacity is a function of one's ability to expend and recover energy. Every thought, feeling, and action has an energy consequence. Energy is the most important individual and organizational resource."

That distinction between time and energy rings true with me. When I'm energetic, I'm able to work smarter and think faster, and I can get a lot more done. Time is not the limiting factor of my productivity; energy is.

When we focus on our frustration with problems that are not immediately or easily solvable, there is a bigger issue at stake than wasted time: wasted energy. Energy is the leverage that allows us to make effective use of time. When we allow ourselves to become frustrated at our inability to prevent waste, we deplete our energy. Our frustration with waste is often more wasteful than the waste we're frustrated with in the first place.

I'm not saying you should just put up with all the wasteful policies you're asked to comply with. But avoid the bitch sessions. Be aware of how focusing on your frustration affects your energy level, and act accordingly.

Are Your Unit Tests Too DRY?

While tackling some long-overdue refactoring of a particularly tangled part of our current project, Dmitri and I were faced with the task of updating some long and unreadable unit tests that we had just caused to fail.

We had made a classic mistake earlier in the project and neglected technical issues like dependency management in a rush to be productive and show our customer some quick progress. We ended up with poorly structured and highly-interdependent classes. This made writing simple tests impossible. Trying to test a single piece of functionality often required 20 to 30 lines of setup code. As I mentioned last time, this resulted in a downward spiral: poorly structured code made it hard to write good tests, and a lack of good tests made it hard to refactor the code.

So here we were. We had struggled for several hours to decipher some tangled code and find a good starting point for the refactoring (what Michael Feathers calls a seam), we had made a small change, and we'd run the unit tests. Now we were staring at a list of 4 failing tests, and not only did we not know why they were failing, we had no idea what they were trying to test and how they were testing it.

One of the problems with the tests was interesting. Normally, I am a huge proponent of DRY programming. If there is no good reason for duplicating data or logic, don't duplicate it -- and there is hardly ever a good reason. However, we had just stumbled upon one of the good reasons.

Unit tests should function as documentation. You should be able to read a unit test in 10-15 seconds and know what requirement it is testing, and how client code should interact with the code that implements that requirement. If our tests had had this property, we would have saved ourselves the hours of trying to figure out what the code was doing, and the hour or so it took us to figure out what the tests were doing, why they were failing, and how to fix them.

If you feel the urge to DRY up your tests, stop for a minute and question whether you are working at the best point of leverage available to you.

If DRYing up the tests seems like the only way to set up tests in under 30 lines of code, your real problem is that the code you are testing is poorly isolated. This is probably your problem if you find yourself writing methods named setupTypicalUserAccount(), setupOtherThingyForTest(), etc.

If DRYing up the tests seems like the only way to make the tests readable, the code under test probably doesn't have a useful or meaningful interface. For example, you may be tempted to create a factory method in your test that calls the three or four methods necessary to put your target object into a valid state. Put the factory method in the target class instead. In general, if you can remove duplication in tests by making the interface of the code under test more humane, do it. If code is difficult for your tests to use, it will be difficult for other client code to use too.

Of course, soemtimes DRYing up tests is a good idea. If you want to replace repetetive lists of assertions with your own assertion methods, go for it. There's nothing wrong with an assertUserAccountIsValid(Account account) method that contains half a dozen assertions. There are lots of times when DRYing up your tests really is a good idea. Just make sure you've got a good reason and are not running on instinct.

Think of your tests the same way you'd think of a user manual or other documentation. You wouldn't want to repeat yourself on every second page, but you also wouldn't want to eliminate duplication entirely from a user manual. You'd drive your users nuts. In the case of a user manual, readability trumps maintainability, and the same goes for your tests. (Of course, you want both readability and maintainability if you can get it.)

If you found it impossible to write a clear, concise user manual, you might start worrying about the user interface of the program itself. If you find it impossible to write clear, concise tests, start worrying about programming interface of the program itself. That's probably your best point of leverage.

Why You Won't Fix It Later

We've all been there. The deadline is looming, everything is behind schedule, and you're in a rush to finish the FooBar module. You're puzzling over one last glitch. You know how to fix it, but it looks like it will take a minor redesign of the module... probably 4-5 hours of work. You just don't have that kind of time.

Suddenly a clever idea strikes you. Hmmm... it just might work. You realize deep down it's not the right way to do it. Maybe it means adding some temporal/implicit dependencies. ("as long as no one starts calling foo() before initBar(), everything should keep working.")

Maybe it means throwing in a magic string that will only work until January 3 next year. ("No problem, I'll just come back to this code after the deadline. We shouldn't be too busy then.")

Maybe it means breaking the design and making the code untestable. ("Well, it would be nice to have automated tests around this, but it seems to be working. Hopefully no one makes any changes to this code before the deadline.")

Maybe it means living with intermittent bugs. ("Hmmm. The system only times out 8% of the time. We need to figure out why before we go into production, but that should be good enough for testing.")

Maybe it means removing one bug and introducing another one. ("Well, at least we can submit the page now. Hopefully none of the users double-clicks the submit button until I've had a chance to revisit the code after the deadline. I'll fix it later.")

That's the magic word. Later. It makes a great warning signal that you may be heading down a dangerous path. When you catch yourself thinking "I'll fix it later", stop for a minute. You're feeling that little twang of guilt for a reason (even if it's masked by the little ego boost you get from coming up with such a clever workaround). Think about the real consequences of this decision. Will you really get back to it later? What will happen if you don't? What are the risks you're introducing? Ask another developer for an opinion. Ask the customer for an opinion (if you can phrase it in customer language). Think a little longer about other solutions.

There are several popular variants of "I'll fix it later":

• I'll fix that bug later.
• I'll verify with the customer that I've built what they actually need later.
• I'll write unit tests later.
• I'll remove the fragility from the unit tests later.
• I'll make the unit tests readable later.
• I'll make the unit tests fast later.
• I'll integration test later.
• I'll usability test later.
• I'll remove that copy/paste duplication later.
• I'll bounce my idea/design/code off another developer later.
• I'll remove that workaround/hot fix/complete hack later.
• I'll make the code readable/maintainable later.
• I'll worry about performance/reliability later.

The problem is, we usually don't get around to doing any of those things we plan to do "later". After dealing with the consequences of "I'll fix it later" a few too many times, my friend Dave LeBlanc coined LeBlanc's Law:

"Later equals Never."

Why is this? There are a few reasons that I've noticed:

1. When you cut corners in order to deliver on time, you're giving management and your customer a false sense of how fast you can reliably deliver. Agile teams use the term 'velocity' to describe the estimated amount of customer value they can deliver per iteration. If there is still work left to be done, you are effectively lying to your customer about how fast you can deliver value. Since your customer thinks you can deliver more than you really can, you will be overloaded with work again next time. You will start accumulating technical debt. There is no easy cure for technical debt (the most common cure being a complete re-write), so prevention is the best medicine. The best way to prevent technical debt from accumulating is to establish realistic expectations about how fast you can effectively work.


2. When you skimp on automated tests, and even when you write tests but don't ensure they are readable, atomic, and easily-maintained unit tests, you limit your ability to effectively refactor. When you can't easily refactor, it begins to get hard to write readable, atomic, and easily-maintained unit tests. Not only that -- because it's harder to evolve your design, you will face a stronger temptation to fix bugs with workarounds and hacks that will come back to bite you later. You will spend more time debugging and bug fixing, leaving you less time to write tests and refactor. It's a downward spiral that results in reduced velocity.
   

Agile developers often work with what they call a "definition of done". You are not finished with a feature until it meets the definition of done. It acts as a checklist or set of heuristics that help you realize (and admit) when you have more work to do. A definition of done might include things like these:

• unit tested
• verified by customer & customer tests
  
• usability tested
• integrated
• integration tested
• documented
• performance tested
• peer reviewed (via pair programming or some other mechanism)
• refactored, readable, duplication-free
• bug-free

Of course, when you first introduce this idea, your definition won't be this comprehensive. Start small (coded, unit tested, peer reviewed, and refactored makes a good start). Every few iterations, if you are successfully meeting your current definition, add something to it. Eventually you will have a pretty comprehensive definition of done, and each time you finish a feature, you'll have a lot less stuff left over to finish "later".

Do you have any other "I'll fix it later" variants to add to my list? Stories about how planning on fixing something later came back to haunt you, or how adhering to a definition of done saved a lot of potential pain? When is it ok to "fix it later"? Where's the fine line between LeBlanc's Law and YAGNI? Please share your thoughts in the comments section!

Traffic Jams and Software Development

I was sitting in construction-induced gridlock a few days ago when I had an epiphany. In retrospect, I can't believe I made the same mistake that everyone else makes. I can't believe I considered traffic jams a bad thing. Traffic jams are awesome!

On the surface, this isn't obvious. But once you begin applying best practices of software project management, it becomes clear that a traffic jam is a sign of a healthy freeway. Let me explain.

First of all, I noticed that the freeway was fully utilized, a situation I have rarely seen in my driving history. As anyone who knows anything about cost accounting can tell you, when your resources are not fully utilized, the cost per unit processed goes up. It really seems irresponsible of the government to build as many freeways as they do, considering that most of the time they are underutilized. What a waste of taxpayer money! Why build a 4-lane highway that generally only ends up half full of cars? They could have built a 2-lane highway instead and saved half the money! (The traffic jam I was in when I realized this occurred when 2 lanes were closed for construction; luckily I didn't miss this opportunity for insight.)

Not only that, I also realized all the drivers were working really hard. I could tell we were working hard because we had been there so long. We all know how important it is to have a hard-working team, and obviously time spent at work is the best measurement of how hard we're working. (Otherwise, why would managers be insisting on 80-hour work weeks at crunch time?) Knowing this, it was really uplifting to realize I was surrounded by such a committed commute-force.

If only I was surrounded with such committed individuals at work, and we managed to keep all the resources fully utilized, I'm sure we'd be much more successful. It's a shame.

There's just one thing bothering me. I still can't figure out why I used to find traffic jams so much more frustrating than software development. Strange.

A Source of Conflict During Agile Adoption

Many organizations try to adopt agile development by plugging in a set of engineering practices without changing the organization's management style. Fortunately, depending on your starting point, some big gains can be made by adopting more effective engineering practices like test-driven development and continuous integration. Unfortunately, to make any big leaps in productivity and predictability, you also need to change the way you manage projects.

By only implementing low-level changes, and avoiding changes that have broader impact, organizations generally don't see the improvements they are hoping for (although they often see just enough improvement to become complacent with their new half-agile method of development).

This is a specific example that illustrates a broader point: there are two ways of solving any given problem:

1. Work more effectively withing your current context.
2. Change your context.
   

As in the above example, you can see some gains by working more effectively within your context, but to have major breakthroughs you usually have to change your work context.

Here's the killer: changing your work context in a large organization can be extremely, extremely challenging. When it works, it often takes a lot of time and effort to get the ball rolling. Because of the difficulty in bringing about context change, many people give up on trying to do it at all. They think that it isn't worth the effort, or almost forget it's even an option. (If you've heard the words "that won't work here" or "that's corporate America, learn to live with it", you probably know what I'm talking about.) Truth be told, often they are right. However, that doesn't stop those of us who believe strongly there is a better way of doing things. Sometimes, it is worth it.

I recently realized that this conflict of approaches to problem-solving is the underlying cause of a lot of conflict between agile evangelists and others on a team struggling towards agile development. Here's how it usually happens:

The team is faced with some particular obstacle, and gets together to figure out how to proceed. Let's say the team has to jump through some bureaucratic hoops to deploy a new version of the software to production. As a result, releases incur a lot of overhead, making frequent releases impractical. A few people make suggestions that minimize amount of time or effort it takes to jump through the hoops. The team begins discussing the benefits and drawbacks of each. An agile evangelist realizes the problem isn't an essential one, but one caused by the current work context. Frustrated with the short-sightedness of his teammates, he expresses his dissatisfaction with all the suggestions so far, and asks "Why do we have to do this paperwork every time we release anyway?", and tries steering the conversation towards strategies to change this aspect of the work context. Less idealistic members of the team realize this won't solve the problem immediately (if it's possible at all), and reiterate their own suggestions. A heated argument ensues.

The problem is neither side feels listened to. They are attacking different aspects of the problem, so there is no actual conflict between them. Both solutions could be applied. Knowing that it will probably take time to change the context, there is probably some benefit to working more effectively within the context in the meantime. Similarly, even if you can reduce the inconvenience of working within the current context, if it is possible to eliminate the inconvenience entirely by changing the context, it's probably worth trying to do that too.

But, because each side is focused so completely on their view of the problem, they don't acknowledge the other's suggestion as useful. Fortunately, I think this is one of those problems that is half-way solved once you're aware of it. Remember that you are focusing on a different way of solving the problem than someone else, and that they may both have benefit. Try to remain aware of how you share your perspective (and for that matter, focus on sharing your perspective, rather than making your point). Practice saying "Yes, and..." instead of "Yeah, but...".

Often agile evangelists and other change agents shoot themselves in the foot by letting their enthusiasm lead the way. This will leave others feeling unheard, which sets up conflict and resistance. Remember, organizational change is an emotional domain, not necessarily a rational one. Before you can understand and deal with others' emotional reactions, you have to understand how your emotions drive your behavior, and how this contributes to the reactions of others.

Work With the Customer, Not For the Customer

If you think your job is to do what the customer* tells you to do and build what the customer tells you to build, you are at risk of creating this:

We as software professionals have a responsibility to our customers that goes beyond giving them total control. We are responsible not just for writing code; we are responsible for helping to create a useful product.

A few days ago, I was listeneding to an IT Conversations interview with Joel Spolsky. He talks a bit about how XP teams work with the customer role, and makes a few good points:

• "Customers will not invent the great features. They will not come up with the 'big leap' ideas." (Your average music aficionado would never have invented the iPod, for example.)
  
• "A good programmer... will come up with features customers never would have imagined were possible." (due to knowlege of program internals and what's technically possible, combined with understanding of domain and the needs of the user.)
• You will discover more interesting insights when you say "Tell me about your job and how you use [our product]" rather than ask "What features should we do?"
• When you ask that second question, "... you get customers asking you for features that seem like obvious features to ask for but which they're never going to use or care about or need."
  

I think what Joel is getting at is that often the customer on an XP team is someone with operational experience, not product development experience. Product development expertise is needed to create a coherent, useful, and elegant product, but teams new to agile development sometimes neglect to make use of their experience to help the customer do this. Sometimes, we throw out the heavy up-front planning and design, but neglect to replace it with a deep level of collaboration with the customer throughout the development effort. This often results in a product without a coherent vision. Users may not find this product useful, but may not be able to articulate what's wrong with it.

We are all experts at living in homes, but if you designed and supervised the construction of your own home without advice from architects, engineers, and construction experts, you probably wouldn't be very happy with the result. Likewise, if you do not help shape the vision of the product, you'll likely build something your customer and users aren't very happy with (despite it being exactly what they asked for).

Unless your customer has significant product development experience, the expertise of both parties is needed when shaping the vision of the product. (I'm assuming here there is a business analyst, markitect, usability expert, or someone else - hopefully several people - with significant product development experience on the development team, and that the customer is a domain expert with a keen understanding of the problem the product is meant to solve and the people who will be using it.) As my colleague Dmitri Dolguikh recently remarked:

"[Software development] is not about the customer asking you for something and you building it for them. It's about you and the customer working together to figure out what it actually is that they need."

Responsible software development means working with your customer to create the best possible product, not working for your customer and abdicating all responsibility for the product being built.


* Please read this as "Product Owner" if you prefer Scrum to XP.

The Essence of Agile Software Development

I finally got around to reading an excellent article by Alistair Cockburn entitled Are iterations hazardous to your project?. He states the case that often iterations devolve into mere planning windows, without resulting in valuable feedback from customers on running, tested features.

Feedback is the key to agile software development. Agile techniques (from pair programming to TDD to tracking velocity and generating release and iteration burndown charts) work because they provide rapid feedback. The more feedback, and the sooner we get it, the better. Mike Bria put it best in the XP mailing list:

"feedback... is to agile as water is to ice"

Feedback is not just an element of agile software development; it is the very essence of it. Keep this in mind and you'll be able to steer your way out of almost any pitfall in your adoption of agile techniques.

However, as my good friend Daniel De Aguiar is fond of pointing out, feedback by itself is not sufficient. We need to analyze the information we're getting, and act on it, to make feedback useful. In response to a sneak preview of this post, Danny replied:

"Opportunities for feedback analysis throughout the iteration are critical to success. Otherwise, the feedback is just noise."

I agree. Many teams are getting feedback through some of the practices mentioned above but don't squeeze enough benefit out of it. For example, often teams will diligently create burndown charts but ignore the red flags that such charts are meant to expose. If you're going to gather feedback, make use of it!

You can read Danny's thoughts on the matter here.

The Hidden Cost of Delaying Bug Fixes

I recently completed Certified Scrum Master training with Mishkin Berteig. One of the many interesting tidbits that came up over the course of the weekend was Mishkin's preference for treating every bug as the team's #1 priority as soon as it is discovered. (This is similar to the stop-the-line mentality of lean development.) I found this interesting, because it goes against the Scrum practice of putting decisions about what the team should work on in the hands of the Product Owner.

I also disagree with giving the Product Owner total control of what to work on, because I believe developers are in a better position to appreciate the real impact of bugs on the product, and it's best to put decisions in the hands of the people best able to understand the consequences. If the development team thinks a bug needs to be fixed pronto, it should be fixed pronto.

Unfortunately, even amongst development teams, there is not always consensus about the real cost of letting bugs linger. Many developers don't mind living with bugs while they continue to pump out new features, figuring they will get back to them "later". In the hopes of changing this situation, I'd like to talk about a hidden cost of leaving bugs in the code.

The Hidden Cost

Of course, there is the standard argument that the later a bug is fixed, the more expensive it is to fix. I'm going to assume that everyone has heard that argument before, and more-or-less agrees with it. I want to talk about something a little more insidious. When you fix a bug not only affects the cost of fixing it, but also the way you fix it.

As development work continues in the presence of a known bug, new dependencies on the code where the bug lives are likely to be added. Sometimes these dependencies will be on bug itself. Often, this happens without the developers realizing it. In general, as a piece of code acquires dependencies, it becomes riskier to change. I think this is even more true of bug-ridden code.

So, the longer you wait before fixing the bug, the more risk is introduced in fixing it. When the risk of fixing a bug increases, the reaction of most developers is to stick with the potential fix that appears least risky (even if there is a 'better' alternative).

A Brief Aside

Okay, I've used a dangerous word: 'better'. Anytime someone uses evaluative words like 'good', 'bad', 'better', or 'worse' without qualifying them with a context, I get suspicious. So here's my context:

Kent Beck makes a distinction between code quality and code health in this talk. I think this is an important distinction to make. Code quality relates to the number of bugs in your code, and how well it does what it's supposed to do. Code health refers to your code's ability to react to change. It relates to how well-factored your code it. You can just call it maintainability if you want.

The reason this distinction is important is that its possible to improve code quality without improving code health. In fact, it's quite easy to improve code quality while reducing code health. Since code health is extremely desirable, I would assert that a fix that improves code quality and code health is better than one that improves code quality but reduces code health.

Back to Our Regularly Scheduled Program

Now we're getting down to the point. When you put off a bug fix, you are going to feel pressure to go with the (perceived) less risky solution later (the quick hack that fixes the bug but reduces code health). And lets face it, most of us are not good with dealing with pressure. If we were, we would probably insist on fixing the bug now rather than later in the first place.

So not only are you going to take longer to fix the bug later, you're going to take on technical debt, and all the pain and suffering (and reduced velocity) that technical debt entails.

Let the Experts Make the Decisions

In many traditional organizations, developers end up making business decisions that they have no right making. Maybe they are not able to get timely feedback from a business expert. Maybe they are so 'sure' of the answer they just don't bother asking. Maybe they are so knee deep in technical details they don't realize they are making a business decision. If they make the wrong choice, this can have some pretty nasty consequences for the business in question.

As a result, agile practitioners have a strong focus on putting business decisions in the hands of business people. The business expert (product owner in Scrum lingo, customer in XP lingo):

• prioritizes features
• provides feedback on working software on a regular, frequent basis
• defines acceptance criteria for features
• remains available to work with developers and answer questions to ensure the features under development are meeting business needs.

These are all excellent practices. They give the business expert the opportunity to steer the project by making all the small business decisions that crop up in the course of development. This is the best way to maximize the business value of the project.

Too much of a good thing

Sometimes, I see an agile team give the business expert too much control. I know, that sounds like heresy. Too much control in the hands of the customer? Is there such a thing? I think there is. Hear me out.

The reason agile practitioners put business decisions in the hands of business people is pretty obvious: they are the experts. They have the deepest understanding of the business needs. They are best able to understand the relevant consequences of the decision. As a result, they are in the best position to make the call.

For some decisions, though, the developers are the ones best able to understand the relevant consequences. I am not talking about most "technical" decisions. If you can easily translate the technical consequences to business consequences, it's not really a technical decision. When that's the case, explain the consequences in business terms, and let the business expert decide.

So what am I talking about?

I'm talking about decisions regarding software development process and practices. Consequences of these decisions are not clear, obvious, and easily comparable. They are subtle and complex. They require experience and expertise to understand. These decisions should not be left in the hands of a novice, and when it comes to software development, chances are your business expert is a novice.

Consider this exchange:

Bob the Developer: "Hi Jim, I wanted to ask you something. I'm working on feature x, and I've noticed that it would really help if we refactored the code we wrote when we implemented feature y. It's kinda becoming a mess."

Jim the Business Expert (looking confused): "I've heard you guys talk about refactoring before, but I'm not really sure what you mean... do you mean you'll have to go back and fix the code for feature y? I thought we were done with that one."

Bob: "Well, the feature is working fine, but we took some shortcuts in the code. It's making it hard to work in that part of the code now. It's going to be a real problem as we move forward."

Jim: "Hmm. How long will it take?"

Bob: "It'll take about half a day to refactor the existing code and add the new feature."

Jim: "Can you add the new feature without doing this refactoring?"

Bob: "Hmmm. Well, I'd rather not, but I think we could hack it in. Doing it that way would take about an hour."

Jim: "Well... we have a lot of features to work on, and marketing is expecting the first release by June. I think you should do it the quick way. I'd rather not waste a few hours on refactoring if it's not really necessary."

Whether or not to refactor as you develop a new feature is not a decision that should be posed to a business expert. It takes a fair amount of software development experience and expertise to understand the effects of refactoring vs. not refactoring. As developers, we are the experts when it comes to software development process and practices. We should recognize that, and act accordingly.

I like how James Shore puts it:

"If a technical option simply isn't appropriate, don't mention it, or mention your decision in passing as part of the cost of doing business."

Letting the customer steer the project is great, but don't give them the opportunity to steer it into the ground by making inappropriate technical decisions. Stand firm when it comes to responsible development practices. Use your best judgement. Remember, you're the expert.

Trust vs. Camaraderie

I was watching the movie Goodfellas a few nights ago, and a scene towards the end got me thinking. In the scene, Ray Liotta's character Henry is on bail after being busted by federal agents. He's having lunch with Robert De Niro's character Jimmy. Henry is trying to figure out if Jimmy's going to have him killed, and Jimmy is wondering if Henry is going to turn him over to the feds. Meanwhile, by all outward appearances, they're having a perfectly normal, friendly conversation.

Obviously this is an extreme example of the difference between camaraderie and trust. In particular, it shows that a relationship can have a high degree of camaraderie and a low degree of trust. Why is this important? Because teams that have camaraderie but not trust often don't realize they have a problem.

Why do we care about trust?

Trust is a wonderful thing. When a team has a high level of trust, they don't waste time with political maneuvering. They don't hold back suggestions that can help the team improve. Team members aren't afraid to ask for help, and are willing to help each other. They spend time collaborating to create solutions rather than arguing over details. When a team has a high level of trust, goals of individuals are aligned with goals of the team. There are no distractions. Productivity can soar.

When a team doesn't have trust, it's not a team at all. It's a group of individuals, each with seperate and often conflicting goals. This wastes time and reduces productivity.

Why do so many teams have camaraderie but not trust?

Building camaraderie is easy. Socializing is easier for some than for others, but if you put a bunch of people in a room together all day, day after day, soon enough they will be comfortable chatting and joking around. Building camaraderie does not require commitment.

Building trust does. It requires commitment both from individuals, and from the organization. Individuals must remember to give their co-workers the benefit of the doubt, always. Beware of these detrimental trains of thought (in yourself, not your co-workers). In particular point number 2:

In a study conducted by Sukhwinder Shergill and colleagues at University College London, pairs of volunteers were connected to a device that allowed each of them to exert pressure on the other volunteer’s fingers. The researcher began by exerting a fixed amount of pressure on the first volunteer’s finger. The first volunteer was then asked to exert the same amount of pressure on the second volunteer’s finger. The second volunteer was then asked to exert the same amount of pressure on the first volunteer’s finger, and so on. Although volunteers tried to respond with equal force, they typically responded with about 40 percent more force than they had just experienced. Each time a volunteer was touched, he touched back harder, which led the other volunteer to touch back even harder. Is this why parties in a conflict invariably think they are both "right"?

Remember, your perception colors everything you experience. If you only give someone as much benefit of the doubt as you think they're giving you, you'll end up in a downward spiral. Trust begets trust; lack of trust begets lack of trust. So trust people more than you think they deserve. Usually you won't be disappointed.

The hard part

As I mentioned, building trust also requires commitment from the organization. A culture of trust cannot co-exist with a culture of blame. People can only trust each other when they are not worried about who is trying to shift future potential blame to them, and who they can shift it to.

Unfortunately, a culture of blame is the standard in most large organizations. When something goes wrong, the first question in such organizations is "Who screwed up?". This is not a useful question, because discovering the answer doesn't help us understand why the problem happened, and how we can prevent it in the future. It doesn't help the organization improve*.

A better question is "What can we learn from this mistake?". This is a useful question, because discovering the answer practically forces the enterprise to improve. It makes us think about the aspects of the system that allowed the problem to occur, and what changes we can make to make it harder for the problem to occur again. It treats a mistake as a learning oppertunity. More importantly, by removing the emphasis on blame, it makes a culture of trust possible.



* Not only that, it's also a trick question, because it implies there is an individual source of the problem. Most problems in a complex enterprise do not have an individual source. Assuming they do is useless at best, and more often downright harmful.

On Stevey on Agile

The blogosphere is abuzz about Steve Yegge's condemnation of what he calls 'Bad Agile' (that is, agile anywhere except Google). Everyone has been contributing their two cents worth, and there have been some interesting responses. The best one I've seen so far comes from Jeff Atwood over at Coding Horror:

"The whole FogCreek/Google-Is-So-Totally-Awesome thing is an annoyance. But I have a deeper problem with this post. I think Steve's criticisms of agile are hysterical and misplaced; attacking Agile is a waste of time because most developers haven't even gotten there yet! The real enemy isn't Agile, it's Waterfall and Big Design Up Front. Even "bad" Agile is a huge quality of life improvement for developers stuck in the dark ages of BDUF. I know because I've been there."

This comes very close to hitting the focal point of this whole debate, I think. I've noticed a pattern in the high-profile bloggers/developers who've taken a very poor view of Agile: it seems that someone has tried to push a particular Agile methodology on them even though they had no pain for it to solve. It seems to me that Joel and Steve already have their own versions of agile development, and they are doing quite well with them. They didn't need XP, Scrum, or any other Agile methodology. So, naturally, they can't help but think Agile is a load of crap designed to make money for consultants.

Someone on the XP mailing list suggested that Steve just "doesn't get it". I disagree. I think he gets it. In fact, he gets it so well that he doesn't need our help. He's already agile. And that's great, even if he doesn't want to fly the (capital-A) Agile flag.

There are really two problems:

1) There are people in the Agile community so excited about their chosen methodology or approach that they start treating it like a Golden Hammer. This is not agile.

Not everyone who isn't doing Scrum/XP/Crystal/whatever is doing BDUF/Waterfall. There are lots of folks out there who are working in a agile way, but not practicing any given 'methodology'. For example, I would categorize the Eclipse Foundation as agile, but they don't use any spefic methodology; they use a mix of practices that works for them. If they are happy and successful, why push XP (or any other methodology) on them? Even if you sincerely believe XP will help them, pushing it on them is not only condescending, it's not agile!

2) Steve (and other people like him who are extremely effective with their natural agile approach) rails so strongly against Agile that he will scare everyone away from it, whether it would be useful to them or not. That's a shame. This is the point Jeff at Coding Horror makes, and I couldn't agree more.

Scrum Pitfall I: Focusing on People Instead of Features

We have been using Scrum since I joined the team at the end of January. Over the course of the last 8 months, we have started using several XP practices. It took a long time, but we've finally gotten to the point of using a story card wall. Well, at this point it's actually a task card wall, but it's a start :). (For those not familiar with Extreme Programming, or XP, a story card wall is a big wall used to visibly track progress of user stories throughout an iteration. User stories are written on index cards, which are placed on the wall. The location of each card on the wall provides information about its status). It's caused an almost unbelievable improvement in our communication, productivity, and visibility into problems.

Why was there such a big difference? I'll get to that in a minute, but first, a little intro to Scrum:

For those not familiar with Scrum, the daily scrum is stand-up meeting where each participant answers three questions:

1) What did you do yesterday?
2) What will you do today?
3) What is blocking progress?

It is intended to allow the team to work together to remove obstacles blocking progress and keep momentum during the iteration.

The burndown is a graph of the amount of estimated work remaining over the course of an iteration. The days of the iteration are written on the x axis, and the amount of remaining work is measured on the y axis. Each day, a new data point is added to the graph. The amount of work remaining is calculated from an accompanying list of tasks and time estimates. The time estimate for each task is updated daily.

We were getting very little value from these two tools. But why?

As I suspect is the case in many large organizations adopting Scrum for the first time, our Scrum had degenerated into a status report rather than a tool to resolve problems and increase productivity. We often forgot to mention what we would be working on in the upcoming day, and hardly anyone ever admitted to being blocked. Most of the time, we just reported what we had been working on.

As for the burndown, we had developed a habit of ending every iteration with only slight slippage from the 'ideal burndown'. Here's an example of what our burndown graphs almost always looked like:
The blue line is an ideal burndown. Notice it reaches zero at the end of the iteration. The red line is our typical burndown. Doesn't look too bad, right? The problem was, even though there weren't many hours left on the burndown at the end of each iteration, there would be lots of incomplete tasks. We would always have a lot of tasks "almost done" and very few actually done.

There were three main deficiencies in our current process that caused this problem:

1) The burndown was deceiving. It didn't show us how many tasks were currently completed and incomplete. To find that out you had to squint at the task list. As a result we were often suprised at the end of the iteration.

2) We had no way to prioritize tasks on the task list. Developers simply claimed a new task at random when they were available. As a result, often we would finish an iteration with important tasks incomplete but less important tasks completed.

3) In the daily scrum, any given developer would usually only mention that he was blocked if he couldn't make any progress at all. He would not mention if a task he was responsible for was blocked, as long as there were other tasks we could work on. Because of this, blockages would often go unresolved.

Here's how we solved these problems: We divided our task wall into three columns: Available, Started, and Completed, so we could tell at a glance how many tasks remained incomplete. We divided the Available column into three sections based on priority: High, Medium, and Low, to keep us from starting low priority tasks when higher priority tasks were available. Finally, we put bright yellow sticky notes on task cards that were blocked. At a glance, we can now tell how many tasks are currently blocked, and can take action if we spot dangerous trends (like most of the available high priority tasks being blocked). The visibility the task wall gives us makes it very easy to realize when we're heading for trouble while we still have plenty of time to do something about it, rather than at the end of the iteration.

This experience reveals a dangerous pitfall that Scrum projects can fall into when not supported by XP practices. The daily scrum iterates focuses on each individual in the team. The burndown shows one view of overall progress. Both make it easy to neglect the actual tasks that need to be completed to make an iteration a success.

I'm eager to hear the opinions and perspectives of others who have used Scrum. Do you support Scrum with XP practices (or vice versa)? Have you encountered this same pitfall? Are there other pitfalls I should keep an eye out for (I've got a few others in mind)?