Four Problems with Master Plans: (1) Precision

Dear Reader,

It is simply not possible to fix today what the environment should be like twenty years from today, and then to steer the piecemeal process of development toward that fixed, imaginary world.

This escapes us when we agree to create a master plan. We forget the simple truth that we're not gods or prophets; predicting the future is never our forte but often our downfall.

Our grand designs fail in four ways.

Today we discuss the first: a master plan is too precise. By predicting the precise shape of the future - the following buildings or software modules will be built in the following ways, and will contribute to the design like so - it leads us into precise collisions with reality when some parts of the plan prove impossible, as always happens.

Murphy's Law isn't very funny to software engineers; it's the reality under which we operate, a reality that's too easy to ignore during planning but impossible to ignore later when our designs come crashing down around all our heads at great expense.

VISTA programming is designed to take advantage of Murphy's Law. It's a highly adaptive process in which we immerse ourselves in the problem with our adopters as guides, in which we do not set out to predict the future, only to solve some specific, immediate problem. A nonprogrammer simply cannot imagine how much failure is involved before one achieves success, but that string of failures is why the VISTA model works and master planning doesn't.

By making planning a cumbersome, expensive process that comes at the beginning, before Murphy's Law has taught us the many ways we're wrong about what's possible, master planning builds a profound ignorance about what's possible into the development process from the start. It saves the exposure of all its failures for the end, when the differences between what's possible and what the plan proposes finally accumulate so greatly that everything collapses in a great crash. Ironically, it's the act of struggling to preserve the plan's success that increases the scale of the failure, since covering up problems and investing more time and money help intensify the inevitable collapse.

By contrast, the VISTA methodology begins making mistakes up front, one at a time, while they're still small - a string of little crashes. We continue making mistakes the whole way, from beginning to end, but they remain small because (1) there is no big, expensive investment in a master plan to try to protect so we can and do change directions repeatedly, (2) the future user of the software is sitting right there the whole time saying "No, we can't do it like that," and (3) instead of struggling to adhere to an increasingly irrelevant plan the team is building up a map of the terrain of the project, a practical guide to many different ways to succeed or fail.

The contrast is stark. Master plans are prophecies. VISTA plans are histories. Master plans are expensive, produced separately from the software. VISTA plans are cheap, produced alongside the software. Master plans are written and finished at the beginning, when we know as little as possible about the problems and its solution. VISTA plans are written during programming and not finished until the end, when we know as much as possible about what didn't work and what did.

The excessive precision of a master plan comes from its focus on the solution, which means trying to pin down the future. VISTA plans avoid that by focusing on understanding the problem and letting the solution emerge gradually and often unexpectedly.

That is, in the master-planning approach we create the illusion of knowing where we're going without first really understanding the nature of the problems we claim to be solving. In the VISTA planning approach, we acknowledge from the start that we have only a vague idea where we're going, only that we know we're going to solve a specific problem by getting to know it very well.

The VISTA approach involves a lot of false starts and backtracking, a lot of mistakes - it looks sloppy and loose compared to the professionalism of a master plan - but that's its strength and the weakness of a master plan.

The VISTA methodology was first advocated by John D. Chase, Jack Brooks, and Ted O'Neill. Here's what Ted O'Neill and Marty Johnson had to say about the difference between the master-planning approach and their own, in their memo to Ken Dickie back on June 10, 1981:

Requirement specifications can be derived effectively and quickly by developing a basic functioning system with the intended user, then further refining the system through user-suggested modifications. The resulting system, in daily use, will provide a much more complete and accurate reflection of user requirements than any narrative description of a system that has not been built. Moreover, by using available modern techniques, such as natural language application generators, database management systems and other software tools, fully operational prototype systems can usually be produced more rapidly than can paper descriptions thereof. Requirement specifications that are produced before any prototype testing activity has been undertaken are now recognized by most computer professionals as inadequate.

Now go back and reread Alexander's quote from the top of this blog entry.

These are two different ways of saying the same thing. They have the same criticism of master plans, independently arrived at from two different perspectives. Our own history bears out the truth of this convergent revelation.

Master plans predict the shape of the future with too much precision, and so they fail.

Yours truly,
Rick

Postscript: Master planning isn't the only kind of planning. The Principle of Diagnosis involves a very different form of up-front planning that helps to create natural order instead of totalitarian order. There are also forms of community planning that focus on studying the problems communities face - the trends, developing problems, and so on - and then draw general, reasonable conclusions to help communities plan. Likewise, there are dynamic forms of project planning designed to accommodate the unpredictable path of research and development. Each of these approaches is the opposite of the master planning approach because they focus on what we can know best - the nature of our present and developing problems - and sketch lightly and prudently in recommending how and when those problems can be solved.