Principle 1: Organic Order: The Problem of Chaotic Order

Dear Reader,

Chaos is not what most people think it is.

To begin with, I'm not using the mathematical term, which sweeps up areas of mathematics outside the traditional disciplines to show there's mathematical order beyond that we usually consider. Although an interesting subject, it's only metaphorically related to true organizational chaos. I'm using the other meaning for chaos: the results of a complete lack of planning, the outcome feared by planners and managers everywhere.

With that in mind, I must next emphasize that chaos isn't the absence of patterns. On the contrary, even fully random processes produce emergent patterns. That kind of patternless chaos people imagine actually takes quite a bit of planning and control to pull off. True chaos always generates two kinds of patterns: accidental and intrinsic.

First, even the chaotic results of random processes produce both statistical ones (like bell curves) and specific ones (like the constellations among the stars) as a normal and expected result of the lack of planning. These macro- and micro-patterns can be used as the seeds for future organization.

Second, when living things are involved (intelligent or otherwise), their needs drive them to produce nonrandom patterns of behavior and organization so they can live and thrive. Firstly, the forces within living things aren't random but instead will lead them to organize their lives in ways that meet their needs - like returning to rivers to drink (or to hunt the drinkers) on a regular schedule. Secondly, living things inhabit ecosystems whose powers and forces reward and punish their behavior, leading living things to adapt in common ways to common forces. For example, even such different life forms as fish, reptiles, and mammals will gradually evolve into the same shapes if they occupy the same ecological niches because the same forces will work upon them over their generations. This is why the tuna, the ichthyosaur, and the dolphin end up closely resembling each other despite very different origins - they converge on the only viable solution to the pressures they're under.

Advocates of imposed order usually underestimate the value of these forms of chaotic order. A great deal of useful order can be achieved without imposition, simply because of people's intrinsic need for order. For example, despite all the screaming and panicking shown in disaster movies, which help to train into us a terror of chaos and of other people (and an artificial craving for imposed leadership that is well understood by juntas when they execute their coups d'etat), during and after disasters people frequently organize themselves to help out family, friends, and neighbors, without any leadership at all - not always, but more often than fearmongers would have us believe.

Advocates of chaotic order, such as anarchists, point to this kind of natural collaboration and contrast it with the inevitable evils of totalitarian order to argue their position that any form of central organization is both unnecessary and unwanted. Anyone who's bothered to study Nazi Germany or Stalinist Russia, or who's thought long and hard about how readily republics loose their democratic bonds to become fascist empires, or who's observed how the disturbing trend toward decreasing personal freedom and privacy changes our culture more and more toward the dystopia of Orwell's novel 1984, has to give this position more than a moment's consideration. There are serious points raised by advocates of chaotic order, and we must do justice to those issues or suffer the consequences of those who fail to learn from history.

Nevertheless, I'm not persuaded, and you shouldn't be either.

There are two excellent reasons why chaotic order is not sufficient.

1) The order produced in chaotic systems is generally offset by the disorder also produced by chaotic systems. Without central coordination, some enclaves will self-organize into more or less peaceful communities, true, but others will devolve into internicine feuds (Internet flame-wars, anyone?), and still others will become predatory. Just as I'll not grant advocates of totalitarian order their insistence on the absence of order within chaos, so I'll also not grant the advocates of chaotic order the absence of destructive consequences.

2) Chaotic order is almost always local, almost never universal. That's enough to solve certain categories of problems but not others, not problems beyond a certain scale. As a VISTA theorist, I am particularly bothered by this reason, because the complete VISTA lifecycle requires that (a) a great number of specialists be given the time and focus needed to master their chosen parts of VISTA, (b) a large number of autonomous teams follow a common culture to ensure that what they produce is compatible, that it creates a living whole, and (c) people not only act with good intentions but also actually do the right thing, since lives and privacy are at stake. These things require more than local or temporary order. These requirements unfortunately fall into chaotic order's demonstrated weaknesses, making it a poor choice of approaches for developing VISTA.

But beyond two there is an even better third reason to reject it for VISTA: it is predicated on a false dilemma, namely that one must choose between totalitarian order and chaos, or some compromise between them. As Alexander argues and I'll explore in this weblog, natural order represents a superior alternative to either of these dysfunctional approaches.

That is, since we already have a model for a system of organic order for managing VISTA development, we don't need to pay the price of chaotic order's weaknesses in order to gain its benefits.

Nevertheless, we're going to return repeatedly to the subject of chaotic order, because in management terms (and now for the punchline) chaos is just a pejorative term for freedom, reflecting a deep unacknowledged terror in all of us. Until we examine our ambivalent relationship to freedom, any cost-benefit analysis of chaotic order is at best highly suspect.

Yours truly,
Rick

Interlude: Ordering The Oregon Experiment

Dear Reader,

You can order a copy of The Oregon Experiment from your friendly neighborhood independent bookstore, or from any number of online book services, including:

A Libris
Abe Books
Amazon
Powell's

Yours truly,
Rick

Principle 1: Organic Order, part 1: The Three Kinds of Order

Dear Reader,

Alexander diagnoses a nearly universal illness in our systems of architectural planning: we use master plans to try to guide our development, to plan for the future:

. . . master plans, are intended to coordinate the many hundreds of otherwise independent acts of building . . . to make sure . . . that the many acts of building in a community will together gradually help to create a whole, instead of merely making up an aggregation of unrelated parts, a chaos.

. . . the master plan, as currently conceived, cannot create a whole. It can create a totality, but not a whole. It can create totalitarian order, but not organic order . . . although the task of making sure that individual acts of building cooperate to form a whole is real, the conventional master plan - based on a map of the future - cannot possibly perform this task . . . because it is too rigid to do so - and because, in addition, it creates an entirely new set of other problems, more devastating in human terms than the chaos it is meant to govern.

It's not enough to replace chaos with order. Order can be worse than chaos.

The problem with chaos is that the "whole" (really just a collection) is less than the sum of the parts. The entire reason for planning is to do better than this, but master plans only give us chaos's opposite: totalitarian order, when the parts are designed to support the whole. The problem with totalitarian order is that the parts don't adequately support themselves or each other, only the whole, and the whole doesn't support the parts, only itself.

Seen with a clear understanding of what we really want in the world, chaos and totalitarian order strongly resemble each other, because they both lack what we most need. What we're striving for isn't a compromise between order and chaos; it's nowhere on that spectrum, because that spectrum doesn't take into account what really matters.

What we need is what Alexander calls "natural order" or "organic order":

We define organic order as the kind of order that is achieved when there is a perfect balance between the needs of the parts, and the needs of the whole.

In an organic environment, every place is unique, and the different places also cooperate, with no parts left over, to create a global whole - a whole which can be identified by everyone who is a part of it.

This kind of order is called natural because it's the kind of order nearly universally created by natural processes, and only rarely by artificial ones. Living processes result in parts that are fully, distinctively, amazingly themselves and yet are equally completely part of and contributing to a greater whole.

The balance between the needs of the parts and the needs of the whole allows the parts to be simultaneously highly specialized, optimized to do their jobs as well as they can be done, while simultaneously helping to create a healthy whole that couldn't exist without those parts.

Consider the relationship between the parts and the whole in the human body. A stomach and a brain are astonishingly different, each highly adapted to its job - if you had to know how to digest an apple you'd starve to death, and we all know what happens to people who try to think with their stomachs - yet each organ fully contributes to the success of the whole. You need them both to live. They're indispensably part of a whole, yet wildly different from each other.

Or consider an ecosystem. A cheetah is an amazing creature. Watch a film of a cheetah chasing down prey and you can hardly believe what you're seeing is possible. It gives every appearance of being a completely original, independent design, a master at what it does. And yet, it is simultaneously utterly dependent on its ecosystem for its survival. Without enough habitat and prey, the cheetah cannot survive. Likewise, without the cheetah, its prey overpopulate and strip bear the vegetation, causing ecological collapse and the widespread death of its prey.

In both examples, the relationship among the parts and between the parts and the whole are nothing like those relationships in a chaos or a totalitarian order.

The parts in a collection (chaotic order) are unrelated, random things. They rarely work as a whole, and never for long; they work against one another at least as often as they work together.

The parts in a totality (totalitarian order) are cogs in a great machine, all resembling each other because they're designed by the same designers. They break down readily because they do not support themselves or each other, only the whole. Place a living thing within such an order and try to use it as a part and it soon perishes because nothing in a totality fully supports the needs of the part.

The parts in a whole (organic order) look very different from one another, each highly adapted to its situation in order to be great at what it does, but if you study them long enough you begin to see the subtle common principles of their designs and to understand the ways in which they're all dependent on one another and contribute to a common whole that in turn strengthens each of them as parts.

Put another way, in organic order each part is also a whole, identifiable and supported by its own parts, by the other wholes around it, and by the whole of which it is just a part. Conversely, each whole stands for more than just itself; it also contributes to the health of its own parts, to the other wholes around it, and to the greater whole they make up together.

This quality is what makes living systems so much more scalable than artificial ones. Each scale of organic organization supports not only itself but also the greater and lesser scales. This is why as totalitarian structures get larger they get weaker - the increasing demands of the whole parasitically sap the strength of the parts - but as organic ones get larger they get stronger - the existing whole and its parts strengthen each new part added to it, and each new part in turn strengthens the other parts and the whole.

This is why the cosmos creates wholes and parts that are so vastly more complex than anything made by man, and yet those systems of wholes and parts last for millions or billions of years instead of the mere decades and centuries we manage. Chaotic order and totalitarian order may be sufficient for the trivial levels of complexity and brief time frames people usually work in, but if you need to create something vastly more complex to last for much longer, then only organic order will do.

VISTA is at that scale of complexity and longevity. It only exists because it was created using living processes that resulted in natural order, and it's precisely because VA turned away from those processes toward more totalitarian ones that it lost the ability to effectively manage or develop VISTA.

Organic order is the grail. This is what you want, and nothing less. It's a key to everything in VISTA. It's what Alexander in later works refers to as a living system, what he uses in his new definition of life in The Nature of Order. Understanding what organic order is, learning to detect its presence or absence, figuring out why it matters so much and how to create it - these are the central challenges in becoming a great VISTA manager.

Yours truly,
Rick

Postscript: Throughout this exploration, I refer to the principal author as a shorthand for the team of authors who developed these ideas and wrote this book. To set aside expediency for a moment and give credit where credit's due, here's the full bibliographical entry on this book:

The Oregon Experiment. Christopher Alexander, Murray Silverstein, Shlomo Angel, Sara Ishikawa, and Denny Abrams. New York: Oxford University Press, 1975. ISBN 0-19-501824-9. Volume 3 in a series from The Center for Environmental Structure.

Our Story So Far . . .

Dear Reader,

There's an emerging order to this weblog. We began with a little introduction to VISTA before turning to the long project of investigating the VISTA software lifecycle. The first four chapters of that investigation are related in a flow of their own.

First, we started by breaking up the ground, by challenging the most common misunderstandings and filters that prevent people from being able to understand the lifecycle even when they're looking right at it. This was the chapter on the eight essential points.

Second, now, we're taking our first steps into understanding the lifecycle by using something similar that's already been well explained as a frame of reference. That's this new chapter on the principles of organic development as explained in The Oregon Experiment.

Third and fourth, after establishing that point of reference we'll drive right to the heart of the VISTA software lifecycle by examining its two essential characteristics that together set up the pattern that makes sense out of every detail of its architecture or policy: flow (fluxus quo) and life (living systems).

Thereafter, in subsequent chapters, with the main misunderstandings dealt with, a common frame of reference to compare and contrast with, and the two essential characteristics fully described, it should be easy enough to work our way through the details of how to manage VISTA effectively.

Like all plans, I'm sure this won't survive much contact with the subject matter, but let's find out.

Yours truly,
Rick

Crickets

Dear Reader,

Are you there? Do you care? Am I making sense or losing your understanding or interest? Question me. Challenge me. Criticize me. I want to do a better job, but my perspective on myself is limited; I need help finding ways to make this better. I am but a man, with a man's many flaws, trying to carry this particular load for the community.

The ideas I'm trying to capture and express are a crucial part of the theoretical framework for VISTA, the part that says how to develop it. If these ideas were better understood and accepted, the evils VISTA has suffered and that veterans have suffered as a result could not have occurred without immediate public outcry. The lack of understanding of this lifecycle created the vulnerability that has been exploited for the last fifteen years. If we want to put a stop to this kind of attack on our patients, we need this book. We've needed it for a long time.

We need this book to be as good a book as we can make it, to be clear and compelling. It will be much better if we engage in a dialog now, while we can work out the ideas in electronic format. Once it goes into print, it's too late to fix the ideas or improve the explanations.

I'll make you a deal.

If you help me find the flaws in my arguments and improve the ways we explain this lifecycle, I will credit you for those improvements explicitly in the print version of the book. Public credit is as much a VISTA tradition as friendly public criticism.

Yours truly,
Rick

Principles of Organic Development

Dear Reader,

Although The Oregon Experiment is the best written description so far of the VISTA software lifecycle, it doesn't describe it perfectly.

Why would it? VISTA's lifecycle unfolded according to its own path of development, in response to its own internal and external pressures, many of which emerged after this book was written. This book's such a good way to begin learning that lifecycle because it so clearly describes such a good example of an organic development-process.

Living systems, like VISTA and like physical architecture, can't be adequately managed by mechanical development-processes.

They don't need to be. There's an alternative.

Instead of mechanical development-processes, we can adopt organic development-processes, which mimic the processes living system use to produce new living systems, instead of the processes people use to produce machines. VISTA's history (including its current problems) proves we must replace VA's current mechanical VISTA-planning processes with its original (or improved) organic ones, but doing that will require understanding organic development-processes at least as well as we currently think we understand the mechanical ones.

That's where Alexander's book comes in. Alexander proposes six principles that together create an organic process for architectural planning. If you learn how these six principles guide this architectural lifecycle, which is clearly and cogently explained in The Oregon Experiment, then you're halfway to understanding VISTA's software lifecycle. Discussing this book isn't a diversion; it's a shortcut.

The similarities to the VISTA software lifecycle started by O'Neill and the original hardhats and developed over time within VA and IHS are eerie, but the differences are equally important and illuminating. We'll examine both as we explore Alexander's six principles of organic development.

There are two particular problems with this model I'll highlight throughout the exploration that follows.

First, as Alexander makes very clear, these principles aren't the ideal suite to guide organic development-processes. They're a compromise designed for overly centralized situations in which budgetary authority is and must remain centralized. This wasn't, needn't, and shouldn't be the case in VA, where local hospitals can, did, and must control part of the IT budget so they can share in the decision making. The current totalitarian degree of centralization in VA's management and funding of VISTA is a grotesque organizational illness that new VA CIO Roger Baker needs to cure for the continuing health of the veterans whose care is now his charge. The importance of class-three software in VA (a subject of its own we'll thoroughly explore in the weeks ahead) demands a return to a balance between central office and the hospitals. That healthier balance also represents a shift away from Alexander's compromise in this book back toward the purer model he envisioned but didn't document here.

Second, software and buildings share a surprising amount in common, but their differences are also important. Among other things, software's far more malleable than buildings are, so the pace of software change in a healthy VISTA organization is one or more orders of magnitude faster than the pace of construction change in a neighborhood can ever be. This creates both solutions and problems Alexander didn't have to address in The Oregon Experiment.

We'll explore his six principles of organic development with these and other differences in mind.

Yours truly,
Rick

The Oregon Experiment: Six Principles

Dear Reader,

The Oregon Experiment is a quick read worth reading more slowly. Since this book is so important to understanding the VISTA software lifecycle, let's examine what it can teach us about VISTA.

Here are the book's six principles:

Specifically, we believe that the process of building and planning in a community will create an environment which meets human needs only if it follows six principles of implementation:

1. The principle of organic order.
2. The principle of participation.
3. The principle of piecemeal growth.
4. The principle of patterns.
5. The principle of diagnosis.
6. The principle of coordination.

We recommend that the University of Oregon, and any other institution or community which has a single owner, and a centralized budget, adopt these six principles to replace its conventional master planning and conventional budgetary procedures, to provide the administrative resources which will guarantee people the right to design their own places, and to set in motion the democratic processes which will ensure their flexible continuation.

For the sake of concreteness, and to give you an overview of the book, we now outline these six principles.

1. The principle of organic order.
Planning and construction will be guided by a process which allows the whole to emerge gradually from local acts.

2. The principle of participation.
All decisions about what to build, and how to build it, will be in the hands of the users.

3. The principle of piecemeal growth.
The construction undertaken in each budgetary period will be weighed overwhelmingly towards small projects.

4. The principle of patterns.
All design and construction will be guided by a collection of communally adopted planning principles called patterns.

5. The principle of diagnosis.
The well being of the whole will be protected by an annual diagnosis which explains, in detail, which spaces are alive and which ones dead, at any given moment in the history of the community.

6. The principle of coordination.
Finally, the slow emergence of organic order in the whole will be assured by a funding process which regulates the stream of individual projects put forward by users.

Next, let's examine these principles in detail, beginning with the principle of organic order.

Yours truly,
Rick

Book Recommendation 1: The Oregon Experiment

Dear Reader,

To Roger Baker, the new chief information officer of the U.S. Department of Veterans Affairs, I offer this advice: if you want to be a great VISTA manager, a great VA CIO, I can recommend nothing more important than to read this book. It'll prepare you for your new job in ways no meeting or position paper ever will. It'll tell you things you need to know, things none of your subordinates can tell you. Unless you do this, you'll almost certainly repeat the mistakes of your predecessors despite your best intentions. With it, you create the possibility of becoming the second-greatest manager VISTA has ever known.

Why?

Because in the year 1975, in 190 small pages packed with illustrations, Christopher Alexander and his team came as close to describing the VISTA lifecycle as anyone has yet done in writing. It should be required reading for anyone who intends to manage a VISTA project.

The surprises here are (1) that VISTA didn't exist in 1975, except as a set of observations and principles in the possession of VISTA founder Ted O'Neill, so of course Alexander had never heard of it; and so (2) he and his team weren't writing about VISTA or even software development; they were writing about architecture and architectural planning.

They were trying to describe what seemed to most people like an unusual approach to architectural planning; it was the original approach used by humanity for millennia but abandoned in the nineteenth and twentieth centuries. This is analogous to what Ted O'Neill and his team were setting out to do in the VA: to try out a promising approach to medical-software development that defied mainstream ideas about how to develop software.

Both projects began by studying the past, studying what worked and what didn't, building up lists of tactics for how to succeed or fail, and using those lists to develop a coherent strategy. The two strategies resemble each other closely, but Alexander and his team wrote a book about their strategy, whereas we haven't yet written the book about the VISTA strategy.

Other similarities abound between the two projects. Like physical architecture, medical-software architecture is complex, easy to get wrong, and affects many people who really care about the results, and the costs of failure are paid in time, money, morale, and sometimes human lives.

Most crucially though, both kinds of efforts largely succeed or fail on the effectiveness of their management processes. You can have the best builders of buildings or code in the world, but if you manage them badly you guarantee failure - and it's easy to manage them badly. Indeed, in both disciplines bad management's not an accident but a carefully learned behavior, in which management practices that work well in other situations work badly in these two yet are adhered to forcefully even though they leave behind a track record of failure after failure.

The reason both disciplines are so vulnerable to bad management is that the two subjects share two vital characteristics: (1) flow, and (2) life. Less tersely put, (1) they both have to be managed using organic development-processes, because (2) both physical architecture and VISTA itself are living systems that only superficially resemble mechanical systems.

The full significance of that paragraph will take a few weeks to illuminate adequately, so let's get to it, step by step.

Meanwhile, order yourself a copy and read it carefully.

Yours truly,
Rick

The Essential Point

Dear Reader,

Companion to the essential principle is the essential point.

What's the essential point for understanding the VISTA software lifecycle?

There are many wrong answers to that question and only one right one. The problem is that each of us prefers one or more wrong answers, for reasons we think are really good reasons. Nothing I can say can compare to the attraction of those reasons, so I'm not going to answer this question for you.

Instead I leave you with the exhortation that you have work to do on this problem, that there is no shortcut to answering it, that the answer matters, and that you'll have to work diligently and scrupulously over a long time to peel away the layers of your misunderstanding of this essential point in search of the truth.

That's not a bad thing. Work is good. Work on the right things and that work will transform you into a better person. This work, the work to discover the key to understanding the VISTA lifecycle - the real key, not the one you like - will do that too, and it'll make you a better VISTA manager.

Without this work it's all too easy, inevitable really, to violate the essential principle, as VISTA's history demonstrates over and over.

So I'll leave you with this task - it was yours already - and turn away from the eight essential points for now to a different perspective on the VISTA software lifecycle.

Yours truly,
Rick

The Importance of Project Management

Dear Reader,

Lest there be misunderstanding, allow me to clarify that when I wrote

"To sacrifice medical science on the altar of project management is a form of human sacrifice"

I in no way meant to imply that project management is not needed with VISTA. On the contrary, the effectiveness of the project management seems to correlate directly with the odds of success in the last decade of projects to install VISTA outside the federal government.

High-quality VISTA-project managers seem to be among the rarest of the rare specialties in increasing demand as the expertise bottleneck gets a good grip on the community.

A great deal of what I'll be writing about in the weeks ahead assumes and depends upon a solid project-management foundation for the lifecycle to succeed.

Rather, what I meant is that within the VISTA-software lifecycle, project management must serve medical science, not attempt to dictate terms to it. Just as the approach to software development must be adjusted to be compatible with the complexity and flow of medical science, so too must project management.

To do otherwise, for example to prioritize deadlines over medical correctness, is literally to endanger our patients. There has been a great deal too much worship of timelines in VISTA management over the last fifteen years, and the health of our patients and of the software that supports them has suffered.

Replacing that kind of poor-quality, mechanical project management with experienced, fluent, and well-adapted project management is one of the major steps in launching this new VISTA renaissance.

Yours truly,
Rick

A Culture of Criticism

Dear Reader,

Miss Manners, many blessings upon her, gently reminds us that it is not proper to try to parent other adults (or even children other than your own, prior to their graduation into adulthood), to correct them or criticize them, in public or otherwise.

There are two exceptions.

1) Parents can and should correct their children, preferably gently, by example, and in a way that both helps them develop into good people and also preserves their dignity.

2) Teachers can and should correct their students, much as parents correct their children but not across as broad a range of subjects, more focused upon the subjects under consideration but still including an emphasis on helping them develop into good people and preserving their dignity.

This second case applies to the VISTA community.

According to the seven-part VISTA model, our software lifecycle depends upon another lifecycle, an expertise lifecycle. The VISTA community has to be structured as a perpetual-learning organization, with each community member acting as both student and teacher to the best of their abilities. We have to cultivate our expertise and help each other develop, because we are embarked upon such a difficult mission.

In an educational context, especially one dedicated to scientific and engineering pursuits, we have to be straight with each other about our mistakes, our flaws, our bugs, our opportunities to learn and improve. This can and should be done respectfully, but it must also be done unambiguously, and if the students don't get the message with subtler forms of suggestion, more overt and direct language is called for.

The key to open-source-software development is that we work together to find and fix the problems so we can keep making our software better.

The key to an expertise lifecycle is that we do the same thing to make ourselves and our development processes better.

Even the best hardhat, the most experienced or expert hardhat, the wisest hardhat, is a perpetual student of VISTA, whose mastery will always be exceeded by the things he still has to learn.

To improve the software we must always be straight with each other about the bugs in our code.

To improve the software lifecycle, we must do likewise with our distribution of authority and our management of projects.

So, when I criticize the VA managers for their mismanagement of VISTA, it is in this spirit. I'm not setting myself up in opposition to them, or as superior to them, or on the other team from them. Mostly they're friends of mine, people I know and respect.

And in the same spirit that I critique the flaws in the programming style of my newest students, in the same say I criticize my own code when I'm team programming with my peers, I will call out the flaws in the approach of the VISTA managers. The best of them expect this of me, demand it of me.

And in doing these things, I am not doing anything special, not in the VISTA community. I am just doing my duty as a VISTA hardhat. Any of us would do the same.

In the VISTA community, we're straight with one another about the things that need to be fixed. Anyone who can't handle that must either learn to handle it or else find themselves another line of work. What we do is too important to lie to ourselves about our performance. We have to improve, always.

Our responsibility to VISTA adopters and their patients demands it of us.

Yours truly,
Rick

Humility and Courage

Dear Reader,

VISTA humility requires that when Congress asks us how long something will take, our answer is "I don't know," unless it's something we've done repeatedly and consistently.

For example, we can answer how long it'll take to add a simple new report to a package we know extremely well. Or, if we've already written an interface between the McKesson Series Scheduling package and VISTA and installed it at numerous sites, we're within our rights to report how long it's taken the last five times we installed it and to suggest a similar timeframe.

If, however, we're asked how long it'll take to write an interface between a VISTA package and an external clinical system to keep the two systems in sync, we have to say "I don't know," even if we think we understand the scale of the problem. There's all the difference in the world between visualizing a solution and actually solving the problem that seems to elude most VISTA hardhats and managers alike.

Usually, about the only honest answer to questions about how long a task will take with VISTA is "Longer than either of us thinks." We're guessing about anything we haven't done repeatedly and consistently with VISTA. The moment we peg down a date on a task outside this rather confined circle of experience, we are lying. We have left humility and caution behind in favor of bold hybris, which feels intoxicating, sure, but is fraught with peril.

Why?

Because VISTA is bigger than we think it is. VISTA's ugly facts will sink your beautiful theories. VISTA sneers at your petty delusions of predicting the future. VISTA is filled with surprises.

Because these things are true of medical research, as well. Ask yourself how many times so far the answer to the question "When will we cure cancer?" has been correct. Does the comparison seem unfair? Then you are still underestimating the scale of VISTA. When I told you you were getting it wrong, you thought I was resorting to shallow rhetorical flourishes, but I'm not. From one perspective, VISTA's function is to eventually capture and represent everything we know about medicine in a consistent and logical system of data and programs that can reliably support medical practice. Is the entire field of medicine complex enough to get your attention, to make you even for a moment set aside your genetic predisposition to overconfidence and begin to quail at the scope of it? Imagine trying to sort out and systematize that field to the level of mind-numbingly stringent literalism that a computer requires.

Because medical-software engineering is not as trivial a discipline as building planes that don't crash or rockets that don't blow up on liftoff. They only have to deal with physics and weathering (and human error and politics and economics and the other universal problems). Medicine makes a lot less mathematical sense than physics, and software engineering's much easier to get wrong than materials engineering because of the relative lack of physical constraints, yet computers are vastly less tolerant of sloppiness in specification than human pilots are. Put that all together and you get a messy, difficult, complicated situation.

Because at its best software engineering is more like research and development than like engineering a bridge. The discipline of software engineering, when practiced correctly, is nearly the opposite of the discipline of physical engineering. The omnipotent malleability of software collides with the highly idiosyncratic needs of software users to create vast and shifting uncertainties. Properly practiced, software engineering is designed to be highly adaptive to the ocean of unknowns we have to deal with.

Because nine times out of ten our statement of the problem is wrong. We think we know what we're asking about, but the truth is we've grossly underestimated how much we really need or how many other features are actually involved in what we thought of as a simple, stand-alone feature.

At multiple points in the process of developing anything but the simplest changes, we're going to get checked by reality. Surprises and adjustments creep in. Sometimes it's not until the moment we begin seeing drafts of the "finished" software and realize something's very wrong here, but often the programmers return after just barely beginning the task to report they already need to change the plan because it left out something important. And usually, those two patterns occur over and over (and over and over and over . . .) throughout the development process.

Traditional project management loathes this kind of shifting of the parameters of a task. They call it "feature creep" and blame it on the users and programmers. They believe the way you keep projects on time is to stop feature creep. That's an idiotic position to take, because it leads to delivering the wrong thing on time and actually being proud of that accomplishment. It's an example of how hybris can twist people into the most ridiculous postures, into defending the most indefensible positions.

I believe, practiced properly, feature creep is the key to VISTA's success.

Giving medical practitioners what they actually need rather than the gross caricature we envision at the start of a project is what results in software that doctors and nurses can actually use to improve health care. When you're making a game or a word processor, maybe it's okay to sacrifice features to meet a deadline, but when you're making medical software do you really think it's a clever idea to pressure your programmers to cut corners or to ignore the negative feedback from doctors about early drafts of the software, just so you can meet a date that - let's face it - you had no right to give in the first place, that no matter how you came up with it was in the end a ridiculously arrogant attempt to predict the future?

To sacrifice medical science on the altar of project management is a form of human sacrifice.

There are better ways to manage medical-software projects. There are specialized forms of project management that do not begin with the pretense that R&D can be controlled, that instead harness it like an ecology, like a garden that regularly produces good things on its own schedule so long as it's properly cultivated. In a nutshell, that's precisely the VISTA software lifecycle's approach to project management.

But it all begins with admitting that VISTA development is, in fact, R&D, that it's filled with unknowns and surprises, that, in fact, "we don't know" how long significant projects will take to complete, with only the rarest of exceptions.

So here's the problem with the principle of humility.

Good VISTA-project management begins with recognizing the difference between hybris and humility, a subject that most people think they understand but that the evidence of history soundly proves otherwise. Most people, while engaged in acts of the utmost hybris, feel they're acting in a professional, responsible, and cautious manner.

In the case of VISTA managers, this is often because the political pressure they endure from Congress makes them feel small and powerless, afraid for their careers and for the future of VISTA. It is while in this mindset that the worst acts of hybris take place.

It can be very confusing, so I'll make it clear.

You should be more afraid of VISTA than you are of the President of the United States himself. If POTUS demands you tell him how long it'll take to replace the VISTA Laboratory package with a modern, off-the-shelf or custom-built proprietary solution, humility requires you to refuse to take the bait. Humility requires you to say the following:

"Mr. President, I don't know, and neither does anyone else. Most software projects fail either partly or completely, and big ones fail most of all. Replacing VISTA Laboratory's on the far end of big. Anyone who tells you such an undertaking's likely to succeed let alone when is ignorant or lying and probably both. I advise against it, Mr. President."

If you couldn't give this answer, if it seems arrogant and dangerous, it's because you haven't yet really come to terms with the degree of humility VISTA management demands of you. You're still afraid of the wrong things.

You still think angering powerful people or institutions is the worst thing that could happen to you. You are so wrong. POTUS is merely an extremely powerful man. He can only bring the human world against you if you cross him. (Besides, you never know, the President might prefer people who tell him the truth).

If to avoid disappointing him or any other man you go up against the laws of information science, it's the cosmos itself that'll grind you down. The laws of information science are as implacable as the laws of physics and far less well understood.

You should be humble. You should know your place. The health of patients cared for by the hospitals, clinics, and offices that use VISTA depend upon you to respect and fear the seriousness of your work. They need you to protect them from your hybris, no matter how the mighty incite you to it.

Managing VISTA requires humility; and, in the human world, humility requires courage.

Yours truly,
Rick

The Essential Principle

Dear Reader,

The eight essential points all derive from one essential point and one essential principle.

First, the principle: humility.

Science begins with the admission "I don't know," one of the most powerful sentences in English.

Science is about the recognition that reality matters more than opinion, that when you and the world disagree, you are wrong. Kafka summed it up best:

"In any contest between you and the world, side with the world."
-- Franz Kafka, "Aphorism 52," Unpublished Works 1916-1918

It's obvious, so any sane species would hold their opinions about the world lightly, would attend carefully to every dissonance between their opinions and the truths of the world. Unfortunately, we are not quite sane, because we usually set aside caution where our own prejudices and hopes are concerned.

Science, therefore, must also be about recognizing how easy it is for people to lose track of these deep truths, how easy it is to be convinced by one's own opinions and rhetoric, to be fooled into thinking it's okay turn away from the truth of the world in favor of the seductive quality one's own ideas and prejudices have. Something in mankind finds the feeling of confidence intoxicating:

"The partisan, when he is engaged in a dispute, cares nothing about the rights of the question, but is anxious only to convince his hearers of his own assertions."
-- attributed to Socrates by Plato, Phaedo, tr. Benjamin Jowett

This is the heart of the software-lifecycle problem with VISTA. Managing VISTA's flow is a complex problem, but it's already been solved successfully. Repeatedly. We have a reliable model to follow. And yet we don't. The problem's not technical; it's human.

In Peopleware, my favorite book about how to manage IT projects, authors Tom DeMarco and Timothy Lister make the crucial observation that most people suffer from the illusion that software development is a technical field, but it's not. The technology is conspicuously present, but you can't succeed by focusing on it. It doesn't matter how great a technologist you are if you don't give people what they really need - and you don't know what they need (and they may not know either). At its core, software development is a human-relations field, in which the people who best understand the priorities, the people who best understand the technology, and the people in charge are three different groups of people. Getting them to work together well, to communicate well, and to divide up their responsibilities properly makes or breaks a software project long before the technical issues have a chance to.

VISTA illustrates their claim well, which is why the core principle is not a technical one, why the fundamental principle is one of human character. Without humility, even small VISTA projects are likely to fail.

When the question is the entire VISTA software lifecycle and how to manage it, the sober, realistic assessment is that the scale of the problem is vast and the valid judgment of each individual involved is limited. Willingness to recognize that the scale of the problem exceeds our grasp separates the humble realists from the arrogant dreamers. Humble realists have proven their ability to work miracles with VISTA, just as the arrogant dreamers have proven their ability to squander unimaginable sums of money and to endanger patient care in the pursuit of their grand designs.

Some hard scientists claim that history isn't a science, but by being managed alternately by humble realists and arrogant dreamers in succession, VA has turned its own history into a perfect laboratory that demonstrates over and over decade after decade consistent results that back up DeMarco and Lister's thesis about managing software projects.

So here we are. We have one of those situations all-too-rare in the soft sciences, in which we have a perfect match between theory and evidence. What we ought to do is obvious to anyone with the humility to do justice to the rights of the question by spending the time learning from our history and then letting that evidence guide them.

The only thing stopping us from succeeding with VISTA is our addiction to our own opinions, the rush of the eternal adolescent fantasy that with enough power we could solve all the world's problems, the illusion that the problems we must overcome are outside of ourselves.

All hail the immortal Walt Kelly for his observation "We have met the enemy and he is us." Humility demands that we accept the truth of this. It's the only real explanation for the mess the VISTA community finds itself in today.

Fortunately, it's also the key to getting out of this mess. If we embrace the principle of humility, if we acknowledge the limitations of our expertise, then we can roll up our sleeves and overhaul our allocation of VISTA authority to break up its unhealthy and insane centralization, then we can distribute each kind of VISTA authority into the hands of only those people who hold that kind of expertise.

Humility does not require us to give up, or to assume a posture of helplessness. This is not a jump off a cliff into the unknown. This is a return to the most effective model of management VISTA has ever known, in which authority is decentralized but concentrated according to expertise, reaping all the supposed benefits of centralized VISTA control with none of the all-too-real flaws.

This is why the eight essential points all derive from the principle of humility. If we can muster the courage to admit that no one of us nor any small group of us can possibly know enough to manage something of VISTA's scale of complexity, then all the elements of the VISTA software lifecycle follow as inevitable solutions to that problem.

Yours truly,
Rick

Eight Essential Misunderstandings

Dear Reader,

The eight essential points I have summarized so far aren't the sum total of the things you need to understand about the VISTA lifecycle. They aren't even a comprehensive overview. They don't even represent the majority of the main issues.

They're just the clearest refutations of the eight most common misunderstandings about managing the VISTA software. You can't really understand the VISTA software lifecycle unless you can recognize when you aren't doing it.

So here they are. See how many of these destructive VISTA practices you recognize.

-1. VISTA should be managed with an industry-standard software lifecycle.
-2. Managing VISTA requires a central code repository.
-3. Managing VISTA requires strong, centralized leadership.
-4. Expert project managers must control VISTA's software lifecycle.
-5. User requests should be submitted to a sophisticated review process.
-6. Programmers should work on whatever their managers tell them to do.
-7. Politically important projects should get top priority.
-8. VA is the VISTA producer, everyone else VISTA consumers.

Yours truly,
Rick

Point 8: Declare Interdependence & Confederate

Dear Reader,

Once upon a time, Ted O'Neill and Marty Johnson set into a motion the VISTA software lifecycle, which resulted in continuously improving medical software so popular with the users that the VA and congress reluctantly agreed to set aside their multi-million dollar failed boondoggles and do the right thing. For seventeen years the VISTA lifecycle made VA the main hub of VISTA development worldwide. With VA providing such a reliable heartbeat for that lifecycle, everyone else could afford to remain comparatively disorganized so long as they stayed in a symbiotic relationship with VA and its lifecycle.

The future before us is very different. With VISTA spreading all over the world at an increasing pace and VA's mandate restricted to healthcare for veterans, VA will shift from the center of the worldwide VISTA lifecycle to become a peer. That will require a radically different form of relationship among the various VISTA-interested organizations, one in which the heart of the lifecycle does not exist within a single organization but across multiple, sometimes competing organizations. This form of relationship has a formal name. It is called a confederation. The future of the VISTA community is a worldwide confederation.

The present, caught between that past and this future, demands a time of radical but wise change. Now is the time for the transformation of our community. Now is when we must breach the barriers that separated us into autonomous feudal kingdoms, each with its own dialect of VISTA. Now we must declare our interdependence and forge the formal alliances that will pave the way for confederation, supporting our need to compete and innovate while still collaborating on the shared lifecycle and software upon which the health of our patients depends.

The world has begun to marvel at VISTA, to realize the amazing power it has to help doctors and nurses save lives. But we know they haven't seen anything yet. We know the real power is in the software lifecycle that created VISTA, a lifecycle we are about to unleash again to work its miracles for us. We know the world is in for a wonderful surprise.

Yours truly,
Rick