Status Quo Kills

Dear Reader,

Most health IT projects fail because the software industry’s dominant response to the software crisis precisely collides with the unique demands of health IT.

Ever since Edsger Dijkstra began theorizing about how to solve the software crisis in 1972, computer science has focused understandably on the need to drive down the rate of errors in order to produce correct code. This focus on reducing errors has led to a focus on careful specifications and the careful coding designed to measurably meet those specifications. Under this paradigm, which is taught in all computer science departments across the country and followed by the software industry, success is defined as adherence to a software specification and errors as deviations from that specification. This error-avoidance philosophy is so ubiquitous, it is even implicit in the success and failure critieria used by The Standish Group in its CHAOS reports.

The resulting software-development process drives toward achieving the fixed goal of compliance with the specification, toward achieving a kind of crystalline, static perfection. That is, for the software-development process to avoid errors as much as possible, it has to impose extensive testing on all changes, which raises the cost and time required to make changes, which has the effect of discouraging changes. Indeed, everything about the dominant software paradigm rewards compliance with specifications and punishes attempts to modify the specification, especially in mid-course. It is a specification-driven, stasis-seeking methodology hostile to change. It dominates the software industry, including those companies engaged in producing health IT.

The problem with trying to apply such a model to healthcare is twofold.

First, healthcare is extraordinarily complex, essentially impossible to capture in a specification, which guarantees that any specification will deviate from the actual needs of the users. When faced with a choice between a clear specification and ambiguous statements of need, the specification-driven, stasis-seeking paradigm will always choose the specification over reality.

Second, even if it were possible to capture a perfect specification of the needs of health IT adopters, medicine changes continuously in response to the continuous scientific investigation into health and the resulting continuous stream of new discoveries that impact best practices. Hence, even a correct specification would become increasingly incorrect merely through the passage of time.

Healthcare is thus the worst possible case for the dominant software paradigm: an unspecifiably complex problem domain that continuously changes.

Unfortunately, health IT vendors have two overpowering incentives to disregard this fundamental mismatch of their approach and their subject.

First, the logic of avoiding errors seems irresistable, particularly so when backed by the consensus of an entire industry and its corresponding field of science; when struggling to gain control over out-of-control software projects, the retreat to controlling errors intuitively feels like the right move.

Second, the status quo is very, very profitable, especially because of the mismatch with the needs of healthcare. To help discourage changes to the specification in mid-course, it is standard procedure for IT companies to charge exorbitant premiums for such mid-course spec changes. The mismatch between health IT specifications and real healthcare information needs guarantees that health IT projects will involve a great many changes to the specification, for each of which the health IT vendor will earn a premium. The cost overruns that health IT adopters lament are the profits that keep health IT vendors in business.

In short, when applied to healthcare the dominant specification-driven, stasis-seeking, error-phobic software-development culture kills, but it does so both profitably and untraceably.

Yours truly,
Rick

Postscript: Let no one—no one—infer from this posting that I have anything less than respect and admiration for the great Edsger Dijkstra. My argument is that automating rapidly changing fields like medicine is a special case in which Dijkstra's prescription is not the best solution for addressing Dijkstra's concerns.

The Risks of Health Information Technology

Dear Reader,

Health IT success or failure numbers are very hard to come by, in part because some healthcare IT vendors require their customers to sign contracts banning them from publicly discussing the success or failure of their health IT, in part because the legal stakes are much, much higher with health IT failures which creates a disincentive to be forthcoming about dramatic failures.

Most numbers therefore are anecdotal. For example, the American College of Physician Executives (ACPE) surveyed its thousand members in its 2004 Technology Survey, and the results suggested that even successful health IT projects are often more expensive and less useful than expected, often permanently reducing physician productivity.

Informal surveys of non-management medical professionals at hospitals adopting IT reveal a much lower rate of satisfaction than that reported by those in management positions. The most frequent complaints are (1) a poor fit between the software’s design and the needs of its users, (2) poor responsiveness to requests to improve the software, and (3) hidden and horrendous costs that emerge in IT adoption during the process of trying to get badly fitting software changed to truly meet the needs of its users.

Off the record health IT programmers are often surprisingly candid about how bad their software is and about how resistant health IT company management is to making unprofitable changes that would improve its quality. Producing bad health IT software has been very profitable for a long time.

Health IT is the industry that results when one industry in crisis attempts to assist another industry in crisis, creating a collision of crises. The emergence of an American health IT movement is an important step toward improving health in this country, but unless the true state of health IT is understood and dealt with it will almost certainly be a very painful step indeed.

Yours truly,
Rick

The Risks of Information Technology

Dear Reader,

Unfortunately, missing from all this discussion is an appreciation of the dark side of health IT. Unless the problems with American health IT are as openly discussed, understood, and dealt with, as the problems with American healthcare, mixing IT with healthcare could deepen our national healthcare crisis.

The problems with health IT begin with IT itself. According to The Standish Group’s CHAOS 2004 survey results, only 29% of software projects come in on time, on budget, and with all the features originally planned. 53% of software projects fail at least partially, and 18% fail completely. Although people not familiar with the sorry state of the software industry find these numbers shocking, the reality is even worse.

The Standish Group’s numbers are oriented toward tracking the effectiveness of the project management, not the effectiveness of the resulting software. Any experienced IT user can tell you that many “successful” projects are actually failures from a usefulness perspective, bad ideas successfully imposed upon the organization. Similarly, these numbers do not factor in whether the budget was an effective use of the organization’s resources. Many, many projects are vastly more expensive than they should be for the value delivered. Once these crucial but omitted criteria are imposed on the CHAOS numbers, the numbers fall in line with the reality of an industry in crisis, The Software Crisis.

Since 1968, the software industry has understood itself to be immired in a crisis that defies solution. Forty years of innovation have still not managed to fix the kinds of problems measured by The Standish Group. Indeed, they have only clarified that at the root of the problem is a basic but paradoxical principle of complexity: Success breeds failure. Success with software development invariably increases the amount of software to be managed and taken into account in the next round of development. There is no limit to the potential complexity of software, but human capacity for managing complexity begins to fall apart at surprisingly low levels of complexity.

Successful software lifecycles become less successful over time. No wonder so many software adopters cyclically opt to replace what they have with something new! The temptation to blame the current software and start over with something else helps to fuel a major portion of the healthcare IT industry’s annual revenues. This wheel spinning may be successfully executed as projects and reported as such in the CHAOS reports, but it constitutes ongoing waste and failure disguised as success.

The consequences of our limitations are starkly illustrated by two parallel trends. First, at the same time that the hardware revolution has made computer hardware orders of magnitude smaller, faster, and cheaper, the software crisis has made software orders of magnitude larger, slower and more expensive. Second, the ability to succeed with software projects goes down dramatically as the scale of those projects increases, a factor noted in the CHAOS reports.

Yours truly,
Rick

The Importance of Health Information Technology

Dear Reader,

American healthcare is in crisis. It has been in crisis for decades, but thanks to the Institute of Medicine (IOM)’s Committee on the Quality of Health Care in America we can finally discuss it openly and objectively.

The committee’s reports—To Err is Human: Building a Safer Health System and Crossing the Quality Chasm: A New Health System for the 21st Century—have changed history by ending forever the viability of our industry’s de facto strategy of denying the problem. These landmark reports have also given us a framework for understanding the nature of the problems we face and the broad structure of the solutions we have to create and bring to bear to improve healthcare in America.

In Crossing the Quality Chasm, one of the IOM’s prescriptions for improving American healthcare is the use of information technology (IT): “IT has enormous potential to improve the quality of health care with regard to all six of the aims set forth in Chapter 2.” Indeed, the report’s ninth recommendation reads:

Congress, the executive branch, leaders of health care organizations, public and private purchasers, and health informatics associations and vendors should make a renewed national commitment to building an information infrastructure to support health care delivery, consumer health, quality measurement and improvement, public accountability, clinical and health services research, and clinical education. This commitment should lead to the elimination of most handwritten clinical data by the end of the decade.

This recommendation has launched a national discussion of the importance of health IT and strengthened the arguments of health IT advocates. Where health IT was once a technical topic of limited interest, it is now understood to be an essential component of our national strategy for improving healthcare. For example, the Commonwealth Fund’s 2003 Annual Report, Achieving a High-Performance Health System, uses language very similar to IOM’s in the report’s step four toward building a truly high-performance health system:

Invest in health information technology. Other countries are quickly surpassing the United States in adopting electronic medical records and prescribing systems. Their governments have invested in infrastructure and established the necessary standards, and the United States needs to do the same.

Increasing state and federal interest, funding, and mandates for health IT are symptoms of what has become a national health IT movement, much of which was catalyzed by the IOM’s reports. After all these years, there is a growing national consensus about the potential of IT to improve healthcare and a growing political will to do something about it.

Yours truly,
Rick

Fluxus Quo: A New Series

Dear Reader,

At Rodney Kay's request, let's postpone the continuing discussion of master planning in favor of what I agree is a long overdue topic for understanding VISTA and organic order: fluxus quo.

Since information out of context is noise, let's first explore how fluxus quo applies to health information technology (IT), to make clear why we care. Over the next few weeks, we'll run through a draft chapter of the book I'm writing on the VISTA lifecycle. This chapter makes the VISTA community's central argument about what's wrong with health IT and how to fix it. Fluxus quo is both the problem and the solution.

Yours truly,
Rick

Interlude: Horatio Smith on Shelley's Ozymandias

OZYMANDIAS
or
ON A STUPENDOUS LEG OF GRANITE, DISCOVERED STANDING BY ITSELF IN THE DESERTS OF EGYPT, WITH THE INSCRIPTION INSERTED BELOW

In Egypt's sandy silence, all alone,
Stands a gigantic Leg, which far off throws
The only shadow that the Desert knows:
"I am great OZYMANDIAS," saith the stone,
"The King of Kings; this mighty City shows
"The wonders of my hand." The City's gone,
Nought but the Leg remaining to disclose
The site of this forgotten Babylon.
We wonder, and some Hunter may express
Wonder like ours, when thro' the wilderness
Where London stood, holding the Wolf in chace,
He meets some fragments huge, and stops to guess
What powerful but unrecorded race
Once dwelt in that annihilated place.

The Principle of Organic Order: Our Journey So Far

Dear Reader,

Let's resume our quest to understand the VISTA software lifecycle by continuing our exploration of Christopher Alexander's small masterpiece, The Oregon Experiment. In this book he describes how architectural design used to be organized according to a timeless way of building called organic development and describes the problems that have accumulated in modern architecture as a result of abandoning it. Organic development is the same powerful process harnessed in the classic VISTA software lifecycle. Since the VISTA lifecycle's never been adequately described in writing, we're trying to understand it by reviewing Alexander's description of organic development and by comparing and contrasting it with VISTA's model.

Alexander describes organic development in terms of six principles, with a chapter on each. When we left off in September, we were in chapter one, which describes the Principle of Organic Order. Alexander summarizes this principle as:

Planning and construction will be guided by a process which allows the whole to emerge gradually from local acts.

To illustrate the importance of this principle, Alexander explores the nature of the modern architectural-planning process, in which planners imagine the problem of creating good architecture boils down to a struggle between chaos and order in which their job is to impose order (a vision, an aesthetic, a design) on an otherwise chaotic situation. Planners' main tool in this battle is the modern master plan, which attempts to fix in advance the more-or-less complete solution to the problems they've been hired to solve.

Alexander demonstrates that master plans themselves are deeply problematic, because in good architectural design there's a far more important quality at work than either order or chaos, what he calls organic order. Master plans by their very nature drive organic order out of a project because they're oblivious to it. As long as planners try to reduce their mission to a struggle between two choices, order and chaos, they'll remain unable to understand the damage they do with their master plans, because they'll only look for the elimination of chaos, the creation of order, and call that successful - even though the form of order they create causes worse problems than the chaos they've eliminated.

As with architecture, so with enterprise-scale software-development projects: the creation of a master plan is the first major milestone of any serious project, and the plan remains the fundamental organizing tool for everything that follows. To accuse master plans of being the root problem in modern architecture (or software engineering) is a radical act. It demands more than accusations, so Alexander supports his claim by examining the essential strengths and weaknesses of all master plans, to show that the problems they cause can't be avoided without abandoning this entire approach to planning.

In our review chapter one, we began by exploring the problem of chaos. Although chaotic situations do spontaneously generate forms of order, they also spontaneously generate forms of disorder, and the conflict between the two ensures that chaotically generated order cannot remain stable over long periods of time and also cannot create higher forms of order. For something as complex as medicine with stakes as high as life or death, master planners are right to reject chaotic order as a source of organization for software architecture. The fundamental strength of master plans is that they provide an alternative to chaotic order.

Their fundamental weakness is that in its place they impose totalitarian order, which is essentially hostile to organic order, so we next began reviewing the four unavoidable problems of master plans described by Alexander.

First, master plans are too precise about their solutions. They deal with unknowns by false prophecy, by trying to project a detailed picture of the future. They do this in part because their focus isn't on understanding the problem; it's on selling a solution, and to do that they have to paint a bold, emotionally compelling picture to win the contract. The buyers contribute to this because they're uncomfortable with the inescapable uncertainties at the beginning of any project, so they demand a false clarity to comfort them and give them enough confidence to proceed. The many details fixed in the master plan lend it persuasiveness, verisimilitude, but not truth. Later in the project those same details will tear the project apart as they collide with reality in precise and (only from the perspective of hindsight) predictable ways.

Second, master plans are too imprecise about the problems they mean to solve. As any experienced builder or programmer will tell you, only after you solve a problem do you fully understand it. The process of construction creates a flow of discoveries about the nature of the problems we face, surprises that reveal the many ways we're wrong about what we have to overcome to succeed. We know the most about the overall problem at the end, but master plans are created at the beginning, thus locking in the maximum amount of ignorance about those problems. Master plans propose solutions to problems the participants will never again understand so poorly.

Next we'll explore the third problem, alienation.

Yours truly,
Rick

Interlude: Do the Right Thing

Dear Reader,

Do you want VISTA to succeed, to help us drive medical error out of the top five killers in America? Here are seven steps to make that much more likely:

1) Cancel every VISTA-replacement project going on in the VA and never fund another.

2) Reassemble the primary package-development teams and resume the traditional cycles of rapid patch development and regular releases of new versions of all packages. That lifecycle made VISTA great; violating it crippled VISTA in the DOD, and is crippling it in VA. Put authority over the software back in the programmers' hands and leave it there; they know more about what's possible than you do.

3) Reorganize VA to put the majority of programmers back to work for local hospitals and regional networks. Put authority over the hospital computer systems back in the hands of hospitals and local networks; they know more about how to make VISTA serve their hospitals than you do.

4) Invest heavily in training up a new generation of VISTA experts. This is the second most important item on this list, just behind the next one:

5) Let users' requests for bug fixes and enhancements directly drive all VISTA development. Put authority over development priorities back in the users' hands and leave it there; they know more about what they need than you do.

6) Gut the Clinger Cohen Act so VA and IHS can do the right thing and invest in the people they need instead of being forced to outsource all their expertise and capabilities. The Clinger Cohen Act requires VA and other federal agencies to strangle themselves, which they've been working on doing since 1996. The results speak for themselves.

7) Invest heavily in rebuilding the tattered support community VISTA had before VA shredded it in the mid 90s, the support community the VA needs and cannot thrive without. VA needs to give up its siege mentality, needs to tear down the iron curtain it has built between itself and the rest of the world and start really collaborating with other VISTA organizations, beginning with IHS. Terrorists are not assaulting the VA, and VA's own VISTA experts are not the enemy; VA needs to spend less time and money investigating its VISTA experts for possible "collaboration without permission" and instead seek out and invest in opportunities to work together with others. Having frustrated, insulted, and driven away half of its top-tier VISTA expertise, VA will never again be the sole arbiter of VISTA's future. VA will never again be in a position to do as much damage to VISTA as it did over the last fifteen years. But if VA does all of the right things, if for example it does these seven things among many others, then VA could become a good VISTA organization again, instead of a hostile organization where good VISTA professionals sacrifice and struggle to do the right thing even though they know they may well be punished for it.

I'm hopeful about new VA CIO Roger Baker, but I'm looking for more than hope. It will be easy to measure whether he can fulfill the promise of his good words and good intentions, because deeply experienced VISTA hardhats know what he needs to do to turn things around.

"It's the thought that counts" is a pleasant thought, but in the end what matters is that we find a way to do the right thing. You're not in the dark. You're not lost. You're surrounded by people who know what you need to do to succeed with VISTA. Listen to them. Help them help you to do the right thing.

Yours truly,
Rick

Interlude: Shelley on Hybris

OZYMANDIAS

I met a traveller from an antique land
Who said: Two vast and trunkless legs of stone
Stand in the desert. Near them on the sand,
Half sunk, a shatter'd visage lies, whose frown
And wrinkled lip and sneer of cold command
Tell that its sculptor well those passions read
Which yet survive, stamp'd on these lifeless things,
The hand that mock'd them and the heart that fed.
And on the pedestal these words appear:
"My name is Ozymandias, king of kings:
Look on my works, ye Mighty, and despair!"
Nothing beside remains. Round the decay
Of that colossal wreck, boundless and bare,
The lone and level sands stretch far away.

Four Problems with Master Plans: (2) Imprecision

Dear Reader,

Yes, after the first criticism this second one doesn't seem fair, but it's true. Life, which isn't a logical syllogism, is filled with seeming contradictions. Here's one: not only are master plans too precise, paradoxically they also aren't precise enough.

Specifically while they're too precise about what the future solutions should look like, they're far too vague about the nature of the problems they're supposed to solve. As a result, even in theory master plans only address some parts of the problem and not others.

This is bad but understandable.

Master plans can't possibly identify, let alone address, the full scope of the problems they're supposed to solve because they're created at the beginning, before any serious effort has been made to solve the problem, and thus before Murphy's Law has had its hundreds of chances to teach us how wrong we are about the full extent of the problem. At the time we write a master plan, we just don't know enough about the problem to get its details clear in our heads, and without those details we can't really solve the problem, only take a clumsy swipe at it.

This is understandable but bad.

How bad?

Sometimes it's inconvenient.

Maybe while planning your day, you fail to pay attention to what you're doing at the moment and lock your keys in the car, thus ruining your plans.

Maybe to save three million dollars you replace a deep, stiff bridge design that lets the wind pass through it with a cheaper, shallower, more flexible one that makes the wind pass over and under it, creating aerodynamic lift so it flutters in the wind, tears itself apart, and collapses into the deeps.

Maybe you overplan your bombing runs by putting precisely the right amount of gas in the planes for them to fly out, bomb their targets, and return, but then they encounter a headwind that makes them burn up extra gas so they drop out of the skies before returning home, forcing the pilots to parachute to safety.

Sometimes it's catastrophic.

Maybe you use an explosive gas for airship buoyancy and then try to control all the many scenarios that could cause it to ignite, only to discover that sooner or later your planning or control slips, killing thirty-six people.

Maybe it seems inconvenient to standardize your fire-hose couplings, so you don't, so when your city catches fire the fire engines from neighboring cities arrive only to find they can't help you, and your city burns to the ground.

Maybe your regulations only require a total lifeboat capacity of 1,178 people, even though your ship can carry 3,547, causing the deaths of 1,517 people.

Details matter.

It is the details of problems that make them problems. Something as small as an O-ring seal can kill you if you get it wrong. Engineering is the discipline where Don't sweat the small stuff is a recipe for disaster (and where . . . and it's all small stuff is criminal negligence).

In medical informatics, more starkly obvious than in much of life, the problems we face seem to shift when we discover their nature is different than we at first thought, and they also actually shift as the nature of medicine and software shift around us. The details of our solutions have to be able to shift along with the problems or we can't solve them.

Master plans prevent that dance of solutions with problems in four ways.

First, they emphasize the big picture over the details, wasting energy on prophecy that should have been spent on understanding the problems better, so the necessary details of the solution are missing or vague.

Second, what details they do specify are shaped by design to create (i.e., to serve) the totality, not by discovery to actually solve the problems, so they tend to be out of sync with the reality of the problems (i.e., wrong).

Third, they rigidly lock in those details, preventing them from moving with the problems as the problems shift, soon making any accurate details outdated and incorrect.

Fourth, the nature and interactions of the problems we aim to solve are far too complex to capture in a plan to begin with, ensuring that no matter how much effort goes into a medical-informatics master plan, its worldview always ends up being a ridiculous cartoon of the situation it is meant to deal with.

Any one of these characteristics would be enough to cripple a master plan. Together, they create irresistible forces that cause all VISTA master plans to converge on the same brief lifecycle of (1) ballyhoo, (2) bogging down, and (3) breakdown. And yet the inevitability of failure evidently can't compete with the intoxicating feeling of control a master plan creates, judging by the relentless parade of VISTA-replacement (or "modernization") boondoggles. It would be tragic if after fifteen years it weren't so drearily risible.

Still, at least we can serve as an object lesson to validate Alexander's point:

Master plans fit the shape of the problems with too little precision, and so they fail.

Yours truly,
Rick

Postscript: And so, Alexander's critique of architectural master plans holds even more true for enterprise-scale medical-informatics than it does for architecture: Thus, as a source of organic order, a master plan is both too precise, and not precise enough. The totality is too precise: the details are not precise enough. It fails because each part hinges on a conception of a "totality," which cannot respond to the inevitable accidents of time and still maintain its order. And it fails because as a result of its rigidity, it cannot afford to guide the details around the buildings which really matter; if drawn in detail, these details would be absurdly rigid.

Interlude: Why Some People Just Don't See It

Dear Reader,

In response to my post on 30 August 2009, "Principle 1: Organic Order, part 1: The Three Kinds of Order", on August 31, 2009 at 6:58 AM Die Anyway wrote:

're:"...it's precisely because VA turned away from those processes toward more totalitarian ones that it lost the ability to effectively manage or develop VISTA."

'I see it and you see it so why don't the ivory-tower, pointy-hairs see it? Or do they see it and ignore it because they have an entirely different agenda?

'In any case, as a biologist and programmer I like the idea of organic design even if it did originate in those pre-historic times of the mid '70s.

'Eat well, stay fit, die anyway!'


My answer doesn't fit within a blogger comment, so I'll reply here, as this post:


Dear Die Anyway,

The VISTA managers in VA who try to overly centralize and control VISTA, i.e., who try to impose totalitarian rather than organic order, (1) are not necessarily the majority, just the most powerful or in-favor managers (I know some good VISTA supervisors and managers in the VA and elsewhere), and (2) they do not see what we see because they lack the proper understanding of their context.

They think they're facing an entirely different kind of problem than they actually are. If you interpret their actions from the standpoint of their worldview, their approach makes sense for a while. As we'll explore later in this weblog, their approach leads to a gradually increasing breakdown that eventually becomes so dire that even they can see things aren't working. By then they're usually too burned out and despondent to be capable of taking responsibility for their actions by steering the organization in a healthier direction. There's a certain amount of morale a manager has to have to be effective, and unfortunately when one is committed to a false idea one tends to use up one's effective energies on the dead ends.

The problem with the actual cosmos is that its forces and principles are subtle, easily overlooked. Any reductionist intellectual can mentally reduce an intricate organic system to a trivial mechanical one. When one is tasked with something impossible, like managing VISTA while pleasing Congress, one's mind finds otherwise implausible oversimplifications oddly irresistible, because they offer a desperately needed false hope.

Once people decide on an interpretation of reality, they have an astonishing ability to see every "fact" that confirms that interpretation, and an equally astonishing ability not to see everything else that disproves that interpretation. To the outside observer this seems to result in irrational behavior, but from within the interpreter's reality bubble, within the framework of interpretations, the behavior may be completely rational, even inevitable.

It is a great disappointment to discover that rationality has been overhyped. Reason can be used to reach the falsest or vilest conclusions through irresistible logic drawn from false premises.

Rationality is worthless, even dangerous, unless it is harnessed to two things grossly undervalued - even invisible - in our culture:

(1) profound insight into the essential principles at work in creating each situation, and

(2) mature, discriminating taste capable of balancing priorities among competing values to figure out which good must give way for which other good.

Insight is vital because the truth of hardly any situation is visible on the surface, but instead must be sought in the nuanced and subtle but powerful forces that create that surface.

Taste is vital because contrary to just about everything we teach through schools, the arts, and common sense, the bad things in the world do not result from a great conflict between good and evil, but from conflicts between various goods.

Values conflict, which the classical Greeks knew but which we do not, and which good deserves priority over the others shifts and flows from situation to situation depending on the hidden forces at work.

Neither the truth about the source of problems nor the two vital qualities needed to deal with that truth end up in position descriptions, evaluation criteria, or the law. Instead, managers inside the government and out are held to simple-minded, mechanical criteria, and if they do not bend themselves to fit those laws, criteria, and requirements then they cannot thrive in their careers. They have to believe what they're doing is right, so they do.

In short, I think the answer to your important question is answered best by Upton Sinclair:

"It is difficult to get a man to understand something when his job depends on not understanding it."

Part of why I love biology is that it is difficult to explain in simplistic mechanistic terms, so its study tends to compel you to develop an appreciation of deeper, essential principles.

Thank you for your comments.

Yours truly,
Rick

Four Problems with Master Plans: (1) Precision

Dear Reader,

It is simply not possible to fix today what the environment should be like twenty years from today, and then to steer the piecemeal process of development toward that fixed, imaginary world.

This escapes us when we agree to create a master plan. We forget the simple truth that we're not gods or prophets; predicting the future is never our forte but often our downfall.

Our grand designs fail in four ways.

Today we discuss the first: a master plan is too precise. By predicting the precise shape of the future - the following buildings or software modules will be built in the following ways, and will contribute to the design like so - it leads us into precise collisions with reality when some parts of the plan prove impossible, as always happens.

Murphy's Law isn't very funny to software engineers; it's the reality under which we operate, a reality that's too easy to ignore during planning but impossible to ignore later when our designs come crashing down around all our heads at great expense.

VISTA programming is designed to take advantage of Murphy's Law. It's a highly adaptive process in which we immerse ourselves in the problem with our adopters as guides, in which we do not set out to predict the future, only to solve some specific, immediate problem. A nonprogrammer simply cannot imagine how much failure is involved before one achieves success, but that string of failures is why the VISTA model works and master planning doesn't.

By making planning a cumbersome, expensive process that comes at the beginning, before Murphy's Law has taught us the many ways we're wrong about what's possible, master planning builds a profound ignorance about what's possible into the development process from the start. It saves the exposure of all its failures for the end, when the differences between what's possible and what the plan proposes finally accumulate so greatly that everything collapses in a great crash. Ironically, it's the act of struggling to preserve the plan's success that increases the scale of the failure, since covering up problems and investing more time and money help intensify the inevitable collapse.

By contrast, the VISTA methodology begins making mistakes up front, one at a time, while they're still small - a string of little crashes. We continue making mistakes the whole way, from beginning to end, but they remain small because (1) there is no big, expensive investment in a master plan to try to protect so we can and do change directions repeatedly, (2) the future user of the software is sitting right there the whole time saying "No, we can't do it like that," and (3) instead of struggling to adhere to an increasingly irrelevant plan the team is building up a map of the terrain of the project, a practical guide to many different ways to succeed or fail.

The contrast is stark. Master plans are prophecies. VISTA plans are histories. Master plans are expensive, produced separately from the software. VISTA plans are cheap, produced alongside the software. Master plans are written and finished at the beginning, when we know as little as possible about the problems and its solution. VISTA plans are written during programming and not finished until the end, when we know as much as possible about what didn't work and what did.

The excessive precision of a master plan comes from its focus on the solution, which means trying to pin down the future. VISTA plans avoid that by focusing on understanding the problem and letting the solution emerge gradually and often unexpectedly.

That is, in the master-planning approach we create the illusion of knowing where we're going without first really understanding the nature of the problems we claim to be solving. In the VISTA planning approach, we acknowledge from the start that we have only a vague idea where we're going, only that we know we're going to solve a specific problem by getting to know it very well.

The VISTA approach involves a lot of false starts and backtracking, a lot of mistakes - it looks sloppy and loose compared to the professionalism of a master plan - but that's its strength and the weakness of a master plan.

The VISTA methodology was first advocated by John D. Chase, Jack Brooks, and Ted O'Neill. Here's what Ted O'Neill and Marty Johnson had to say about the difference between the master-planning approach and their own, in their memo to Ken Dickie back on June 10, 1981:

Requirement specifications can be derived effectively and quickly by developing a basic functioning system with the intended user, then further refining the system through user-suggested modifications. The resulting system, in daily use, will provide a much more complete and accurate reflection of user requirements than any narrative description of a system that has not been built. Moreover, by using available modern techniques, such as natural language application generators, database management systems and other software tools, fully operational prototype systems can usually be produced more rapidly than can paper descriptions thereof. Requirement specifications that are produced before any prototype testing activity has been undertaken are now recognized by most computer professionals as inadequate.

Now go back and reread Alexander's quote from the top of this blog entry.

These are two different ways of saying the same thing. They have the same criticism of master plans, independently arrived at from two different perspectives. Our own history bears out the truth of this convergent revelation.

Master plans predict the shape of the future with too much precision, and so they fail.

Yours truly,
Rick

Postscript: Master planning isn't the only kind of planning. The Principle of Diagnosis involves a very different form of up-front planning that helps to create natural order instead of totalitarian order. There are also forms of community planning that focus on studying the problems communities face - the trends, developing problems, and so on - and then draw general, reasonable conclusions to help communities plan. Likewise, there are dynamic forms of project planning designed to accommodate the unpredictable path of research and development. Each of these approaches is the opposite of the master planning approach because they focus on what we can know best - the nature of our present and developing problems - and sketch lightly and prudently in recommending how and when those problems can be solved.

Principle 1: Organic Order: The Problem of Chaotic Order

Dear Reader,

Chaos is not what most people think it is.

To begin with, I'm not using the mathematical term, which sweeps up areas of mathematics outside the traditional disciplines to show there's mathematical order beyond that we usually consider. Although an interesting subject, it's only metaphorically related to true organizational chaos. I'm using the other meaning for chaos: the results of a complete lack of planning, the outcome feared by planners and managers everywhere.

With that in mind, I must next emphasize that chaos isn't the absence of patterns. On the contrary, even fully random processes produce emergent patterns. That kind of patternless chaos people imagine actually takes quite a bit of planning and control to pull off. True chaos always generates two kinds of patterns: accidental and intrinsic.

First, even the chaotic results of random processes produce both statistical ones (like bell curves) and specific ones (like the constellations among the stars) as a normal and expected result of the lack of planning. These macro- and micro-patterns can be used as the seeds for future organization.

Second, when living things are involved (intelligent or otherwise), their needs drive them to produce nonrandom patterns of behavior and organization so they can live and thrive. Firstly, the forces within living things aren't random but instead will lead them to organize their lives in ways that meet their needs - like returning to rivers to drink (or to hunt the drinkers) on a regular schedule. Secondly, living things inhabit ecosystems whose powers and forces reward and punish their behavior, leading living things to adapt in common ways to common forces. For example, even such different life forms as fish, reptiles, and mammals will gradually evolve into the same shapes if they occupy the same ecological niches because the same forces will work upon them over their generations. This is why the tuna, the ichthyosaur, and the dolphin end up closely resembling each other despite very different origins - they converge on the only viable solution to the pressures they're under.

Advocates of imposed order usually underestimate the value of these forms of chaotic order. A great deal of useful order can be achieved without imposition, simply because of people's intrinsic need for order. For example, despite all the screaming and panicking shown in disaster movies, which help to train into us a terror of chaos and of other people (and an artificial craving for imposed leadership that is well understood by juntas when they execute their coups d'etat), during and after disasters people frequently organize themselves to help out family, friends, and neighbors, without any leadership at all - not always, but more often than fearmongers would have us believe.

Advocates of chaotic order, such as anarchists, point to this kind of natural collaboration and contrast it with the inevitable evils of totalitarian order to argue their position that any form of central organization is both unnecessary and unwanted. Anyone who's bothered to study Nazi Germany or Stalinist Russia, or who's thought long and hard about how readily republics loose their democratic bonds to become fascist empires, or who's observed how the disturbing trend toward decreasing personal freedom and privacy changes our culture more and more toward the dystopia of Orwell's novel 1984, has to give this position more than a moment's consideration. There are serious points raised by advocates of chaotic order, and we must do justice to those issues or suffer the consequences of those who fail to learn from history.

Nevertheless, I'm not persuaded, and you shouldn't be either.

There are two excellent reasons why chaotic order is not sufficient.

1) The order produced in chaotic systems is generally offset by the disorder also produced by chaotic systems. Without central coordination, some enclaves will self-organize into more or less peaceful communities, true, but others will devolve into internicine feuds (Internet flame-wars, anyone?), and still others will become predatory. Just as I'll not grant advocates of totalitarian order their insistence on the absence of order within chaos, so I'll also not grant the advocates of chaotic order the absence of destructive consequences.

2) Chaotic order is almost always local, almost never universal. That's enough to solve certain categories of problems but not others, not problems beyond a certain scale. As a VISTA theorist, I am particularly bothered by this reason, because the complete VISTA lifecycle requires that (a) a great number of specialists be given the time and focus needed to master their chosen parts of VISTA, (b) a large number of autonomous teams follow a common culture to ensure that what they produce is compatible, that it creates a living whole, and (c) people not only act with good intentions but also actually do the right thing, since lives and privacy are at stake. These things require more than local or temporary order. These requirements unfortunately fall into chaotic order's demonstrated weaknesses, making it a poor choice of approaches for developing VISTA.

But beyond two there is an even better third reason to reject it for VISTA: it is predicated on a false dilemma, namely that one must choose between totalitarian order and chaos, or some compromise between them. As Alexander argues and I'll explore in this weblog, natural order represents a superior alternative to either of these dysfunctional approaches.

That is, since we already have a model for a system of organic order for managing VISTA development, we don't need to pay the price of chaotic order's weaknesses in order to gain its benefits.

Nevertheless, we're going to return repeatedly to the subject of chaotic order, because in management terms (and now for the punchline) chaos is just a pejorative term for freedom, reflecting a deep unacknowledged terror in all of us. Until we examine our ambivalent relationship to freedom, any cost-benefit analysis of chaotic order is at best highly suspect.

Yours truly,
Rick

Interlude: Ordering The Oregon Experiment

Dear Reader,

You can order a copy of The Oregon Experiment from your friendly neighborhood independent bookstore, or from any number of online book services, including:

A Libris
Abe Books
Amazon
Powell's

Yours truly,
Rick

Principle 1: Organic Order, part 1: The Three Kinds of Order

Dear Reader,

Alexander diagnoses a nearly universal illness in our systems of architectural planning: we use master plans to try to guide our development, to plan for the future:

. . . master plans, are intended to coordinate the many hundreds of otherwise independent acts of building . . . to make sure . . . that the many acts of building in a community will together gradually help to create a whole, instead of merely making up an aggregation of unrelated parts, a chaos.

. . . the master plan, as currently conceived, cannot create a whole. It can create a totality, but not a whole. It can create totalitarian order, but not organic order . . . although the task of making sure that individual acts of building cooperate to form a whole is real, the conventional master plan - based on a map of the future - cannot possibly perform this task . . . because it is too rigid to do so - and because, in addition, it creates an entirely new set of other problems, more devastating in human terms than the chaos it is meant to govern.

It's not enough to replace chaos with order. Order can be worse than chaos.

The problem with chaos is that the "whole" (really just a collection) is less than the sum of the parts. The entire reason for planning is to do better than this, but master plans only give us chaos's opposite: totalitarian order, when the parts are designed to support the whole. The problem with totalitarian order is that the parts don't adequately support themselves or each other, only the whole, and the whole doesn't support the parts, only itself.

Seen with a clear understanding of what we really want in the world, chaos and totalitarian order strongly resemble each other, because they both lack what we most need. What we're striving for isn't a compromise between order and chaos; it's nowhere on that spectrum, because that spectrum doesn't take into account what really matters.

What we need is what Alexander calls "natural order" or "organic order":

We define organic order as the kind of order that is achieved when there is a perfect balance between the needs of the parts, and the needs of the whole.

In an organic environment, every place is unique, and the different places also cooperate, with no parts left over, to create a global whole - a whole which can be identified by everyone who is a part of it.

This kind of order is called natural because it's the kind of order nearly universally created by natural processes, and only rarely by artificial ones. Living processes result in parts that are fully, distinctively, amazingly themselves and yet are equally completely part of and contributing to a greater whole.

The balance between the needs of the parts and the needs of the whole allows the parts to be simultaneously highly specialized, optimized to do their jobs as well as they can be done, while simultaneously helping to create a healthy whole that couldn't exist without those parts.

Consider the relationship between the parts and the whole in the human body. A stomach and a brain are astonishingly different, each highly adapted to its job - if you had to know how to digest an apple you'd starve to death, and we all know what happens to people who try to think with their stomachs - yet each organ fully contributes to the success of the whole. You need them both to live. They're indispensably part of a whole, yet wildly different from each other.

Or consider an ecosystem. A cheetah is an amazing creature. Watch a film of a cheetah chasing down prey and you can hardly believe what you're seeing is possible. It gives every appearance of being a completely original, independent design, a master at what it does. And yet, it is simultaneously utterly dependent on its ecosystem for its survival. Without enough habitat and prey, the cheetah cannot survive. Likewise, without the cheetah, its prey overpopulate and strip bear the vegetation, causing ecological collapse and the widespread death of its prey.

In both examples, the relationship among the parts and between the parts and the whole are nothing like those relationships in a chaos or a totalitarian order.

The parts in a collection (chaotic order) are unrelated, random things. They rarely work as a whole, and never for long; they work against one another at least as often as they work together.

The parts in a totality (totalitarian order) are cogs in a great machine, all resembling each other because they're designed by the same designers. They break down readily because they do not support themselves or each other, only the whole. Place a living thing within such an order and try to use it as a part and it soon perishes because nothing in a totality fully supports the needs of the part.

The parts in a whole (organic order) look very different from one another, each highly adapted to its situation in order to be great at what it does, but if you study them long enough you begin to see the subtle common principles of their designs and to understand the ways in which they're all dependent on one another and contribute to a common whole that in turn strengthens each of them as parts.

Put another way, in organic order each part is also a whole, identifiable and supported by its own parts, by the other wholes around it, and by the whole of which it is just a part. Conversely, each whole stands for more than just itself; it also contributes to the health of its own parts, to the other wholes around it, and to the greater whole they make up together.

This quality is what makes living systems so much more scalable than artificial ones. Each scale of organic organization supports not only itself but also the greater and lesser scales. This is why as totalitarian structures get larger they get weaker - the increasing demands of the whole parasitically sap the strength of the parts - but as organic ones get larger they get stronger - the existing whole and its parts strengthen each new part added to it, and each new part in turn strengthens the other parts and the whole.

This is why the cosmos creates wholes and parts that are so vastly more complex than anything made by man, and yet those systems of wholes and parts last for millions or billions of years instead of the mere decades and centuries we manage. Chaotic order and totalitarian order may be sufficient for the trivial levels of complexity and brief time frames people usually work in, but if you need to create something vastly more complex to last for much longer, then only organic order will do.

VISTA is at that scale of complexity and longevity. It only exists because it was created using living processes that resulted in natural order, and it's precisely because VA turned away from those processes toward more totalitarian ones that it lost the ability to effectively manage or develop VISTA.

Organic order is the grail. This is what you want, and nothing less. It's a key to everything in VISTA. It's what Alexander in later works refers to as a living system, what he uses in his new definition of life in The Nature of Order. Understanding what organic order is, learning to detect its presence or absence, figuring out why it matters so much and how to create it - these are the central challenges in becoming a great VISTA manager.

Yours truly,
Rick

Postscript: Throughout this exploration, I refer to the principal author as a shorthand for the team of authors who developed these ideas and wrote this book. To set aside expediency for a moment and give credit where credit's due, here's the full bibliographical entry on this book:

The Oregon Experiment. Christopher Alexander, Murray Silverstein, Shlomo Angel, Sara Ishikawa, and Denny Abrams. New York: Oxford University Press, 1975. ISBN 0-19-501824-9. Volume 3 in a series from The Center for Environmental Structure.

Our Story So Far . . .

Dear Reader,

There's an emerging order to this weblog. We began with a little introduction to VISTA before turning to the long project of investigating the VISTA software lifecycle. The first four chapters of that investigation are related in a flow of their own.

First, we started by breaking up the ground, by challenging the most common misunderstandings and filters that prevent people from being able to understand the lifecycle even when they're looking right at it. This was the chapter on the eight essential points.

Second, now, we're taking our first steps into understanding the lifecycle by using something similar that's already been well explained as a frame of reference. That's this new chapter on the principles of organic development as explained in The Oregon Experiment.

Third and fourth, after establishing that point of reference we'll drive right to the heart of the VISTA software lifecycle by examining its two essential characteristics that together set up the pattern that makes sense out of every detail of its architecture or policy: flow (fluxus quo) and life (living systems).

Thereafter, in subsequent chapters, with the main misunderstandings dealt with, a common frame of reference to compare and contrast with, and the two essential characteristics fully described, it should be easy enough to work our way through the details of how to manage VISTA effectively.

Like all plans, I'm sure this won't survive much contact with the subject matter, but let's find out.

Yours truly,
Rick

Crickets

Dear Reader,

Are you there? Do you care? Am I making sense or losing your understanding or interest? Question me. Challenge me. Criticize me. I want to do a better job, but my perspective on myself is limited; I need help finding ways to make this better. I am but a man, with a man's many flaws, trying to carry this particular load for the community.

The ideas I'm trying to capture and express are a crucial part of the theoretical framework for VISTA, the part that says how to develop it. If these ideas were better understood and accepted, the evils VISTA has suffered and that veterans have suffered as a result could not have occurred without immediate public outcry. The lack of understanding of this lifecycle created the vulnerability that has been exploited for the last fifteen years. If we want to put a stop to this kind of attack on our patients, we need this book. We've needed it for a long time.

We need this book to be as good a book as we can make it, to be clear and compelling. It will be much better if we engage in a dialog now, while we can work out the ideas in electronic format. Once it goes into print, it's too late to fix the ideas or improve the explanations.

I'll make you a deal.

If you help me find the flaws in my arguments and improve the ways we explain this lifecycle, I will credit you for those improvements explicitly in the print version of the book. Public credit is as much a VISTA tradition as friendly public criticism.

Yours truly,
Rick

Principles of Organic Development

Dear Reader,

Although The Oregon Experiment is the best written description so far of the VISTA software lifecycle, it doesn't describe it perfectly.

Why would it? VISTA's lifecycle unfolded according to its own path of development, in response to its own internal and external pressures, many of which emerged after this book was written. This book's such a good way to begin learning that lifecycle because it so clearly describes such a good example of an organic development-process.

Living systems, like VISTA and like physical architecture, can't be adequately managed by mechanical development-processes.

They don't need to be. There's an alternative.

Instead of mechanical development-processes, we can adopt organic development-processes, which mimic the processes living system use to produce new living systems, instead of the processes people use to produce machines. VISTA's history (including its current problems) proves we must replace VA's current mechanical VISTA-planning processes with its original (or improved) organic ones, but doing that will require understanding organic development-processes at least as well as we currently think we understand the mechanical ones.

That's where Alexander's book comes in. Alexander proposes six principles that together create an organic process for architectural planning. If you learn how these six principles guide this architectural lifecycle, which is clearly and cogently explained in The Oregon Experiment, then you're halfway to understanding VISTA's software lifecycle. Discussing this book isn't a diversion; it's a shortcut.

The similarities to the VISTA software lifecycle started by O'Neill and the original hardhats and developed over time within VA and IHS are eerie, but the differences are equally important and illuminating. We'll examine both as we explore Alexander's six principles of organic development.

There are two particular problems with this model I'll highlight throughout the exploration that follows.

First, as Alexander makes very clear, these principles aren't the ideal suite to guide organic development-processes. They're a compromise designed for overly centralized situations in which budgetary authority is and must remain centralized. This wasn't, needn't, and shouldn't be the case in VA, where local hospitals can, did, and must control part of the IT budget so they can share in the decision making. The current totalitarian degree of centralization in VA's management and funding of VISTA is a grotesque organizational illness that new VA CIO Roger Baker needs to cure for the continuing health of the veterans whose care is now his charge. The importance of class-three software in VA (a subject of its own we'll thoroughly explore in the weeks ahead) demands a return to a balance between central office and the hospitals. That healthier balance also represents a shift away from Alexander's compromise in this book back toward the purer model he envisioned but didn't document here.

Second, software and buildings share a surprising amount in common, but their differences are also important. Among other things, software's far more malleable than buildings are, so the pace of software change in a healthy VISTA organization is one or more orders of magnitude faster than the pace of construction change in a neighborhood can ever be. This creates both solutions and problems Alexander didn't have to address in The Oregon Experiment.

We'll explore his six principles of organic development with these and other differences in mind.

Yours truly,
Rick

The Oregon Experiment: Six Principles

Dear Reader,

The Oregon Experiment is a quick read worth reading more slowly. Since this book is so important to understanding the VISTA software lifecycle, let's examine what it can teach us about VISTA.

Here are the book's six principles:

Specifically, we believe that the process of building and planning in a community will create an environment which meets human needs only if it follows six principles of implementation:

1. The principle of organic order.
2. The principle of participation.
3. The principle of piecemeal growth.
4. The principle of patterns.
5. The principle of diagnosis.
6. The principle of coordination.

We recommend that the University of Oregon, and any other institution or community which has a single owner, and a centralized budget, adopt these six principles to replace its conventional master planning and conventional budgetary procedures, to provide the administrative resources which will guarantee people the right to design their own places, and to set in motion the democratic processes which will ensure their flexible continuation.

For the sake of concreteness, and to give you an overview of the book, we now outline these six principles.

1. The principle of organic order.
Planning and construction will be guided by a process which allows the whole to emerge gradually from local acts.

2. The principle of participation.
All decisions about what to build, and how to build it, will be in the hands of the users.

3. The principle of piecemeal growth.
The construction undertaken in each budgetary period will be weighed overwhelmingly towards small projects.

4. The principle of patterns.
All design and construction will be guided by a collection of communally adopted planning principles called patterns.

5. The principle of diagnosis.
The well being of the whole will be protected by an annual diagnosis which explains, in detail, which spaces are alive and which ones dead, at any given moment in the history of the community.

6. The principle of coordination.
Finally, the slow emergence of organic order in the whole will be assured by a funding process which regulates the stream of individual projects put forward by users.

Next, let's examine these principles in detail, beginning with the principle of organic order.

Yours truly,
Rick

Book Recommendation 1: The Oregon Experiment

Dear Reader,

To Roger Baker, the new chief information officer of the U.S. Department of Veterans Affairs, I offer this advice: if you want to be a great VISTA manager, a great VA CIO, I can recommend nothing more important than to read this book. It'll prepare you for your new job in ways no meeting or position paper ever will. It'll tell you things you need to know, things none of your subordinates can tell you. Unless you do this, you'll almost certainly repeat the mistakes of your predecessors despite your best intentions. With it, you create the possibility of becoming the second-greatest manager VISTA has ever known.

Why?

Because in the year 1975, in 190 small pages packed with illustrations, Christopher Alexander and his team came as close to describing the VISTA lifecycle as anyone has yet done in writing. It should be required reading for anyone who intends to manage a VISTA project.

The surprises here are (1) that VISTA didn't exist in 1975, except as a set of observations and principles in the possession of VISTA founder Ted O'Neill, so of course Alexander had never heard of it; and so (2) he and his team weren't writing about VISTA or even software development; they were writing about architecture and architectural planning.

They were trying to describe what seemed to most people like an unusual approach to architectural planning; it was the original approach used by humanity for millennia but abandoned in the nineteenth and twentieth centuries. This is analogous to what Ted O'Neill and his team were setting out to do in the VA: to try out a promising approach to medical-software development that defied mainstream ideas about how to develop software.

Both projects began by studying the past, studying what worked and what didn't, building up lists of tactics for how to succeed or fail, and using those lists to develop a coherent strategy. The two strategies resemble each other closely, but Alexander and his team wrote a book about their strategy, whereas we haven't yet written the book about the VISTA strategy.

Other similarities abound between the two projects. Like physical architecture, medical-software architecture is complex, easy to get wrong, and affects many people who really care about the results, and the costs of failure are paid in time, money, morale, and sometimes human lives.

Most crucially though, both kinds of efforts largely succeed or fail on the effectiveness of their management processes. You can have the best builders of buildings or code in the world, but if you manage them badly you guarantee failure - and it's easy to manage them badly. Indeed, in both disciplines bad management's not an accident but a carefully learned behavior, in which management practices that work well in other situations work badly in these two yet are adhered to forcefully even though they leave behind a track record of failure after failure.

The reason both disciplines are so vulnerable to bad management is that the two subjects share two vital characteristics: (1) flow, and (2) life. Less tersely put, (1) they both have to be managed using organic development-processes, because (2) both physical architecture and VISTA itself are living systems that only superficially resemble mechanical systems.

The full significance of that paragraph will take a few weeks to illuminate adequately, so let's get to it, step by step.

Meanwhile, order yourself a copy and read it carefully.

Yours truly,
Rick

The Essential Point

Dear Reader,

Companion to the essential principle is the essential point.

What's the essential point for understanding the VISTA software lifecycle?

There are many wrong answers to that question and only one right one. The problem is that each of us prefers one or more wrong answers, for reasons we think are really good reasons. Nothing I can say can compare to the attraction of those reasons, so I'm not going to answer this question for you.

Instead I leave you with the exhortation that you have work to do on this problem, that there is no shortcut to answering it, that the answer matters, and that you'll have to work diligently and scrupulously over a long time to peel away the layers of your misunderstanding of this essential point in search of the truth.

That's not a bad thing. Work is good. Work on the right things and that work will transform you into a better person. This work, the work to discover the key to understanding the VISTA lifecycle - the real key, not the one you like - will do that too, and it'll make you a better VISTA manager.

Without this work it's all too easy, inevitable really, to violate the essential principle, as VISTA's history demonstrates over and over.

So I'll leave you with this task - it was yours already - and turn away from the eight essential points for now to a different perspective on the VISTA software lifecycle.

Yours truly,
Rick

The Importance of Project Management

Dear Reader,

Lest there be misunderstanding, allow me to clarify that when I wrote

"To sacrifice medical science on the altar of project management is a form of human sacrifice"

I in no way meant to imply that project management is not needed with VISTA. On the contrary, the effectiveness of the project management seems to correlate directly with the odds of success in the last decade of projects to install VISTA outside the federal government.

High-quality VISTA-project managers seem to be among the rarest of the rare specialties in increasing demand as the expertise bottleneck gets a good grip on the community.

A great deal of what I'll be writing about in the weeks ahead assumes and depends upon a solid project-management foundation for the lifecycle to succeed.

Rather, what I meant is that within the VISTA-software lifecycle, project management must serve medical science, not attempt to dictate terms to it. Just as the approach to software development must be adjusted to be compatible with the complexity and flow of medical science, so too must project management.

To do otherwise, for example to prioritize deadlines over medical correctness, is literally to endanger our patients. There has been a great deal too much worship of timelines in VISTA management over the last fifteen years, and the health of our patients and of the software that supports them has suffered.

Replacing that kind of poor-quality, mechanical project management with experienced, fluent, and well-adapted project management is one of the major steps in launching this new VISTA renaissance.

Yours truly,
Rick

A Culture of Criticism

Dear Reader,

Miss Manners, many blessings upon her, gently reminds us that it is not proper to try to parent other adults (or even children other than your own, prior to their graduation into adulthood), to correct them or criticize them, in public or otherwise.

There are two exceptions.

1) Parents can and should correct their children, preferably gently, by example, and in a way that both helps them develop into good people and also preserves their dignity.

2) Teachers can and should correct their students, much as parents correct their children but not across as broad a range of subjects, more focused upon the subjects under consideration but still including an emphasis on helping them develop into good people and preserving their dignity.

This second case applies to the VISTA community.

According to the seven-part VISTA model, our software lifecycle depends upon another lifecycle, an expertise lifecycle. The VISTA community has to be structured as a perpetual-learning organization, with each community member acting as both student and teacher to the best of their abilities. We have to cultivate our expertise and help each other develop, because we are embarked upon such a difficult mission.

In an educational context, especially one dedicated to scientific and engineering pursuits, we have to be straight with each other about our mistakes, our flaws, our bugs, our opportunities to learn and improve. This can and should be done respectfully, but it must also be done unambiguously, and if the students don't get the message with subtler forms of suggestion, more overt and direct language is called for.

The key to open-source-software development is that we work together to find and fix the problems so we can keep making our software better.

The key to an expertise lifecycle is that we do the same thing to make ourselves and our development processes better.

Even the best hardhat, the most experienced or expert hardhat, the wisest hardhat, is a perpetual student of VISTA, whose mastery will always be exceeded by the things he still has to learn.

To improve the software we must always be straight with each other about the bugs in our code.

To improve the software lifecycle, we must do likewise with our distribution of authority and our management of projects.

So, when I criticize the VA managers for their mismanagement of VISTA, it is in this spirit. I'm not setting myself up in opposition to them, or as superior to them, or on the other team from them. Mostly they're friends of mine, people I know and respect.

And in the same spirit that I critique the flaws in the programming style of my newest students, in the same say I criticize my own code when I'm team programming with my peers, I will call out the flaws in the approach of the VISTA managers. The best of them expect this of me, demand it of me.

And in doing these things, I am not doing anything special, not in the VISTA community. I am just doing my duty as a VISTA hardhat. Any of us would do the same.

In the VISTA community, we're straight with one another about the things that need to be fixed. Anyone who can't handle that must either learn to handle it or else find themselves another line of work. What we do is too important to lie to ourselves about our performance. We have to improve, always.

Our responsibility to VISTA adopters and their patients demands it of us.

Yours truly,
Rick

Humility and Courage

Dear Reader,

VISTA humility requires that when Congress asks us how long something will take, our answer is "I don't know," unless it's something we've done repeatedly and consistently.

For example, we can answer how long it'll take to add a simple new report to a package we know extremely well. Or, if we've already written an interface between the McKesson Series Scheduling package and VISTA and installed it at numerous sites, we're within our rights to report how long it's taken the last five times we installed it and to suggest a similar timeframe.

If, however, we're asked how long it'll take to write an interface between a VISTA package and an external clinical system to keep the two systems in sync, we have to say "I don't know," even if we think we understand the scale of the problem. There's all the difference in the world between visualizing a solution and actually solving the problem that seems to elude most VISTA hardhats and managers alike.

Usually, about the only honest answer to questions about how long a task will take with VISTA is "Longer than either of us thinks." We're guessing about anything we haven't done repeatedly and consistently with VISTA. The moment we peg down a date on a task outside this rather confined circle of experience, we are lying. We have left humility and caution behind in favor of bold hybris, which feels intoxicating, sure, but is fraught with peril.

Why?

Because VISTA is bigger than we think it is. VISTA's ugly facts will sink your beautiful theories. VISTA sneers at your petty delusions of predicting the future. VISTA is filled with surprises.

Because these things are true of medical research, as well. Ask yourself how many times so far the answer to the question "When will we cure cancer?" has been correct. Does the comparison seem unfair? Then you are still underestimating the scale of VISTA. When I told you you were getting it wrong, you thought I was resorting to shallow rhetorical flourishes, but I'm not. From one perspective, VISTA's function is to eventually capture and represent everything we know about medicine in a consistent and logical system of data and programs that can reliably support medical practice. Is the entire field of medicine complex enough to get your attention, to make you even for a moment set aside your genetic predisposition to overconfidence and begin to quail at the scope of it? Imagine trying to sort out and systematize that field to the level of mind-numbingly stringent literalism that a computer requires.

Because medical-software engineering is not as trivial a discipline as building planes that don't crash or rockets that don't blow up on liftoff. They only have to deal with physics and weathering (and human error and politics and economics and the other universal problems). Medicine makes a lot less mathematical sense than physics, and software engineering's much easier to get wrong than materials engineering because of the relative lack of physical constraints, yet computers are vastly less tolerant of sloppiness in specification than human pilots are. Put that all together and you get a messy, difficult, complicated situation.

Because at its best software engineering is more like research and development than like engineering a bridge. The discipline of software engineering, when practiced correctly, is nearly the opposite of the discipline of physical engineering. The omnipotent malleability of software collides with the highly idiosyncratic needs of software users to create vast and shifting uncertainties. Properly practiced, software engineering is designed to be highly adaptive to the ocean of unknowns we have to deal with.

Because nine times out of ten our statement of the problem is wrong. We think we know what we're asking about, but the truth is we've grossly underestimated how much we really need or how many other features are actually involved in what we thought of as a simple, stand-alone feature.

At multiple points in the process of developing anything but the simplest changes, we're going to get checked by reality. Surprises and adjustments creep in. Sometimes it's not until the moment we begin seeing drafts of the "finished" software and realize something's very wrong here, but often the programmers return after just barely beginning the task to report they already need to change the plan because it left out something important. And usually, those two patterns occur over and over (and over and over and over . . .) throughout the development process.

Traditional project management loathes this kind of shifting of the parameters of a task. They call it "feature creep" and blame it on the users and programmers. They believe the way you keep projects on time is to stop feature creep. That's an idiotic position to take, because it leads to delivering the wrong thing on time and actually being proud of that accomplishment. It's an example of how hybris can twist people into the most ridiculous postures, into defending the most indefensible positions.

I believe, practiced properly, feature creep is the key to VISTA's success.

Giving medical practitioners what they actually need rather than the gross caricature we envision at the start of a project is what results in software that doctors and nurses can actually use to improve health care. When you're making a game or a word processor, maybe it's okay to sacrifice features to meet a deadline, but when you're making medical software do you really think it's a clever idea to pressure your programmers to cut corners or to ignore the negative feedback from doctors about early drafts of the software, just so you can meet a date that - let's face it - you had no right to give in the first place, that no matter how you came up with it was in the end a ridiculously arrogant attempt to predict the future?

To sacrifice medical science on the altar of project management is a form of human sacrifice.

There are better ways to manage medical-software projects. There are specialized forms of project management that do not begin with the pretense that R&D can be controlled, that instead harness it like an ecology, like a garden that regularly produces good things on its own schedule so long as it's properly cultivated. In a nutshell, that's precisely the VISTA software lifecycle's approach to project management.

But it all begins with admitting that VISTA development is, in fact, R&D, that it's filled with unknowns and surprises, that, in fact, "we don't know" how long significant projects will take to complete, with only the rarest of exceptions.

So here's the problem with the principle of humility.

Good VISTA-project management begins with recognizing the difference between hybris and humility, a subject that most people think they understand but that the evidence of history soundly proves otherwise. Most people, while engaged in acts of the utmost hybris, feel they're acting in a professional, responsible, and cautious manner.

In the case of VISTA managers, this is often because the political pressure they endure from Congress makes them feel small and powerless, afraid for their careers and for the future of VISTA. It is while in this mindset that the worst acts of hybris take place.

It can be very confusing, so I'll make it clear.

You should be more afraid of VISTA than you are of the President of the United States himself. If POTUS demands you tell him how long it'll take to replace the VISTA Laboratory package with a modern, off-the-shelf or custom-built proprietary solution, humility requires you to refuse to take the bait. Humility requires you to say the following:

"Mr. President, I don't know, and neither does anyone else. Most software projects fail either partly or completely, and big ones fail most of all. Replacing VISTA Laboratory's on the far end of big. Anyone who tells you such an undertaking's likely to succeed let alone when is ignorant or lying and probably both. I advise against it, Mr. President."

If you couldn't give this answer, if it seems arrogant and dangerous, it's because you haven't yet really come to terms with the degree of humility VISTA management demands of you. You're still afraid of the wrong things.

You still think angering powerful people or institutions is the worst thing that could happen to you. You are so wrong. POTUS is merely an extremely powerful man. He can only bring the human world against you if you cross him. (Besides, you never know, the President might prefer people who tell him the truth).

If to avoid disappointing him or any other man you go up against the laws of information science, it's the cosmos itself that'll grind you down. The laws of information science are as implacable as the laws of physics and far less well understood.

You should be humble. You should know your place. The health of patients cared for by the hospitals, clinics, and offices that use VISTA depend upon you to respect and fear the seriousness of your work. They need you to protect them from your hybris, no matter how the mighty incite you to it.

Managing VISTA requires humility; and, in the human world, humility requires courage.

Yours truly,
Rick