VISTA Enterprise Network - Successful Implementation, World Class Support

Tuesday, November 24, 2009

The Principle of Organic Order: Our Journey So Far

Dear Reader,

Let's resume our quest to understand the VISTA software lifecycle by continuing our exploration of Christopher Alexander's small masterpiece, The Oregon Experiment. In this book he describes how architectural design used to be organized according to a timeless way of building called organic development and describes the problems that have accumulated in modern architecture as a result of abandoning it. Organic development is the same powerful process harnessed in the classic VISTA software lifecycle. Since the VISTA lifecycle's never been adequately described in writing, we're trying to understand it by reviewing Alexander's description of organic development and by comparing and contrasting it with VISTA's model.

Alexander describes organic development in terms of six principles, with a chapter on each. When we left off in September, we were in chapter one, which describes the Principle of Organic Order. Alexander summarizes this principle as:

Planning and construction will be guided by a process which allows the whole to emerge gradually from local acts.

To illustrate the importance of this principle, Alexander explores the nature of the modern architectural-planning process, in which planners imagine the problem of creating good architecture boils down to a struggle between chaos and order in which their job is to impose order (a vision, an aesthetic, a design) on an otherwise chaotic situation. Planners' main tool in this battle is the modern master plan, which attempts to fix in advance the more-or-less complete solution to the problems they've been hired to solve.

Alexander demonstrates that master plans themselves are deeply problematic, because in good architectural design there's a far more important quality at work than either order or chaos, what he calls organic order. Master plans by their very nature drive organic order out of a project because they're oblivious to it. As long as planners try to reduce their mission to a struggle between two choices, order and chaos, they'll remain unable to understand the damage they do with their master plans, because they'll only look for the elimination of chaos, the creation of order, and call that successful - even though the form of order they create causes worse problems than the chaos they've eliminated.

As with architecture, so with enterprise-scale software-development projects: the creation of a master plan is the first major milestone of any serious project, and the plan remains the fundamental organizing tool for everything that follows. To accuse master plans of being the root problem in modern architecture (or software engineering) is a radical act. It demands more than accusations, so Alexander supports his claim by examining the essential strengths and weaknesses of all master plans, to show that the problems they cause can't be avoided without abandoning this entire approach to planning.

In our review chapter one, we began by exploring the problem of chaos. Although chaotic situations do spontaneously generate forms of order, they also spontaneously generate forms of disorder, and the conflict between the two ensures that chaotically generated order cannot remain stable over long periods of time and also cannot create higher forms of order. For something as complex as medicine with stakes as high as life or death, master planners are right to reject chaotic order as a source of organization for software architecture. The fundamental strength of master plans is that they provide an alternative to chaotic order.

Their fundamental weakness is that in its place they impose totalitarian order, which is essentially hostile to organic order, so we next began reviewing the four unavoidable problems of master plans described by Alexander.

First, master plans are too precise about their solutions. They deal with unknowns by false prophecy, by trying to project a detailed picture of the future. They do this in part because their focus isn't on understanding the problem; it's on selling a solution, and to do that they have to paint a bold, emotionally compelling picture to win the contract. The buyers contribute to this because they're uncomfortable with the inescapable uncertainties at the beginning of any project, so they demand a false clarity to comfort them and give them enough confidence to proceed. The many details fixed in the master plan lend it persuasiveness, verisimilitude, but not truth. Later in the project those same details will tear the project apart as they collide with reality in precise and (only from the perspective of hindsight) predictable ways.

Second, master plans are too imprecise about the problems they mean to solve. As any experienced builder or programmer will tell you, only after you solve a problem do you fully understand it. The process of construction creates a flow of discoveries about the nature of the problems we face, surprises that reveal the many ways we're wrong about what we have to overcome to succeed. We know the most about the overall problem at the end, but master plans are created at the beginning, thus locking in the maximum amount of ignorance about those problems. Master plans propose solutions to problems the participants will never again understand so poorly.

Next we'll explore the third problem, alienation.

Yours truly,


Rodney H. Kay said...

I want to take short exception here because I believe you are deviating from a previous premise that accurately described the software we call VistA. In any type of architectural endeavor, you are attempting to create something finite and concrete - something that will last without changing, dramatically, over a long time span. The Oregon Experiment makes an excellent case for organic order vis-a-vis order/chaos dicotomy. It is an elegant argument on how to achieve the status quo.

With VistA, the goal is not to achieve the status quo, but the status fluxes - to achieve that point of "not quite good enough" to solve, not the master problem, but the problem at hand. VistA, by it's very design and history, is innovative and disruptive at the same time. To attempt even some portion of either organic order or totalitarian order asks that you destroy either innovation or disruption. I would suggest you spend some time in the latest work by Clayton Christensen called The Innovator's Prescription to try and get a grasp on what I am talking about. While not mentioning VistA directly, Christensen talks about disruptive innovation in health care informatics that is right on the mark.

Rick Marshall said...

Dear Rodney,

First off, thanks for the book recommendation. I just ordered a copy, and I look forward to reading it when it arrives.

Second, I don't agree that I'm deviating from the premise, but I do agree that I'm not explaining the current line of argument well enough. The way I've been explaining things does create the cognitive illusion of a conflict between organic order and a core principle of VISTA. I think you're right that I've gone too far without trying to explain fluxus quo, one of the core concepts in understanding VISTA. It turns out that fluxus quo is also one of the core concepts for understanding what Alexander and I mean by organic order, which is a very different kind of thing from either chaotic order or totalitarian order.

The important issues and comparisons you raise in your comment have helped me to understand where I've gone wrong in my explanation. Let me take another stab at this by getting into the nature of fluxus quo, something you rightly wanted me to bring up earlier but that I misjudged as an issue better raised later.

Much appreciated.

Yours truly,