The process of formal risk management is closely related to design and may add insight and perspective to a workshop on Decision-Based Design. When a design has reached a convergent stage and the architecture and components are reliably modeled, it is more valuable to focus on the departures from the ideal state as we attempt to optimize. Thus, when we strive for greater accuracy, we concentrate on the minimization of error; when we strive for greater reliability, we concentrate on minimization of failure rate, and so on. For at least a decade, the Department of Defense has considered the minimization of risk as one of its highest priority design processes.

Although neither risk management nor design is practiced today in a rigorous, formal fashion, they share a very similar future. Both would benefit from the application of modern decision theory which has these foundations: establishment of value functions, generation of viable alternatives, assessment of likelihoods -- all integrated by a logical structure.

Perhaps the most challenging of these foundations is the establishment of a value function that is "rational" to a single decision maker as well as to a group of stake holders. Certainly, in order to optimize a design, the value function must be transitive. In fact, the use of intransitive value functions is often deemed "irrational." Hazelrigg₁ points out that a consequence of Arrow’s Impossibility Thereom is that a group of decision makers, each of which has a transitive value function, can end up with an intransitive value function if the voting is done in a pair-wise, ordinal manner. That is, "rational" decision makers become "irrational" as a group, casting doubt on design processes involving group interactions. Friedman_2,3 asserts the problem is even worse, and provides examples showing that even single rational decision makers can inadvertantly fall into intransitive value functions by using probabilistic simulations. The good news is that these intransitive value functions can always be made transitive -- or rational -- by abandoning pair-wise, ordinal comparisons in favor of global, cardinal comparisons.

One of the purposes of this talk is to attempt to illuminate key issues in decision-based design from the perspective of the older -- but not yet mature -- disciplines of risk management and systems engineering. Both these intimately related disciplines strongly urge its practioners to employ the "systems view" -- that is, use quantitative global optimization rather than qualitative local optimization. Essentially, global choices over cardinal measures are always preferred instead of local, or pair-wise choices over ordinal measures.

usc237(B)

A military commander in charge of the development of new weapon systems examines four proposals from the leading aerospace contractors. His operations research expert has defined a system-level measure of effectiveness for any defined weapons system which is a function of the two major operational variables which are out of his control: a) the battlefield environment that will provide the stage for the weapon system and b) the skill level that will be available to operate the weapon system when the battles occur years hence. Thus, E = f(environment, skill).

Two levels of skill (a and b) and three classes of battlefield environment (x, y and z) are defined and the operations research group computed a weapons systems effectiveness for each of the four contractors as a function of the six possible combinations skill and environment. The results of these computations are summarized in Figure 1.

Fig 1. Scalar Effectivenesses as a Function of Contractor, Skill and Environment

The operations analysts estimate that the joint likelihoods of any skill level and any battlefield environment are equal. The commanding officer’s decision theory specialist strongly recommends that the "best" contractor be decided on pairwise comparisons of E_i across all the circumstances of skill and environment. Specifically, the consistant preference criterion that should be applied to all these pairwise comparisons is to simulate a battle engagement between each pair of contractor’s proposed systems and note how many times a given weapon system wins or loses, based on the E_i for each of the six circumstances of skill and evironment. "OK, Mr. Rational Man," says the CO, agreeably.

"Well, the first contractor clearly beats the second, since his system has a higher effectiveness in four of the six combinations of skill and environment," begins the CO. "The second contractor is preferred over the third since, again, it is superior in four of the six instances, and the third contractor also beats the fourth in four of six instances. It clearly appears that the overall winner is the first contrac --- wait a minute! The fourth contractor beats the first one four out of six times too!!! This is insane."

"Looks as if we’ve got ourselves into a little intransitive loop," mutters the decision specialist tremulously, "and without even invoking Arrow’s Theorem. This is irrationality with even a single decision-maker who made only rational decisions."

"How can this happen? What is the best decision method now?" asks the CO somewhat less agreeably.

"I’ve never seen this behavior before. It must have something to do with the way the six E_i were chosen for each contractor. Perhaps there’s a strange coupling between the different contractors’ sextets of E_i so lets run the computations again, this time examining all 36 possible cases of the six E_i for one contractor and the six E_i for the next contractor."

"OK," agrees the CO cautiously, "perhaps that will eliminate any clever and malicious couplings." But the results still came out the same: 1st > 2nd > 3rd > 4th > 1st .....! "This is crazy," blurts the decision specialist. "Somebody’s been monkeying with the analysis to get such anti-intuitive results. Surely these number sets can’t mean anything in the real world!"

Martin Gardiner was nearby, listening to this conversation. "Obviously, you gentlemen have not read my article in the December 1970 issue of Scientific American (p. 110)," he beams. "In it I describe the apparent paradox of intransitive dice which were designed by Bradley Efrom of Stanford. Perfectly balanced, six-sided dice are designed with a different set of integers on each face, not just the standard one through six. Four dice were designed with their integers exactly as shown in Figure 1, above. Die A consistently beat B, B beat C, C beat D, and D beat A (!), all with a probability of 2/3. Even experienced gamblers and mathematicians just couldn’t believe it! This unexpected intransitivity seems to go against the intellectual grain of most people who are used to and feel comfortable in dealing with only transitive relations. So, these numbers and this intransitive behavior are very much "real world"; they’re just not abstract mathematical creations."

"In fact, many other sets of integers display the same property, and I show a few in my article.

If we open up the variables to permit more than six faces per die and more than four dice, then the maximum advantage of each die over the next increases from 2/3 to a maximum of 3/4."

The above scenario can be repeated for analogous problems in the industrial world -- deciding on an optimum manufacturing system with uncertainties in customer demand and key material availability -- and in the sports world -- deciding on an optimum set of choices in a player draft with uncertainties in the next coach and salary caps.

Back to the CO and decision specialist. "I’ve decided not to employ pair-wise decision making." says the CO. "I’m not as interested in one-on-one jousts between the contractors as I am in providing the most effective weapon system for our fighting forces. Therefore, I’m going to choose contractor three because he promises to produce the highest expected value of effectiveness across all the environmental circumstances. This is my true transitive preference."

"Yes sir," the decision specialist quietly responds, "that seems rational enough. I just felt that in complex situations with many dimensions and interactions, it was easier to decompose the issues into smaller, more comprehensible pieces and make sensible decisions. The final decision can then be made by an aggregation of the smaller decisions."

"That’s a dogma of systems engineering," comments the CO. "It’s really a matter of faith that a complex system can be decomposed, its pieces acted upon, and then sensibly reconstructed. However, in many cases, we miss too many of the interactions through simplification which we convince ourselves is absolutely necessary to attain any level of understanding at all. In these cases, both the decompositions and reconstructions cause serious problems with the accuracy and fidelity of our models. In other cases, such as insisting on the simplification of pairwise decisions rather than global decisions, we end up with anti-intuitive preference sequences which we arrogantly label, ‘irrational’."

"You’re possibly correct," says the specialist, "even Arrow’s Theorem simplifies decision making...."

The CO interrupts, "What did you say your name was?"

"John Smyth, sir."

"Indeed? Are you related to the John Smyth who was the 16th century English military technology expert?"

"Why, yes! He was my great^x-grandfather. How did you know of him?"

"From my studies in the history and evolution of weapons. In 1591, Sir John Smyth gave an impassioned presentation to the privy council claiming that the best weapon for the English infantry was the bow and arrow, because it possessed higher firepower, was more accurate, less sensitive to wet weather and presented less danger to the operator than the "upstart’ technology of firearms."

"Interesting. But how does this relate...?"

"Your family seems to have an obsession with Arrows."