RISKS-LIST: RISKS-FORUM Digest Monday 8 May 1989 Volume 8 : Issue 68 FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Low-Probability / High-Consequence Accidents -- and the Midland 737? (PGN) "Probing Boeing's crossed Connections" (Werner Uhrig) An Atlantis spacecraft computer problem resolved nicely (PGN) "Life's Risks: Balancing Fear Against Reality of Statistics" (Marc Rotenberg, Jerry Leichter) Hear No Evil (Kevin Driscoll) Computer Ethics Course/Resource Volunteers Wanted (long) (Bob Barger) ---------------------------------------------------------------------- Date: Mon, 8 May 1989 8:34:12 PDT From: Peter Neumann Subject: Low-Probability / High-Consequence Accidents -- and the Midland 737? I would like to consider here a class of problems that has not been addressed specifically in RISKS, although its components are familiar. The RISKS Forum has addressed alarm systems that could not adequately be debugged under truly real circumstances. There was also the example of the earliest Antarctic ozone depletion data, which was systematically rejected by the analysis program for being *too* anomalous. The potential for a combination of these two types of problems might occur in aircraft monitoring during flight, as follows. Sensitive sensors in hostile environments (such as engines) sometimes report unrealistic or off-scale readings due to noise or interference. Consequently software monitoring the sensor may be programmed to ignore values beyond a certain threshold, on the grounds that such extreme readings must be the results of extraneous events. If the ignored sensor reading was "real", however, other more remote sensors might pick up -- and accept -- less extreme readings. This appears to be a potential problem in a variety of control systems. In the absence of any definitive information about the British Midland 737 crash, such a hypothesis seems just as plausible as any other. The *left* engine was reportedly vibrating wildly (possibly due to a broken fan blade), but the pilots for some reason(s) shut off the (good) right engine. The extreme vibration in the left engine might indeed have produced hitherto unexperienced sensor readings that designers -- or the software folks -- felt would have to be impossible. The vibration from the left engine would have been transmitted -- much attenuated -- through the entire airframe, and might have been reported at a much more "reasonable" intensity by the vibration sensors of the right-hand engine. It does not take much of a leap in imagination for the computer program to conclude that it was the *right* engine that was malfunctioning. In any event, this possible fault mode represents another case of LOW-PROBABILITY / HIGH-CONSEQUENCE ACCIDENTS [1], and thus deserves explicit attention. Unfortunately it is just one more such combinatory fault mode. [1] See Koshland's editorial (title above, in CAPS) in Science, vol 244 no 4903, 28 April 1989, p. 405, discussing the Exxon Valdez spill and conclusions that should be drawn from it. ------------------------------ Date: Mon, 8 May 1989 4:53:45 CDT From: Werner Uhrig Subject: ``Probing Boeing's crossed connections'' [The title is that of an article in IEEE Spectrum, May 1989, pp. 30-35, subtitled ``Misconnected circuits and hoses found on 94 in-service Boeing aircraft raise concern about design, test, and maintenance of aircraft safety systems''. Author is Karen Fitzgerald.] At the very end of the article is a further reference of interest to this group: For a minute-by-minute account of the British Midland crash from knowledge gathered to date, see Special Bulletin S2/89 of the Air Accidents Investigation Branch of the Department of Transport in Farnborough, England, March 20, 1989. [I recommend the Spectrum article, and would like to see the Bulletin. PGN] ------------------------------ Date: Mon, 8 May 1989 12:13:25 PDT From: Peter Neumann Subject: An Atlantis spacecraft computer problem resolved nicely One of Atlantis' main computers (one of the processors in the two pairs of the 2x2 + 1 backup architecture) failed on 7 May. For the first time ever the astronauts made repairs -- in this case by substituting a spare processor. It took them about 3.5 hours to gain access to the computer systems by removing a row of lockers on the shuttle mid-deck, and another 1.5 hours to check out the replacement computer. It is ironic that such a replacement was so difficult, but not surprising. My old friend Al Hopkins, who at MIT Instrumentation Lab (now Draper Lab) designed the Apollo on-board guidance computer, told me years ago how carefully they had planned the packaging so that the astronauts would be able to make repairs on the fly (as it were). NASA officials would have none of it, and buried the computer several layers underneath other equipment. Apparently that tradition has continued. Perhaps the success of the Atlantis crew will change things. During STS-9, Nov-Dec 83, multiple primary computers on the Columbia failed at the same time, and delayed the return to earth. On one hand, the calculations say that losing three processors would be a rare event. However, here we have another example of a low-probability / high-consequence accident -- especially if it involved the backup and one of each of the pairs. Furthermore, since the software is the same in all four of 2x2 the main processors, they would all have failed consistently, and been deemed correct. (And we just reported the serious problem in the Magellan software caught before Atlantis' launch, noted in RISKS-8.67!) In the case of pairwise disagreement among both pairs, there is always the fifth, backup, computer, separately programmed. As far as I know, the shuttles have never had to rely on the backup computer software, so it might be preferable to make processor replacements among the main four rather than resort to the backup... ------------------------------ Date: Mon, 8 May 89 12:14:37 -0700 From: mrotenberg@cdp.uucp [Marc Rotenberg] Subject: "Life's Risks: Balancing Fear Against Reality of Statistics" Excerpted from today's New York Times: Is the slight risk of contracting cancer from Alar too high a price to pay for crisper apples? Is the dramatic increase in milk production available through genetically engineered growth hormones worth the unknown risk to children's health? If a few aging aircraft suffer explosive decompressions, should all old airlines be grounded? Risks to health and safety and the complex questions of public policy they create are seemingly everywhere these days. And while there is little statistical evidence that the hazards of daily life are on the rise, a wide range of academic and business experts believe that American's perception of increased peril is stifling technology, wasting billions of dollars, and, ironically, making it more difficult to contain the most serious risks. ... by broad statistical measures, Americans have never been safer ... Even the high-profile threats have not changed the risks of untimely death or injury. The skies may be crowded, the planes aging and the pilots inexperienced, but the trend in aircraft fatalities is downward. ... Life-saving medicines have been less dramatically affected, but even here, the measures to compensate for risk can radically change the economic of distribution ... The Environmental Protection Agency also regards itself as handicapped by Congressional and public misperception of relative risk. ... What explains the public's decreasing tolerance of some risks and apparent indifference to others? ... perceived risk is not always related to the probability of injury. Easily tolerated risks include ones that people can choose to avoid (chain saws, skiing), that are familiar to those exposed (smoking), or that have been around for a long term (fireworks). Poorly tolerated risks are involuntary (exposure to nuclear waste), have long delayed effects (pesticides), or unknown effects (genetic engineering). ... nuclear and chemical technologies fare especially badly in such subjective rankings. Indeed the general acceleration of technical change and integration of new technology in products helps to explain the increase in public anxiety about risk. ... ------------------------------ Date: Mon, 8 May 89 17:17 EDT From: "Jerry Leichter (LEICHTER-JERRY@CS.YALE.EDU)" Subject: Life's Risks ... Today's New York Times (Monday 8 May) has a front-page article title "Life's Risks: Balancing Fear Against Reality of Statistics". It's the first of two articles on "risk and public policy". The article is ... well worth reading. Here's an interesting quotation: Peter W. Huber, and engineer, lawyer and author of "The Legal Revolution and its Consequences" notes that ... "safety taxes" [extra costs charged by suppliers to pay for potential lawsuits] are added to the price of thousands of ... goods and services, distorting production and reducing living standards. By Mr. Huber's reckoning, the safety tax represents 30 percent of the cost of a step ladder, one-third the cost of a ride on a Long Island tour bus and $300 of the cost of giving birth in New York City. -- Jerry ------------------------------ Date: 7 May 89 22:44:01 GMT From: driscoll@draco.src.honeywell.com (Kevin Driscoll) Subject: Hear No Evil On a recent flight, the cabin crew was a bit late in starting the in-flight movie. The flight took less time than expected, so the movie's climactic showdown scene began just after plane touched down. Many of the passengers became noticeably irritated at the flight attendants pre- and post-landing announcements which interrupted the movie's audio. This was a tow-in gate so the engines were shut down well before arriving at the gate. Without engine power, an APU supplies electrical power. On the switch-over, however, the power glitch reset the audio channel controllers to the default channel (8) which is silent. It is common on commercial aircraft to have "unimportant" control systems (such as the individual seat lighting and audio) reset on power glitches. This is not a safety problem. Is it? When the audio went dead on this flight, most of the passengers didn't know what happened and pushed their flight attendant call buttons. Same of the more irate passengers repeatedly pushed it, causing the alert tone to sound almost continuously. (This was what I could see in first class. I can only imagine what was happening in the coach cabin where passengers had to explicitly pay extra for headsets and where there were more passengers.) I would suspect that the official justification for the flight attendant call button system is to alert the crew to emergencies. During this incident, any signaling of an emergency would not have been noticed. I also suspect that a failure analysis of the audio system did not foresee the implications of a power glitch reseting the channel. This is an example of the most common reason for safety problems; the designers don't see all the possible circumstances that the design will face, particularly where people are involved. The fix to this problem is trivial; make the default channel one with some material on it, preferably one of the movie channels (1 through 4). I wonder if the current design was to save some small amount of power. Another disconcerting observation was that the cabin crew did not seem to understand what had happened either. They seemed unable to help the passengers. They made repeated visits to the passengers who contined to re-press their call buttons. All that had to be done was to switch the channel back to where it had been. Disclaimer: I don't represent Honeywell, neither should Don Dodgen. Kevin R. Driscoll, Principal Research Scientist (612) 782-7263 FAX: -7438 POST: Honeywell M/S MN65-2500; 3660 Technology Drive; Mpls, MN 55418-1006 ------------------------------ Date: Wed 03 May 1989 13:51 CDT From: Bob Barger Subject: Computer Ethics Course/Resource Volunteers Wanted Two drafts of the following course were previously printed in RISKS digests. These brought a host of suggestions from readers. Almost all these suggestions were incorporated into the final version below. Volunteers are now being sought to participate in the course this Fall (see Section 3. b. 2. below). These volunteers could contribute items relating to computer ethics for posting on the class bulletin board, correspond by e-mail with individual students on course topics, and/or comment on students' postings on the class bulletin board. The course will run from late August to early December. No money is presently available as compensation for this service, but I will gladly contribute letters of appropriate recognition for those who participate as resource persons in all or part of the course. If interested, send a brief "vita" to Bob Barger at CFRNB@ECNCDC.BITNET. SENIOR SEMINAR EASTERN ILLINOIS UNIVERSITY 1. Catalog Description a. Course Number: EIU 4050 b. Title: Computer Ethics c. Credit: 2-0-2 [2 hrs per week/one semester] d. Term to be offered: On Demand e. Short title: Computer Ethics f. Course Description: The course will investigate current ethical issues involving computers. While it is not a "computer course," students will make frequent use of postings on the electronic bulletin board of the ECN mainframe computer to research and discuss ethical issues. g. Prerequisites: 75 Semester Hours and previous experience with computers. [Class size limit = 15 students for Fall, 1989, semester]. h. Exclusions: None. 2. Outline of topics : Week Topic 1 Orientation to the course (introduction, explanation of course content, class procedures, and evaluation methodology). Consideration of ethical theory: examination of the metaphysical bases and resultant ethical norms of the idealist and naturalist theories. 2 Consideration of ethical theory (continued): examination of the metaphysical bases and resultant ethical norms of the consequentialist and existentialist theories. 3 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 4 Consideration of professional ethics: responsibilities between employer/employee, client/professional, professional/peer, and professional/society. 5 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 6 Consideration of liability for software design, manufacture, and use: legal liability; truth-in- advertising; contracts; warranties; software as product or service? 7 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 8 Consideration of privacy issues: individual privacy rights; institutional "right-to-know" concerns; system security concerns; data-banking concerns. 9 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 10 Consideration of power/control issues: the computer as agent of centralization or decentralization? the computer as agent of conservation or change? the computer as agent of alienation? 11 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 12 Consideration of ownership and theft issues: copyrights; fair usage; patents; trade secrecy and competition; considerations unique to the computer market. 13 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 14 On-line reading of the "Discussion of Ethics in Computing" list, the "Forum on Risks to the Public in Computers and Related Systems" digest, and the "Computers and Society" list (all are available on the ECN bulletin board); written reactions to these readings, and written commentary on other students' reactions. [The instructor will insure that these activities equate to the activities of a traditional two hour class meeting]. 15 Seminar members will reconvene as a group for the last meeting to allow for group reflection on the seminar experience and course evaluation. Exam week Final examination Writing component Students will type thirteen 30-to-50 line (i.e., one-to-two page) reactions to the on-line electronic bulletin board readings. Students will "post" these reactions (i.e., electronically send them to the mainframe computer bulletin board set aside for members of this seminar). In their reactions, students will: 1) identify the particular publication or publications to which they are reacting, 2) identify the particular issue or issues raised in the publication(s), 3) identify the ethical implications of the issue or issues, 4) identify the ethical paradigm used by the author, 5) add their own reasons for agreement or disagreement with the viewpoint of the publication's author, 6) and, finally, offer an alternative solution or viewpoint to that presented by the author, or present other appropriate considerations not raised by the author or covered in their own (i.e., the student's own) previous comments. The instructor will send weekly, by confidential electronic mail, a grade on the student's posted reaction, together with whatever comments the instructor thinks helpful. The student's original posted reaction will also be open to public comment by the other students in the seminar [this is accomplished by posting notes to the bulletin board, referencing the original posted reaction]. These latter comments by the other students in the seminar will be considered along with classroom discussion in computing the "participation" factor of the student's semester grade. Evaluation Each student's semester grade for the seminar will be calculated according to the following weighted formula: - 13 posted reactions (at 5% each) = 65% - Participation (based on class discussion and posted comments on other students' reactions) = 20% - Final Exam = 15% 3. Implementation : a. This course will be taught by: Robert N. Barger, Ph.D. b. Materials in the course will include: 1) Texts: a) Deborah Johnson, Computer Ethics (Englewood Cliffs, NJ: Prentice Hall, 1985) b) Notes on Systematic Philosophies from Dr. Barger's Philosophy 1800 class (furnished without charge to seminar members) c) Postings on the above-mentioned ECN electronic bulletin board lists. 2) Resource people: Computer professionals (e.g., administrators, systems analysts, programmers, etc.) will be utilized as guest contributors to the class. This will be accomplished by personal appearances, as well as by electronically mediated conferencing (e.g., postings, e- mail, relay round-tables, etc.). c. Exceptional costs: None, unless the student wishes to use a modem to access the computer. In this case the student will be responsible for any personal equipment costs and/or long distance phone charges. d. Effective date: Fall, 1989. Date approved by Senior Seminar Committee: February 24, 1989. Date approved by Council on Academic Affairs: April 20, 1989. ------------------------------ End of RISKS-FORUM Digest 8.68 ************************ -------