Brydon Dewitt

Jared Schmidt
• Types of errors and failures
• Examples of failures
• Causes of system failures
• Laws and regulations
• Professional responsibility
• Risks
 Risk vs. Reward
 What is acceptable?
 Who should make such determinations?
1. Cause

2. Seriousness of effects
o Individual vs large scale effects
3. Application area
o Safety-critical applications (e.g. Hospital machines)
Errors with billing systems
• One IL couple was billed for $68 billion in tax
penalties because the IRS generated erroneous bills
after modifying its billing program.
• Inaccurate and misinterpreted database
information
• Mistakenly being on the federally sex offender list
because you live somewhere where a sex offender
previously lived

o Better testing methods can make these

programming errors avoidable.
1) Large population
 Poor database design
2) Automated processing
 Inability to recognize special cases
3) Overconfidence in data accuracy
 Unrealistic or inadequate understanding of risks in a
complex system
4) Failure to update and correct data
5) Lack of accountability for errors
• Modern communications, power grids, medical,
financial, retail, and transportation systems rely
heavily on computer systems.
o Communications systems
o Transportation systems
o Stock market

 Adequate planning and backup provisions

need to be made in case of failures.
• Several companies have gone bankrupt after
investing in computer systems that failed.

 A few dozen companies that bought an inventory

system called Warehouse Manager which reported
incorrect data and did not place orders correctly.

 Despite receiving numerous complaints from several

clients, the company dishonestly said the problems
were unique to the customers.
 Florida voters in 2000 were not able to vote
because their names matched those in a
database of felons.

 Help America Vote Act (2002)

 Authorized $3.8 billion to improve voting systems

• Voting machines still failed because of common

problems:
o Insufficient planning, testing, and training
o Security issues
 Many systems are trashed before they ever
fully conceived.
 Require more money and time than anticipated

 Ford Motor purchasing system ($400 million)

 Hotel and rental car businesses ($125
million)
 FBI Virtual Case File system ($170 million)
 IRS tax modernization plan ($4 BILLION)

• About $1 trillion spent worldwide on

“hopelessly inadequate” projects.
• Complex $193 million baggage handling
system did not work as planned
• Delayed the opening of the airport for a year
• Cost $30mil/month in bond interest and
operating costs
o Insufficient time for development and testing
 Denver changed design specifications after the project
began.
 Legacy systems are reliable, but inflexible.
 Often used by banks, airports, government
agencies, and infrastructure services
o Implementing a new system would be expensive and
disruptive

Y2K Problem
• Two-digit representation of year caused errors in
many systems
o $308 billion spent worldwide ($416 billion in 2012 USD)
 Military, power plant, aircraft, medical
applications
 Between 1988-1993, four A320s crashed.
o Too much confidence in “fly-by-wire” autopilot program
 Traffic control systems
o Must work in real time
o Many different devices and systems working together
 Computer systems fail for two general reasons:

1) The task they are intended for is inherently

difficult
o Errors and ambiguity in program specifications

2) The job is done poorly

o Not considering how the system interacts with real users
or real world problems
o Dishonest reliability or safety estimates to cover up flaws
and avoiding expenses
 Good software engineering techniques at all
development stages are important.
 ACM/IEEE-CS Software Engineering Code of Ethics
and Professional Practice
 ACM Code of Ethics and Professional Conduct
(Appendix A)
• Software engineers who work on safety-critical
applications should have special training.
• Long, careful planning and good management
o Discover and modify unrealistic goals during design
 Well-designed user interfaces can avoid many
computer-related problems.
 Should be consistent
 Provide clear instructions and error messages
 Check input to avoid typos
 AA Flight 965 (1995) – 159 deaths
o Caused by overconfidence in the system and poor user
design
 Adequate, well-planned testing is critical.
 Unfortunately, testing is not always thoroughly
done because of budget or deadlines.
 This is foolish, risky, and irresponsible.
• Beta testing
 Testing by regular users in a real-world environment

 NASA’s independent verification and

validation practice
o Testing and software validation done by independent
company to find flaws
 Building fail-safes into systems to avoid errors
 Especially important when lives and fortunes
are at stake

 Even with extensive testing, there is no

guarantee of bug free code.
 Errors in tests and system recovery routines
 Lawsuits and settlements
 Many contracts for business computer systems
limit or waive the right for the consumer to
recover losses.
 Fraud and misrepresentation are not part of a contract.
• Liability and criminal laws in the US are
flawed and often abused.
 Well-designed laws should:
1) Not be so strict that they discourage innovation
2) Provide incentives to produce good systems
  Licensing agreements typically indicate software is
bought “as-is”.
 Some agreements also include provision that the
vendor may choose the states in which any legal
disputes are settled.
TWO VIEWS:

Agreements are binding Software should have mandatory

contracts warranties
Requirements for warranties would raise: Strict legal requirements for warranties would:

 Increase price of testing, development,  Encourage responsibility and lead to better

and insurance, hurting small companies software
 Reduce innovation and new software  Protect the consumer from unfair agreements
development
 Radiation-therapy machine
 Made by Atomic Energy of Canada Limited
(AECL)
 Used between 1985 and 1987
 Radiation overdoses
 Cancer patients
 Family/friends
 Medical centers
 Technicians handling the machine
 AECL
 FDA
 Section 1.2 Avoid harm to others.
 Section 2.5 Give comprehensive and thorough
evaluations of computer systems and their
impacts, including analysis of possible risks.
 Utilitarian
 Rights-based
 Justice-based
 Take machines offline until fixed AND provide
an alternative treatment
 Best attempt at satisfying all stakeholders'
rights
 Morally, “the right thing to do”
• Therac-25 bypassed rigorous FDA testing
o AECL declared “pre-market equivalence” and was
able to market the machines.

• Strict regulation can keep bad products out of

the market.
o Strict standards can inhibit progress.
o Responsibility would be shifted from the
manufacturer to the government.
o The regulation processes are expensive and
susceptible to bureaucracy.
 Some feel that software developers should be
required to obtain a mandatory license.
 Economic analyses have shown licenses have no
effect on improving quality.
 Opponents of mandatory licensing argue it violates
an individual’s negative right to work.
 Many companies strive to uphold an ethical
policy and pay for problems and damages.
 Intuit has paid interest and penalties for flawed
income-tax software.
• Business pressure can also be a good tool for
insistence on quality.
o Customer satisfaction is critical to success
o Good customer relations maintain a healthy company
reputation
• Are we too dependent on computers?
o Computers are tools; we are far better off with them
than without.

• Many failures stem from:

o Lack of responsibility
o Ignorance
o Overconfidence
 Baase, Sara. “A Gift of Fire: Social, Legal, and
Ethical Issues for Computing and the Internet.“
Upper Saddle River, NJ: Pearson, 2008. Print.
 Calleam Consulting. “Denver Airport Baggage
Handling System Case Study.” 2008. Web.
 Leveson, Nancy. “An Investigation of the
Therac-25 Accidents.” IEEE Computer, Vol. 26,
No. 7, July 1993, pp. 18-41.