Testing Sistem Informasi

Dendi Maysanjaya
Outline
• Software Crisis
• Testing Process
• Software Testing Terminologies
• Software Testing Limitations
Software Crisis
• Software organizations are regularly getting failure reports of their
products and this number is increasing day by day.
• A major problem of the software industry is its inability to develop
error-free software.
Some Software Failure
1. The Explosion of the Ariane 5 Rocket.
• The Ariane 5 rocket was designed by European Space Agency and it was launched on
June 4, 1996.
• It was an unmanned rocket and unfortunately exploded only after 40 seconds of its
take off from Kourou, French Guiana.
• The design and development took ten long years with a cost of $7 billion.
• Unfortunately, the navigation system of Ariane 4 was used in Ariane 5 without
proper testing and analysis.
• The failure of the Ariane 5 was caused by the complete loss of guidance and altitude
information, 37 seconds after start of the main engine ignition sequence (30 seconds
after lift-off).
• This loss of information was due to specification and design errors in the software of
the inertial reference system.
• Such design decisions and poor testing resulted in the explosion of Ariane 5.
Some Software Failure
2. The Y2K Problem
• It was simply the case of using two digits for the year instead of four digits.
For instance, 1965 was considered as 65.
• The developers could not imagine the problem of year 2000. What would
happen on January 1, 2000? The last two digits i.e. 00 may belong to any
century like 1800, 1900, 2000, 2100, etc.
• The simple ignorance or a faulty design decision to use only the last two
digits for the year resulted into the serious Y2K problem.
• Most of the software was re-tested and modified or discarded, depending
on the situation.
Some Software Failure
3. The USA Star-Wars Program
• ‘Patriot missile’ was the result of the USA ‘Star Wars’ program.
• This missile was used for the first time in the Gulf war against the Scud missile of
Iraq.
• Surprisingly, ‘Patriot missiles’ failed many times to hit the targeted Scud missile. One
of the failures killed 28 American soldiers in Dhahran, Saudi Arabia.
• An investigation team was constituted to identify the cause of failure.
• The team re-looked at every dimension of the product and found the reason for the
failure.
• The cause of the failure was a software fault. There was a slight timing error in the
system’s clock after 14 hours of its operation. Hence, the tracking system was not
accurate after 14 hours of operations and at the time of the Dhahran attack, the
system had already operated for more than 100 hours.
Some Software Failure
4. Failure of London Ambulance System
• The software controlling the ambulance dispatch system of London collapsed on October 26-
27, 1992 and also on November 4, 1992 due to software failures.
• The system was introduced on October 26, 1992. The London Ambulance Service was a
challenging task that used to cover an area of 600 square miles and handled 1500 emergency
calls per day.
• Due to such a failure, there was a partial or no ambulance cover for many hours. The position
of the vehicles was incorrectly recorded and multiple vehicles were sent to the same
location. Everywhere people were searching for an ambulance and nobody knew the reason
for non-arrival of ambulances at the desired sites. The repair cost was estimated to be £9m,
but it is believed that twenty lives could have been saved if this failure had not occurred.
• The enquiry committee clearly pointed out the administrative negligence and over-reliance
on ‘cosy assurances’ of the software company. The administration was allowed to use this
system without proper alternative systems in case of any failure.
• The committee also termed the possible cause of failure as [ANDE98, FINK93]: “When the
system went live, it could not cope with the volume of calls and broke under the strain. The
transition to a back-up computer system had not been properly rehearsed and also failed.”
Some Software Failure
5. USS Yorktown Incident
• The USS Yorktown - a guided missile cruiser was in the water for several hours
due to the software failure in 1998.
• A user wrongly gave a zero value as an input which caused a division by zero
error.
• This fault further failed the propulsion system of the ship and it did not move
in the water for many hours.
• The reason behind this failure was that the program did not check for any
valid input.
Some Software Failure
6. Accounting Software Failures
• Financial software is an essential part of any company’s IT infrastructure.
• However, many companies have suffered failures in the accounting system
due to errors in the financial software.
• The failures range from producing the wrong information to the complete
system failure.
• There is widespread dissatisfaction over the quality of financial software.
• If a system gives information in the incorrect format, it may have an adverse
impact on customer satisfaction.
Some Software Failure
7. Experience of Windows XP
• Charles C. Mann shared his views about Windows XP through his article in
technology review as: “Microsoft released Windows XP on October 25, 2001.
That same day, what may be a record, the company posted 18 megabyte of
patches on its website for bug fixes, compatibility updates, and
enhancements. Two patches fixed important security holes. Or rather, one of
them did; the other patch did not work. Microsoft advised (still advises) users
to back up critical files before installing patches.”
• This situation is quite embarrassing and clearly explains the sad situation of
the software companies. The developers were either too careless or in a great
hurry to fix such obvious faults.
Software Crisis
• Most of the software developers are confused about the quality of their
software products.
• If they are asked about the quality of software being produced by
companies, they generally say, “It is getting worse day by day.” It is as if we
say that Boeing’s planes produced in 2009 are less reliable than those
produced in 1980.
• The blame for software bugs belongs to nearly everyone. It belongs to the
software companies that rush products to market without adequately
testing them. It belongs to the software developers who do not understand
the importance of detecting and removing faults before customers
experience them as failures. It belongs to a legal system that has given the
software developers a free pass on error-related damages. The blame also
belongs to universities that stress more on software development than
testing
Testing Process
• What is Software Testing?
• Why Should We Test?
• Who Should We Do the Testing?
• What Should We Test?
Testing Process
• Testing is an important aspect of the software development life cycle. It is
basically the process of testing the newly developed software, prior to its
actual use.
• The program is executed with desired input(s) and the output(s) is/are
observed accordingly. The observed output(s) is/are compared with
expected output(s). If both are same, then the program is said to be correct
as per specifications, otherwise there is something wrong somewhere in
the program.
• Testing is a very expensive process and consumes one-third to one-half of
the cost of a typical development project. It is largely a systematic process
but partly intuitive too.
• Hence, good testing process entails much more than just executing a
program a few times to see its correctness.
What is Software Testing?
• Good testing entails more than just executing a program with desired
input(s).
Program ‘Minimum’ to find the smallest integer out of a set of integer
Inputs and outputs of the program ‘Minimum’
False Definitions of Testing
i. Testing is the process of demonstrating that errors are not present.
ii. The purpose of testing is to show that a program performs its intended functions
correctly.
iii. Testing is the process of establishing confidence that a program does what it is
supposed to do.
• The philosophy of all three definitions is to demonstrate that the given
program behaves as per specifications.
• We may write 100 sets of inputs for the program ‘Minimum’ and show that
this program behaves as per specifications.
• However, all three definitions are not correct. They describe almost the
opposite of what testing should be viewed as. Forgetting the definitions for
the moment, whenever we want to test a program, we want to establish
confidence about the correctness of the program. Hence, our objective
should not be to show that the program works as per specifications.
Definitions of Testing
• But, we should do testing with the assumption that there are faults and our
aim should be to remove these faults at the earliest.
• Thus, a more appropriate definition is “Testing is the process of executing a
program with the intent of finding faults.”
• Human beings are normally goal oriented. Thus, establishment of a proper
objective is essential for the success of any project. If our objective is to
show that a program has no errors, then we shall sub-consciously work
towards this objective. We shall intend to choose those inputs that have a
low probability of making a program fail. On the contrary, if our objective is
to show that a program has errors, we may select those test cases which
have a higher probability of finding errors. We shall focus on weak and
critical portions of the program to find more errors. This type of testing will
be more useful and meaningful.
Some critical/typical situations of the program ‘Minimum’

TIS - STIKI - 2016

Some critical/typical situations of the program ‘Minimum’
Possible reasons of failures for all nine cases
Possible reasons of failures for all nine cases
Reasons for observed output

TIS - STIKI - 2016

Modified program ‘Minimum’ to find the smallest integer out of a set of integers

TIS - STIKI - 2016

Results of modified program ‘Minimum’
Results of modified program ‘Minimum’
Final program ‘Minimum’ to find the smallest integer out of a set of integers
Conclusions for Definition of Software Testing

• Goal of software testing is to find critical situations of any program.

• Test cases shall be designed for every critical situation in order to make the
program fail in such situations. If it is not possible to remove a fault then
proper warning messages shall be given at proper places in the program.
• The aim of the best testing person should be to fix most of the faults.
• This is possible only if our intention is to show that the program does not
work as per specifications.
• Hence, as given earlier, the most appropriate definition is “Testing is the
process of executing a program with the intent of finding faults.”
• Testing never shows the absence of faults, but it shows that the faults are
present in the program.
Why Should We Test?
• Software testing is a very expensive and critical activity; but releasing
the software without testing is definitely more expensive and
dangerous.
• The programs are growing in size and complexity. The most common
approach is ‘code and fix’ which is against the fundamental principles
of software engineering. Watts S. Humphrey, of Carnegie Mellon
University conducted a multiyear study of 13000 programs and
concluded that “On average professional coders make 100 to 150
errors in every thousand lines of code they write.”
Phase Wise Cost of Fixing an Error

TIS - STIKI - 2016

Who Should We Do the Testing?
• Testing a software system may not be the responsibility of a single
person. Actually, it is a team work and the size of the team is
dependent on the complexity, criticality and functionality of the
software under test.
Testing Persons
• The testing persons must be cautious, curious, critical but non-
judgmental and good communicators. One part of their job is to ask
questions that the developers might not be able to ask themselves or
are awkward, irritating, insulting or even threatening to the
developers. Some of the questions are:
I. How is the software?
II. How good is it?
III. How do you know that it works? What evidence do you have?
IV. What are the critical areas?
V. What are the weak areas and why?
VI. What are serious design issues?
VII. What do you feel about the complexity of the source code?
Testing Persons
• The testing persons use the software as heavily as an expert user on
the customer side. User testing almost invariably recruits too many
novice users because they are available and the software must be
usable by them.
• The problem is that the novices do not have domain knowledge that
the expert users have and may not recognize that something is
wrong.
Testing Persons
• Many companies have made a distinction between development and
testing phases by making different people responsible for each phase. This
has an additional advantage.
• Faced with the opportunity of testing someone else’s software, our
professional pride will demand that we achieve success. Success in testing
is finding errors. We will therefore strive to reveal any errors present in the
software. In other words, our ego would have been harnessed to the
testing process, in a very positive way, in a way, which would be virtually
impossible, had we been testing our own software.
• Therefore, most of the times, the testing persons are different from
development persons for the overall benefit of the system. The developers
provide guidelines during testing; however, the overall responsibility is
owned by the persons who are involved in testing.
Persons and Their Roles During Development and Testing
What Should We Test?
• Is it possible to test the program for all possible valid and invalid
inputs? The answer is always negative due to a large number of
inputs.
What Should We Test?
• We may like to test a program for all possible valid and invalid inputs and
furthermore, we may also like to execute all possible paths; but practically, it is
quite difficult.
• Every exit condition of a branch statement is similarly difficult to test due to a
large number of such conditions.
• We require effective planning, strategies and sufficient resources even to target
the minimum possible bottom line. We should also check the program for very
large numbers, very small numbers, numbers that are close to each other,
negative numbers, some extreme cases, characters, special letters, symbols and
some strange cases.
Control flow graph of a 10 to 20 statement program
Terminologies
1. Program and Software
2. Verification and Validation
3. Fault, Error, Bug and Failure
4. Test, Test Case and Test Suite
5. Deliverables and Milestones
6. Alpha, Beta and Acceptance Testing
7. Quality and Reliability
8. Testing, Quality Assurance and Quality Control
9. Static and Dynamic Testing
10. Testing and Debugging
1. Program and Software
• Both terms are used interchangeably, although they are quite
different. The software is the superset of the program(s). It consists of
one or many program(s), documentation manuals and operating
procedure manuals.

TIS - STIKI - 2016

Documentation Manuals
• The program is a combination of source code and object code. Every
phase of the software development life cycle requires preparation of
a few documentation manuals. These are very helpful for
development and maintenance activities.

TIS - STIKI - 2016

Operating Procedure Manuals / Documents

• Operating procedure manuals consist of instructions to set up, install,

use and to maintain the software.
2. Verification and Validation
• Testing = Verification + Validation
• Both are essential and complementary activities of software testing. If
effective verification is carried out, it may minimize the need of validation
and more number of errors may be detected in the early phases of the
software development. Unfortunately, testing is primarily validation
oriented.
• The Institute of Electrical and Electronics Engineers (IEEE) has given
definitions which are largely accepted by the software testing community.
Verification is related to static testing which is performed manually. We
only inspect and review the document. However, validation is dynamic in
nature and requires the execution of the program.
Verification
• Verification: As per IEEE [IEEE01], “It is the process of evaluating the system
or component to determine whether the products of a given development
phase satisfy the conditions imposed at the start of that phase.”
• We apply verification activities from the early phases of the software
development and check / review the documents generated after the
completion of each phase.
• Hence, it is the process of reviewing the requirement document, design
document, source code and other related documents of the project.
• This is manual testing and involves only looking at the documents in order
to ensure what comes out is what we expected to get.
Validation
• Validation: As per IEEE [IEEE01], “It is the process of evaluating a
system or component during or at the end of development process to
determine whether it satisfies the specified requirements.”
• It requires the actual execution of the program.
• It is dynamic testing and requires a computer for execution of the
program.
• Here, we experience failures and identify the causes of such failures.
3. Fault, Error, Bug and Failure
• All terms are used interchangeably although error, mistake and defect are synonyms in
software testing terminology.
• When we make an error during coding, we call this a ‘bug’. Hence, error / mistake /
defect in coding is called a bug.
• A fault is the representation of an error where representation is the mode of expression
such as data flow diagrams, ER diagrams, source code, use cases, etc.
• If fault is in the source code, we call it a bug.
• A failure is the result of execution of a fault and is dynamic in nature. When the expected
output does not match with the observed output, we experience a failure.
• The program has to execute for a failure to occur. A fault may lead to many failures.
• A particular fault may cause different failures depending on the inputs to the program.
4. Test, Test Case and Test Suite
• Test and test case terms are synonyms and may be used interchangeably.
• A test case consists of inputs given to the program and its expected outputs. Inputs may
also contain pre-condition(s) (circumstances that hold prior to test case execution), if any,
and actual inputs identified by some testing methods. Expected output may contain
post-condition(s) (circumstances after the execution of a test case), if any, and outputs
which may come as a result when selected inputs are given to the software. Every test
case will have a unique identification number. When we do testing, we set desire pre-
condition(s), if any, given selected inputs to the program and note the observed
output(s). We compare the observed output(s) with the expected output(s) and if they
are the same, the test case is successful. If they are different, that is the failure condition
with selected input(s) and this should be recorded properly in order to find the cause of
failure.
• A good test case has a high probability of showing a failure condition. Hence, test case
designers should identify weak areas of the program and design test cases accordingly.
Test case template
• The set of test cases is called a test suite. We may have a test suite of
all test cases, test suite of all successful test cases and test suite of all
unsuccessful test cases.
• Any combination of test cases will generate a test suite. All test suites
should be preserved as we preserve source code and other
documents. They are equally valuable and useful for the purpose of
maintenance of the software. Sometimes test suite of unsuccessful
test cases gives very important information because these are the test
cases which have made the program fail in the past.
5. Deliverables and Milestones
• Different deliverables are generated during various phases of the
software development. The examples are source code, Software
Requirements and Specification document (SRS), Software Design
Document (SDD), Installation guide, user reference manual, etc.
• The milestones are the events that are used to ascertain the status of
the project. For instance, finalization of SRS is a milestone;
completion of SDD is another milestone. The milestones are essential
for monitoring and planning the progress of the software
development.
6. Alpha, Beta and Acceptance Testing
• Customers may use the software in different and strange ways. Their
involvement in testing may help to understand their minds and may force
developers to make necessary changes in the software. These three terms
are related to the customer’s involvement in testing with different
meanings.
• Acceptance Testing: This term is used when the software is developed for a
specific customer. The customer is involved during acceptance testing.
He/she may design ad-hoc test cases or well-planned test cases and
execute them to see the correctness of the software. This type of testing is
called acceptance testing and may be carried out for a few weeks or
months. The discovered errors are fixed and modified and then the
software is delivered to the customer.
Alpha and Beta Testing
• Alpha and Beta Testing: These terms are used when the software is developed as
a product for anonymous customers. Therefore, acceptance testing is not
possible. Some potential customers are identified to test the product.
• The alpha tests are conducted at the developer’s site by the customer. These
tests are conducted in a controlled environment and may start when the formal
testing process is near completion. The beta tests are conducted by potential
customers at their sites. Unlike alpha testing, the developer is not present here. It
is carried out in an uncontrolled real life environment by many potential
customers. Customers are expected to report failures, if any, to the company.
These failure reports are studied by the developers and appropriate changes are
made in the software. Beta tests have shown their advantages in the past and
releasing a beta version of the software to the potential customer has become a
common practice. The company gets the feedback of many potential customers
without making any payment. The other good thing is that the reputation of the
company is not at stake even if many failures are encountered.
7. Quality and Reliability
• Software reliability is one of the important factors of software quality. Other factors are
understandability, completeness, portability, consistency, maintainability, usability, efficiency, etc.
These quality factors are known as non-functional requirements for a software system.
• Software reliability is defined as “the probability of failure free operation for a specified time in a
specified environment”. Although software reliability is defined as a probabilistic function and
comes with the notion of time, it is not a direct function of time. The software does not wear out
like hardware during the software development life cycle. There is no aging concept in software
and it will change only when we intentionally change or upgrade the software.
• Software quality determines how well the software is designed (quality of design), and how well
the software conforms to that design (quality of conformance).
• Some software practitioners also feel that quality and reliability is the same thing. If we are
testing a program till it is stable, reliable and dependable, we are assuring a high quality product.
Unfortunately, that is not necessarily true. Reliability is just one part of quality. To produce a good
quality product, a software tester must verify and validate throughout the software development
process.
8. Testing, Quality Assurance and Quality Control
• Most of us feel that these terms are similar and may be used interchangeably. This creates
confusion about the purpose of the testing team and Quality Assurance (QA) team. As we have
seen in the previous section, the purpose of testing is to find faults and find them in the early
phases of software development. We remove faults and ensure the correctness of removal and
also minimize the effect of change on other parts of the software.
• The purpose of QA activity is to enforce standards and techniques to improve the development
process and prevent the previous faults from ever occurring. A good QA activity enforces good
software engineering practices which help to produce good quality software.
• The QA group monitors and guides throughout the software development life cycle. This is a
defect prevention technique and concentrates on the process of the software development.
Examples are reviews, audits, etc.
• Quality control attempts to build a software system and test it thoroughly. If failures are
experienced, it removes the cause of failures and ensures the correctness of removal. It
concentrates on specific products rather than processes as in the case of QA. This is a defect
detection and correction activity which is usually done after the completion of the software
development. An example is software testing at various levels.
9. Static and Dynamic Testing
• Static testing refers to testing activities without executing the source
code. All verification activities like inspections, walkthroughs, reviews,
etc. come under this category of testing. This, if started in the early
phases of the software development, gives good results at a very
reasonable cost.
• Dynamic testing refers to executing the source code and seeing how it
performs with specific inputs. All validation activities come in this
category where execution of the program is essential.
10. Testing and Debugging
• The purpose of testing is to find faults and find them as early as
possible. When we find any such fault, the process used to determine
the cause of this fault and to remove it is known as debugging. These
are related activities and are carried out sequentially.
Limitation of Testing
• We want to test everything before giving the software to the customers. This ‘everything’ is very
illusive and has many meanings. What do we understand when we say ‘everything’? We may
expect one, two or all of the following when we refer to ‘everything’:
i. Execute every statement of the program
ii. Execute every true and false condition
iii. Execute every condition of a decision node
iv. Execute every possible path
v. Execute the program with all valid inputs
vi. Execute the program with all invalid inputs
• These six objectives are impossible to achieve due to time and resource constraints. We may
achieve a few of them. If we do any compromise, we may miss a bug. Input domain is too large to
test and there are too many paths in any program.
• Hence ‘Everything’ is impossible and we have to settle for ‘less than everything’ in real life
situations.
Limitation of Testing
• Errors in the Software Requirement and Specification Document.
• Logical Bugs.
• Difficult to Measure the Progress of Testing.
Errors in the Software Requirement and Specification Document

• These issues are very difficult to identify. If 6+9=20 is written in the SRS document
and our program prints output as 20 when 6 and 9 are inputs, is it a bug? If the
program prints output as 15, when inputs are 6 and 9, how can we interpret? In
this case, the actual output is so obvious that interpretation may not require time
to take a correct decision. But in most of the situations, outputs are not so
obvious. Some requirements may be misunderstood and some may be missed.
Ambiguities of natural languages (like English) may give more than one meaning
to a sentence and make life difficult for testers. Hence, problems in writing good
SRS have also become one of the problems of software testing.
Logical Bugs
• A typical example
Test cases for function
Typical test cases where outputs are different
Logical Bugs
• Logical bugs are extremely difficult to handle and become one of the serious concerns of testing.
• Software testing has inherent difficulties which is making it impossible to completely test the
software. It can only show that bugs are in the software but it cannot show that bugs are not in
the software at all.
• With all the limitations, software testing still is mandatory and a very useful filter to detect errors,
which may further be removed. However we all know that good testing cannot make the software
better, only good coding with software engineering principles makes the software better.
However, good testing techniques may detect a good number of errors and their removal may
improve the quality of the software.
Difficult to Measure the Progress of Testing
• How to measure the progress of testing? Normally we count various things to measure and interpret these
counts. Is experiencing more failures good news or bad news? The answer could be either. A higher number
of failures may indicate that testing was thorough and very few faults remain in the software. Or, it may be
treated as an indication of poor quality of the software with lots of faults; even though many have been
exposed, lots of them still remain. These counts may be illusive and may not help us to measure the progress
of testing.
• This difficulty of measuring the progress of testing leads to another issue i.e. when to stop testing and
release the software to the customer(s)? This is a sensitive decision and should be based on the status of
testing. However, in the absence of testing standards, ‘economics’, ‘time to market’ and ‘gut feeling’ have
become important issues over technical considerations for the release of any software. Many models are
available with serious limitations and are not universally acceptable.
• Software companies are facing serious challenges in testing their products and these challenges are growing
bigger as the software grows more complex. Hence, we should recognize the complex nature of testing and
take it seriously. The gap between standards and practices should be reduced in order to test the software
effectively which may result in to good quality software.
Closing Remarks
• A major problem of the software industry is its inability to develop
error-free software.
• Testing never shows the absence of faults, but it shows that the faults
are present in the program.
• Software testing is a very expensive and critical activity; but releasing
the software without testing is definitely more expensive and
dangerous.
• Most of the times, the testing persons are different from
development persons for the overall benefit of the system.
Closing Remarks
• There are three limitation of software testing: errors in the Software Requirement
and Specification Document, Logical Bugs and Difficult to Measure the Progress
of Testing. With all the limitations, software testing still is mandatory and a very
useful filter to detect errors, which may further be removed. However we all
know that good testing cannot make the software better, only good coding with
software engineering principles makes the software better. However, good testing
techniques may detect a good number of errors and their removal may improve
the quality of the software.