Software testing - what, why and how
Posted February 6th, 2008 by admin
Software testing - that's the buzz word in the IT industry today. I come
across so many youngsters who are excited about career prospects in this field. They are
excited about the number of job opportunities for skilled testers and more importantly, the
high salaries. Many are willing to quit jobs in other fields
Software testing training institutes are flourishing in cities like Bangalore and Pune in India.
They promise to teach Winrunner, Quicktest Pro (QTP) and Loadrunner in a month. These
tools appeal to the youngsters as they are in demand and people with these skills are well
paid.
What they fail to realise is that there is more to testing than the tools.
So, here's an attempt to break the hype and discuss real time situations and problems in
software testing.
What is software testing
It means exactly that - testing the software before making it available for use. Compare that
with what you would do before you buy a car. Do you test drive it? Do you go ahead and buy
it if you think there is a problem?
No. You expect it to work to your satisfaction before you decide to buy. Same goes with
software.
Why should software be tested
Software should be thoroughly tested because in some cases if the software malfunctions, it
can be life threatening. Plane crashes to financial losses in millions of dollars - they have
happenned when software misbehaved.
Check out the Software Hall of Shame to realise the importance of software testing.
How can software be tested
Just like when you go looking to buy a new car, you know exactly what you want which means
you have made a list of requirements that the car you decide to buy should satisfy.
Similarly, the most important thing in software testing is the requirements that the software is
expected to satisfy. A calculator application is expected to provide accurate calculations when
you add, subtract, multiply or divide numbers.
Broadly, once the requirements are clear you can start testing. Depending on the size and
complexity of the project or company, various methodologies, processes and strategies will be
adopted. Fundamentally, software should be tested against well defined requirements.
Types of software testing techniques
Posted February 8th, 2008 by admin
Broadly, software testing techniques can be broken into two types:
1. Functionality testing
2. Non-functionality testing
Functionality testing
This means testing the behaviour of the application to see if it is doing what it is expected to
do. If one person were to use a Windows calculator application, he should expect to see that
addition, subtraction, multiple and division work accurately.
Example: If he enters 2, presses + button, enters 2, presses = button, the result displayed
should be 4.
Which means the programming logic of the application should follow the actual requirement.
This is functionality testing.
Non-functionality testing
Non-functionality testing is not really about programming logic. It's about how the same
application would behave when run under different circumstances.
Example: If the above Windows calendar application works perfectly well for a single user on
a machine with 512KB RAM and 20GB hard disk, assuming that 500 users are going to login to
that machine and use that application at the same time, what would happen to it?
It is possible that the application might crash. If that happens, either the resources of the
machine needs to be improved, like increase RAM or increase CPU, etc or the application code
has to be improved to handle 500 users.
So, unless we run such a test, we will be at risk of the application crashing in production on a
customer site when it comes under heavy load.
It is thus important, during requirements gathering, that not only functional requirements are
detailed but also non-functional requirements.
An example of non-functional requirement can be '500 users must be able to use the Windows
calculator at the same time'. It cannot be something like 'Many users should be able to use
the Windows calculator at the same time'. There should not be any place for ambiguity.
It is the responsibility of the tester to get clarity on these requirements.
o Add new comment
about test plan
On June 26th, 2010 ajaykumarcg says:
Hi, This is ajaykumarcg, i have started my career in testing field before 1.5yrs, but till i didn,t
come across any test plan. Anybody please tell me abt test plan
o reply
A proper test plan is
On August 25th, 2010 admin says:
A proper test plan is typically used only in big service companies. Most small product
companies use a minimal version of it which is just good enough to strategise and plan their
testing activity.
You can take a look here and download a copy to understand more about test plans.
http://www.softwaretestingconcepts.com/download-free-test-plan-template
Software testing - terms, definitions
Posted April 4th, 2008 by admin
o Actual outcome
o The behaviour actually produced when the object is tested under specified
conditions.
o Bug
o (= Fault) A manifestation of an error in software.
o Coverage item
o An entity or property used as a basis for Testing.
o Error
o A human action that produces an incorrect result.
o Expected results
o (= Predicted outcome or Expected outcome) The behaviour predicted by the
specification of an object under specified conditions.
o Failure
o Deviation of the software from its expected delivery or service.
o Fault
o (= Bug) A manifestation of an Error in software. A Fault, if encountered may
cause a Failure.
o Steps
o (= Test Scenario, Input) A sequence of executable actions from an entry point
to an exit point.
o Test Project
o Test Project is top structure in TestLink Data. All data (except users) are
related to active Test Project.
o Test case suite
o A collection of one or more test cases for the software under test.TestLink
uses this term for organization of Test Cases within Test Specification and Test Plan.
o Test Coverage
o The degree, expressed as a percentage, to which a specified Coverage item
has been exercised by a Test case suite.
o Test execution
o The processing of a test case suite by the software under test, producing an
actual outcome. Test Case Suite is defined from Test Cases within Test Specification via
assignment to Test Plan.
o Test Plan
o A record of the test planning process detailing the degree of tester
indedendence, the test environment, the test case design techniques and test
measurement techniques to be used, and the rationale for their choice.
o Test Specification
o For each test case, the coverage item, the initial state of the software under
test, the Steps, and the Expected results.
o Testing
o The process of exercising software to verify that it satisfies specified
requirements and to detect errors.
o Test case
o A set of steps, execution preconditions, and Expected results developed for a
particular objective, such as to exercise a particular program scenario or to verify
compliance with a specific requirement.
Source: Testlink documentation. Reproduced for the benefit of those looking for such
information.
Unit testing
In computer programming, unit testing is a procedure used to validate that individual
modules or units of source code are working properly.
More technically one should consider that a unit is the smallest testable part of an
application. In a Procedural Design a unit may be an individual program, function,
procedure, web page, menu etc. But in Object Oriented Design, the smallest unit is
always a Class; which may be a base/super class, abstract class or derived/child class.
A unit test is a test for a specific unit. Ideally, each test case is independent from the
others; mock objects can be used to assist testing a module in isolation. Unit testing is
typically done by the developers and not by end-users.[3]
What some people fail to grasp is the distinction between unit testing, functional testing,
integration testing, and system testing. Far too frequently what people call ``unit'' tests are
really ``integration'' tests. The distinction is important because frequently if you set out to
write a unit test but find that you can't without first booting up half of the other
components in your app it's a safe bet that you've broken encapsulation and are too tightly
coupled to other pieces of the system. In general you should be able to write any unit test
with basic language objects or the help of a few mock objects
http://en.wikipedia.org/wiki/Mock_Object.
4.1.2 Functional / Acceptance testing
Acceptance testing generally involves running a suite of tests on the completed system.
Each individual test, known as a case, exercises a particular operating condition of the
user's environment or feature of the system, and will result in a pass or fail boolean
outcome. There is generally no degree of success or failure. The test environment is
usually designed to be identical, or as close as possible, to the anticipated user's
environment, including extremes of such. These test cases must each be accompanied by
test case input data or a formal description of the operational activities (or both) to be
performed - intended to thoroughly exercise the specific case - and a formal description
of the expected results.[3]
Now, since we're taking a heuristic approach you can skip anything ``formal'' and be
realistic. You should have some predefined starting and ending conditions, and some,
hopefully automated, way of testing them. But, long before you get to the automated
testing part you should have yourself a sit-down with whoever you're developing this
thing for and walk them through it, or a mock-up of it. You'll find nine times out of ten
that when developing something for a specific group or company it doesn't matter how
well it was described it on paper. They'll still say something to the effect of ``Oh, well
that's nice, but we need it to [insert thing they never mentioned before].''
Once you've gotten something close to what they actually wanted (as opposed to what
they claimed they wanted) you can start putting together some automated tests for it. I
really don't recommend spending too much time writing functional tests before this point
because it's usually easier with tools like Selenium and MaxQ to just record a correct
sequence than it is to tweak an existing one.
Most of the time you'll find that the line between Functional tests and System tests (see
below) tends to blur, and that's ok... most of the time, but you should always try and
decouple the components of your application as much as possible. If you successfully
decouple everything you can use mock objects to represent all the interactions with
remote machines or applications and just test the local functionality. For example.
Imagine your app sends e-mails. Wouldn't it be better if you could test your system
without having to have a working SMTP server, appropriate accounts and permissions,
and net access? Just use a mock SMTP server and you can test all your e-mail sending
routines without having to worry about potential network failures, processing time for
your e-mails as they pass through overloaded spam filters, etc.
4.1.3 Integration testing
Integration testing ... is the phase of software testing in which individual software
modules are combined and tested as a group. It follows unit testing and precedes system
testing.
Integration testing takes as its input modules that have been unit tested, groups them in
larger aggregates, applies tests defined in an integration test plan to those aggregates, and
delivers as its output the integrated system ready for system testing.[3]
A real world example would be a test that makes sure that the models you're loading from
the database and saving to it actually do so correctly. You're testing the integration of
your Object Relational Mapping (ORM) system and your persistent storage system.
Fortunately for most of us those two components were written by the authors of our
favorite web framework and have already been thoroughly tested. Testing them would be
redundant and wasteful. What we do need to test though, is the integration of our
controllers and models as well as our web service APIs.
4.1.4 System testing
System testing is testing conducted on a complete, integrated system to evaluate the
system's compliance with its specified requirements. System testing falls within the scope
of Black box testing, and as such, should require no knowledge of the inner design of the
code or logic.
As a rule, System testing takes, as its input, all of the "integrated" software components
that have successfully passed Integration testing and also the software system itself
integrated with any applicable hardware system(s). The purpose of Integration testing is
to detect any inconsistencies between the software units that are integrated together
(called assemblages) or between any of the assemblages and the hardware. System testing
is a more limiting type of testing; it seeks to detect defects both within the "inter-
assemblages" and also within the system as a whole.[3]
In short, you boot up your app and test that when everything is put together and you start
using it like a real person it doesn't go ``boom.''
5. Triage
If heuristic testing is based on ``a commonsense rule (or set of rules) intended to increase
the probability of solving some problem''. We need to first ask ourselves what the
``problem'' we're trying to solve is. ``Things going 'boom''' is one answer, but lets be
honest with ourselves. We're actually ok with some things going ``boom''. If we weren't
we'd be working for NASA. Every other development house I know of regularly releases
software with bugs in it. As long as nothing too important breaks and nothing breaks in a
way that leaves you looking like an idiot there's a good chance you're willing to live with
it, for a while at lest. So, now that we've admitted the truth to ourselves, we can start
triaging our app.
Triage is a system used by medical or emergency personnel to ration limited medical
resources when the number of injured needing care exceeds the resources available to
perform care so as to treat those patients in most need of treatment who are able to
benefit first.
The word triage is a French word meaning "sorting", which itself is derived from the
Latin tria, meaning "three". The term literally means sorting into three categories.[3]
In medical triaging there are three states ``minor'', ``delayed'', ``immediate'' and
``morgue''. Yeah, I don't know either. Maybe the French didn't want to think about the
dead ones... Anyway, your app's new features, refactorings, and bugs can also be triaged
into the same categories, although you'll probably want to use different names for the
categories depending on where you're doing your triage.
5.2 Is it a Bug or just a Refactoring?
In general when something comes in and you're not sure if you should treat it as a bug or
as a refactoring just ask yourself if you need to prevent it from ever happening again. If
so, it's a bug (see section sec:Bug-Triage). If not, it's just a refactoring. Maybe a really
important refactoring that management wants yesterday, but still...
Text changes are the biggest edge case when it comes to deciding if something is a bug or
a refactoring. You see, if it's so important that it must be fixed then it's a bug and you
need to prevent it from happening again in the future. Otherwise you're just updating
(refactoring) an existing feature. And yes, your textual content is most definitely a feature.
