Practice & Transfer of Learning in

the Teaching of Software Testing

CSEET July 2007
Dublin
Cem Kaner, J.D., Ph.D.
Sowmya Padmanabhan, B.Eng., M.Sc.
Florida Institute of Technology

http://www.kaner.com/pdfs/CSEETdomain2007.pdf

1
Overview

2
Overview: Teaching Domain Testing
In my courses:
• Students generally felt they understood domain testing
after a brief lecture and one or two clear examples
• Diagnostic exercises presented a different picture:
– Students
S d could
ld give back
b k the
h definition
d f / description
d /
rationale
– Students could not reliably apply the technique

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 3

Overview: Analogy to studying mathematics

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 4

Overview: Instructional approach
We tried a teaching style that combined:
• Lecture on the theory
• Procedural suggestions for approaching problems of this
class
• Worked
W k d examples l
• Many exercises, with feedback: Lots of practice

Our long-term goal:

• If this works
works, we can create large pool of test technique
exercises and facilitate a strong improvement in the state
of software testing education.

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 5

Overview: Result
• Test students using questions somewhat similar to the
ones they practiced with:
– Excellent results
• Test students using questions that require a slight stretch
(apply the same principles but in a slightly different way)
– Poor results
• This is shocking to many commercial trainers (my home
community before coming back to university), who struggle
to find time to add a few simple exercises.
• In
I retrospect, similar
i il to traditional
di i l mathematics
h i education
d i
– Students do well on exams
– But cannot apply knowledge in later courses

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 6

Overview: My current puzzle
Should we abandon lecture+problem style in favor of a more
constructivist approach?
1 year Java
1st J (
(co-teaching
hi withi h other
h ffaculty):
l )
• When we try emphasis on larger, meaningful assignments
– Motivates some students, but dismays many others.
– Serious dropout rate.
• Alternate style: Emphasis on narrowly-focused examples (many
smaller exercises)
– Seems easier
– Seems to foster self-confidence & faster connection with basics
– But as with domain testing, indicators of weak transfer.
• I think I’m looking for a balance, rather than abandonment of
procedural instruction. ….

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 7

The Details

8
Software testing …
… is a technical investigation
to provide stakeholders
with quality-related information
about a software product or service

Testers use empirical

p
methods to learn
about quality
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 9
Information objectives
Different
• Find important bugs, to get them fixed objectives
• Assess conformance to specifications drive you
• Assess initial (e.g. first-day-of-use) customer experience towardd
• Help managers make release decisions different:
• Block ppremature product
p releases
•Project
• Help predict and control costs of product support
mgmt styles
• Check interoperability with other products
• Find
Fi d safe
f scenarios
i ffor use off the
th product
d t
•Results
• Certify the product meets a particular standard
reporting
• Ensure the testing process meets accountability standards
methods
• Minimize the risk of safety-related lawsuits •Politics
P liti on
• Help clients improve product quality & testability the project
• Help clients improve their processes •Testing
• Evaluate the product for a third party techniques
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007
Test techniques
A test technique is a heuristic for generating tests
"A heuristic is anything that provides a plausible aid or direction in
the
h solution
l i off a problem
bl bbut iis iin the
h fifinall analysis
l i unjustified,
j ifi d
incapable of justification, and fallible. It is used to guide, to discover
and to reveal. … [A] heuristic has four signatures that make it easy
t recognize:
to i
• A heuristic does not guarantee a solution;
• It may contradict other heuristics;
• It reduces the search time in
solving a problem; and
• Its acceptance depends on the A heuristic is a fallible but
immediate context instead of
on an absolute standard." usefulf l methodh d ffor attempting
– Billy Vaughn Koen,
Definition of the Engineering Method, to solve a problem or reach
( 55, 16
(p. 16-17),
17) 1985
1985.
a decision.
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 11
Domain testing: The field’s most popular technique
Select test cases through stratified sampling:
• Identify the variable of interest
• Identify the set of imaginable values of the variable
(including “invalid” but not inconceivable )
• Partition
P i i the h set into subsets
b that
h are in some way(s)
()
comparable or equivalent
• Select one (or a few) “best best representatives”
representatives of
each set.
– Most commonly used “best representative” is the
b
boundary
d case, which
hi h catches
h inequality
i li errors in
i
specification of numeric ranges

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 12

Teaching domain testing
Teach students:
• the general principle
• how to partition
• how to select a best representative
• how to conduct the test
– enter the value
– look for bad results
¾ immediate failure (problem with filter)
¾ failure on use of the value

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 13

Common student errors
Consider an integer that can take on values from -999
999 to 999 inclusively
• Doesn’t spot a boundary. • Doesn’t explain how a test case relates to a
• Offers excess values. Students offer 998 as well stated risk. When an assignment calls for such an
as the appropriate 999 and 1000. explanation, students may respond inarticulately or
irrelevantly.
• Doesn’t spot a dimension. (a) how many
characters should this field handle? Same for • Doesn’t consider a consequence. In real life
positive and negative numbers? (b) if you delay ( d in
(and i some off our ttestt questions),
ti ) theth tester
t t can
after entering the first character, is there a risk of determine more information than the bare range of
time-out? What delay durations should you test? an input field. The program will do something with
Boundaries? the data entered. It is important,
p , for each of those
uses, to check whether the bounds imposed by the
• Doesn’t articulate a risk. Suppose we explicitly input filter are appropriate to the later use, and
ask students to identify a risk and then identify what consequence will result if they are not.
relevant variable(s) and a powerful test appropriate
to the risk. Rather than describe how the program • Poor generalization. In more complex questions
might fail, the student might reiterate the test or than the integer example here, students often pick
make vague statements, like “fail to process this inappropriate variables for analysis, such as treating
value
l correctly.”
tl ” each value of a binary variable as the best
representative of its own 1-member class.

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 14

Common errors
Students have learned the basic idea
• Bloom’s taxonomy lower levels: know / explain
Students don’t have a higher-level understanding
• apply / analyze / think through what they are learning
How can we increase their depth of understanding?

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 15

Our teaching strategy: Extensive practice
• Teach the basic principle by lecture
• Procedural tips for solving the problem
• Applied to different types of variables, such as:
– Integers
– Dollars
– Floats
– Strings
– Records (non-primitive data types)
• Students
S d workk through
h h exercises,
i f
from simple
i l explicit
li i
cases to “word problems.”
• Expectation: Diversified practice will lead to skill and to
transfer of learning to real application
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 16
Details of the experiment
http://www.testingeducation.org/a/DTD&C.pdf
Sowmya Padmanabhan’s M.Sc. Thesis
Domain Testing: Divide and Conquer
661 pages, including:
• Instructional materials
• Practice exercises / examples
• Final exam
• Assessments by practitioners (James Bach, Pat McGee,
Cem Kaner)

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 17

The course
• 18 classroom hours (5 days)
• 23 paid learners (undergrad and grad) who had not taken
the testing course but had completed the course
prerequisites (discrete math; 2 semesters of programming).
• Taught in 5 replications
replications, 44-55 learners per replication
• 90-minute pre-test (Pre-test A or Pre-test B)
• Classes with many exercises and tests
• 90-minute post-test (Pre-test B or Pre-test A)
• Performance test

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 18

 Sample slides
(Pardon my rush … )

19
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 20
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 21
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 22
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 23
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 24
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 25
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 26
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 27
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 28
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 29
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 30
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 31
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 32
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 33
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 34
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 35
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 36
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 37
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 38
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 39
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 40
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 41
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 42
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 43
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 44
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 45
Results: Formative assessment
• Students performed well on tasks and quizzes during the
instructional sequence.
– Some students required more coaching than others.
• Course evaluations (each day) were very favorable to the
instructor and the materials

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 46

Results: Final exam (Pre-test / Post-test)
All learners improved from pre-test to post-test
• Pre-test mean grade 34.6%
• Post-test mean grade 91.4%

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 47

Final exam: Performance test

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 48

Final exam: Performance test

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 49

Evaluation of performance test
3 graders (Bach, Kaner, McGee)
• each with over 15 years in the field and supervisory
experience
• “Compare the results from this Performance Test to the
results that you would expect from a tester who claimed
to have a year’s experience and who claimed to be good at
domain testing.”

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 50

Performance exam results
All learners:
• approached the task in the same order,
• identified the same variables,
• analyzed them in essentially the same way,
• presented the results in extremely similar tables
• missed the same bugs.

• Result of:
– guiding
idi their
h i workk from
f the
h detailed
d il d procedures
d /
examples
– not from collaboration (cheating) during the exam

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 51

Performance exam example (see paper for more…)
This version of PowerPoint gave an error message for large
pages:
“The current page size exceeds the printable area of
the paper in the printer. Click Fix to automatically fit
the page to the paper. Click Cancel to return to the
Page Setup dialog box. Click OK to continue with the
current page size.”
• Here is new set of equivalence classes
classes—page
page sizes that will
fit the printer versus sizes that will not.
• No learner based a test on this, even though every one
should
h ld have
h seen this
thi message while
hil checking
h ki page sizei lilimits.
it
(In post-experiment interviews months later, some students
remembered seeingg this but could not explain
p whyy theyy didn’t
follow it up. It just didn’t occur to them to follow this up.)

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 52

Performance exam results
In the tested version of PowerPoint, the slide was rescaled.
• Resize slide from 8.5” wide / 11” tall to 56” wide /1” tall,
text on that slide becomes short and wide.
• Some of the distortions look very bad.
•CCouldld trigger intermittent errors (including
( l d a crash) h) by
b
repeatedly resizing pages that had text and graphics.
No learner checked what the slides actually looked like after
dimensions had changed. All tests stopped at the dialog.
The checklists repeatedly mentioned considering output
variables
bl (such
( h as theh actuall displayed
d l d resized d slide),
l d ) but
b never
drilled students on this.
No learner generalized in this way from lecture to
application.
Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 53
What next?
We reviewed the course notes and found various potential
tweaks—ways to
• generalize the instruction
• introduce a wider variety of risks
• practice students
d in checking
h k the h impact off a change
h to a
variable
These would have produced better performance on this
particular performance exam, but what about the next
variation of performance exam?
W didn’t
We did ’ see hhow these
h would
ld lead
l d to a better
b generall
transfer

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 54

What next?
Over the next semesters, Kaner met some of the students
who had taken the course. (They participated anonymously
b t often
but ft voluntarily
l t il self-identified.)
lf id tifi d )
• Their approach was stereotyped and rigid
• They generally looked for procedures for other techniques
• They seemed disadvantaged compared to other students,
when it came to learning new techniques.

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 55

Performance exam results
Conclusion:
• The students learned the procedures well and followed
them closely
• The students did not transfer what they learned from the
procedures and heard in the lectures:
– to consider possibilities / risks beyond those covered in
class, or
– to a slightly more complex example.
• The students did not learn domain testing to the level we
wouldld have
h expectedd from
f modestly-experienced
d l i d
practitioners.

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 56

In retrospect…
To many educational theorists, this is hardly news:
• Lecture is good for transmitting basic information but not
for fostering skilled application, evaluation or extension
• Procedural practice is good for getting across basic
procedures but as with mathematics instruction,
procedures, instruction stellar
performance on examinations doesn’t mean that students
can transfer the knowledge beyond the classroom.

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 57

My response
http://www.testingeducation.org/BBST
Rather than tweaking the procedural approach, Kaner
decided to try a new direction.
• Lectures on video
• Cl
Class-time on meaningful
f l activities
• This is work in progress
• Some results are in Becky Fiedler
Fiedler’ss and my latest NSF
proposal,
http://www.kaner.com/pdfs/CirculatingCCLI2007.pdf

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 58

Implications?
• Back to1st year Java:
– Emphasis on larger, meaningful assignments motivates
some students, but dismays many others. Serious
dropout rate.
– Emphasis on narrowly-focused
narrowly focused examples (many smaller
exercises) seems easier, seems to foster faster
connection with basics and more self-confidence, but
maybe
b weakk transfer.
t f

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 59

Implications?
• Perhaps it is possible to
– structure the course completely around constructivist
experiences / complex enough tasks that they have to
figure things out
– without a high waste (dropout + failure) rate
• But I don’t know how yet.
• What I’m
I m now seeking is
– a balanced approach (procedural foundation plus many
challenging applications)
– that fits within a semester, without creating an
impossible workload.
• Thoughts?

Kaner / Padmanabhan: Practice & Transfer of Learning July 2007 60