Research Design: Planning the

research
Major topics
 The three research designs (approaches) and comparison
 Sampling and related issues
 Details of Qualitative research approaches
 Details of Quantitative research approach
 Details of Design science research approach
Planning the Research Project
 A general strategy = research design (RD)
 RD provides /includes
 Philosophical assumption, Mainly Research approach/ and specific
method
 Overall structure for the procedures
 The data the researcher collects/and techniques
 The data analyses/experimentation/design the researcher conducts
 Evaluation…
 Must plan the overall design carefully to be successful
 Wasted effort if half-prepared (vague ideas)
 Must identify RESOURCES, PROCEDURES, and DATA always with the goal
from the very beginning
 It matters to get your proposal accepted or rejected
Cont…
 A research approach/method is a strategy of inquiry which
moves from the underlying philosophical assumptions to
specific research design and data collection & analysis
 Research methodologies reflect high-level approaches to
conducting research.
 The individual steps within the methodology might vary
based on the research problem being performed.
 There are three broad approaches to be used in research
design
 Quantitative.
 Qualitative.
 Design science

4
Philosophical Assumption: Positivist Vs
Interpretive
 Positivists generally assume that reality is objectively given and
can be described by measurable properties which are
independent of the observer (researcher) and his or her
instruments.

 Interpretive researchers start out with the assumption that

access to reality (given or socially constructed) is only through
social constructions such as language, consciousness and
shared meanings.
Research Approach: Qualitative
 Qualitative research involves the use of qualitative data, such
as interviews, documents, and participant observation data, to
understand and explain social phenomena.

 Examples of qualitative methods are action research, case

study research and ethnography.

 Qualitative data sources include observation and participant

observation (fieldwork), interviews and documents and texts,
and the researcher's impressions and reactions (Myers 2009).
Research Approach: Quantitative
 Quantitative research methods were originally developed
in the natural sciences to study natural phenomena.

 Examples of quantitative methods now well accepted in the

social sciences include survey methods, laboratory
experiments, formal methods (e.g. econometrics) and
numerical methods such as mathematical modeling

 The most popular data collection techniques include:

Questionnaire (survey instruments) , secondary data sources
or archival data, objective measures or tests.
Research Approach: Design
Science(engineering research)
 Is a problem-solving paradigm:

 Seeks to create innovations that define the ideas, practices,

technical capabilities, and products through which the
analysis, design, implementation, and use of IT can be
effectively and efficiently accomplished.

 Employs data collection techniques from qualiatative and

quantitative methods

 The specfice method is also termed as - Iterative Design and

Evaluate
It is recommended to apply
Triangulation
Combining Methods, Data
Some comparisons of the three research
approaches
Quantitative Qualitative Design science
•Explanation, •Explanation, •Motivated to solve
prediction description problems
•Test/Modify Build theories Prescriptive – Develop
theories artifact
•Known variables Unknown variables Known variables
•Large sample Small sample Small sample, case
based
•Standardized Observations, Observation,
instruments interviews interviews, etc
•Deductive Inductive Inductive

10
 Cont’d
Quantitative Qualitative Design science
Statistical •Analytic •Replicable to similar
generalization generalization problems in similar
contexts
•Use numbers Use texts Texts, codes and
drawings
•Standard No standard Pragmatic validity
instrument to instrument,
validation triangulation
•Objective Truth Subjective truth Subjective Truth ???
•Linear Process Iterative Iterative
12
Sampling – Core decision in research design
 Do we need sample first of all?
 Data Sources (-Sampling )
 People (respondents)
 Databases, Computer Log files
 Documents (web documents, news…, reports …)
 Devices like camera
 All require a decision whether to use as is or sample.
 Why sample? How? How many?
 Sampling and Data collection techniques should agree and
 Depends on the type of research methods and type of data
you want to collect
 What is Sampling?
• The act, process, or technique of selecting a
suitable sample, or a representative part of a
population for the purpose of determining
parameters or characteristics of the whole
population.
• There would be no need for statistical theory if a
census rather than a sample was always used to
obtain information about populations.
• A census may not be practical and is almost
never economical.
Sampling
 Sampling Techniques
• Probability sampling
– Each unit of the population will be
represented in the sample.
– Each member of the population has a chance
(equal in the case of random selection) of
being selected.
• Non-probability sampling
– The researcher has no way of forecasting that
each member of the population will be
represented in the sample.
– Some members of the population have little or
no chance of being selected.
 Sampling Techniques … cont’d

Probability Sampling
– Simple random sampling
– Systematic random sampling
– Stratified random sampling
– Proportional stratified sampling
– Cluster sampling
 Probability Sampling Techniques … cont’d

• Simple random sampling

– The least sophisticated of all sampling designs
– Simple random selection where every member
of the population is given an equal chance of
being selected;
– Good for homogeneous population;
– Easy when the population is small and
elements are known;
– Impractical for very large populations
 Probability Sampling Techniques… cont’d

• Systematic random sampling

– Selecting elements of the population in
predetermined sequence;
– Select every kth item on a list (k= N/n)
– Randomness element is in picking up the starting
point
 Probability Sampling Techniques… cont’d
• Stratified random sampling
– Applied when the population has different
layers (strata)
– The researcher samples from each one of the
layers (stratum) equally
– Examples
• Sampling of school children from grades 4, 5 and 6
• Sampling of customers
– Corporate customers
– Residential customers
– etc
Probability Sampling Techniques … cont’d

• Proportional stratified sampling

– When number of the elements of the strata
are different
• 2,000 corporate customers
• 400,000 residential customers
Probability Sampling Techniques … cont’d
• Cluster sampling
– Grouping the population into clusters and then
select members of clusters
Random selection
Population of clusters
C1 C2 C3 C4 C5

C5 C6 C7 C8 C8
C9 C10 C11 C12 C11
 Probability Sampling Techniques …cont’d
Population Appropriate sampling
characteristics technique
Homogeneous members •Simple random sampling
•Systematic random
sampling
Stratified population with Stratified random
approximately equal in size sampling
Stratified population, strata Proportional stratified
different in size sampling
Population with discrete Cluster sampling
clusters with similar
characteristics
 Non-Probability Sampling Techniques

• Non-probability sampling
– Convenience sampling
– Quota sampling
– Purposive sampling
Non-Probability Sampling Techniques… cont’d

• Convenience sampling
– Taking of samples that are readily available
• Ex: those that arrive on a scene by coincidence
– May be appropriate for some less demanding
research
• Purposive sampling
– Units of the sample are chosen purposively.
– Choosing people who we have decided are
“typical” of a group;
 Non-Probability Sampling Techniques… cont’d
• Quota sampling
– A variation of convenience sampling
– Elements are selected in the same proportion as in
the population but not in a random fashion
• Ex: there are equal number of Information
Science (IS) and Computer Science (CS) students
–Quota sampling would choose 20 IS and 20 CS
students without any attempt to random
selection
What should be Sample Size?

How Many?

Key features: Sufficient and

Representative
Identifying a Sufficient Sample Size
 It depends on many factors like type of research …..
 To some extent, the size of an adequate sample depends on how homogeneous or
heterogeneous the population is – how alike or different its members are with
respect to the characteristics of research interest
 Important, too, is the degree of precision with which the researcher wishes to
draw conclusions or make predictions about the population under study
 The basic rule is, the larger the sample, the better

 But such a generalized rule is not too helpful to a researcher

who has a practical decision to make about a specific research
situation

 Statisticians have developed formulas for determining the

desired sample size for a given population

 There are also some guidelines like shown in the next slide
Cont…
 The following are guidelines for selecting a sample size for quantitative survey:
 For small populations (with fewer than 100 people or other units), there is
little point in sampling. Survey the entire population
 If the population size is around 500, 50% of the population should be
sampled
 If the population size is around 1,500, 20% should be sampled
 Beyond a certain point (at about 5,000 units or more), the population size
is almost irrelevant, and a sample size of 400 should be adequate
 Generally speaking, then, the larger the population, the smaller the
percentage (but not the smaller the number!) one needs to get a
representative sample
 For quantitative laboratory experimental -even more
 ~`5000 records
Cont…

 Does this work for qualitative type researches too?

 No
 Most researches in qualitative approaches uses 1 – 30 sample size.
 Actually the concept of saturation determines sample size in
qualitative researches.
 points to recall
• Research Design: Planning the research
• Qualitative
• Case study and action research using interview,
observation, document analysis
• Quantitative
• Laboratory experiment and survey using data from
database or collected through questionnaire
• Design Science
• create innovations that define the ideas, practices,
technical capabilities, and products using qualiatative or
quantitative data
• Triangultion
• Data sources and sampling
Qualitative Research Design
 Qualitative Research Methods
• Its aim is to give a complete, detailed descriptions of
the phenomena to be studied
• Objective facts + values
• Key philosophical assumption
• understanding how people make sense of their
worlds and the experiences people have
• Key concern –
• knowing or understanding from the participants’
perspectives
• Key focus –
• understanding (rather than predicting or controlling)
social settings or social phenomena
When to choose qualitative approach?
 Describe the phenomena
 Identifying inputs/ requirements
 Build a theory
 To gain new insights about a particular phenomena
 Develop new concepts or theoretical perspectives about
the phenomena
 Discover the problem that exists in the phenomena
 Verification – to check the validity of certain assumptions (not
hypothesis), claims, generalization with the real world
 Evaluation – to evaluate the effectiveness of a particular
policies, design artifacts, programs, etc
 Define problem

Review literature Conceptual

framework/
Proposal
Collect Data

Data Analysis
Build Theory
Or Framework
Data Interpretation/
Report Findings
Common Types of Qualitative Research
Designs
 There are many qualitative research methods.
 Case study
 Action research
 Ethnography
 Etc.
Case study research method/design
 Its main purpose is to gain a deep understanding of a phenomenon
 Example:
 To understand the capability of a new tool
 To identify factors affecting communication in code inspections
 To characterize the process of coming up to speed on a project
 In building a framework, model or any design
 Objective of Investigation
 Exploration-To find what’s out there
 Characterization-To more fully describe
 Validation-To find out whether a solution/ proposition is true
 Subject of Investigation in a case study could be
 An intervention, e.g. tool, technique, method, approach to design,
implementation, or organizational structure
 An existing thing or process, e.g. software implementation, project
success, defects, network performance …
Important aspects of a Case Study
Research design/process
 Five main parts of a case study research design (under the
umbrella of the general research process)
 Research questions
 Propositions (if any)
 Unit(s) of analysis
 Logic linking the data to the propositions (analysis)
 Criteria for interpreting the findings
Study Questions(RQ)
 Case studies are most appropriate for research questions that
are of the “how” and “why” variety
 The initial task is to clarify precisely the nature of the study
questions (i.e. make sure they are actually “how” or “why”
questions)
 Examples:
 "Why do developers prefer this tool/model/notation?"
 "How are software inspections carried out in practice?“
 "How does agile development work in practice?"
 "Why do programmers fail to document their code?“
 "How does software evolve over time?“
 "Why have formal methods not been adopted widely for
safety critical applications?“
 "How does a company identify which software development
projects to start?"
Study Propositions (hypothesis)
 Propositions are statements that help direct attention to
something that should be examined in the case study, i.e. point
to what should be studied
 Example: “Organizations collaborate because they derive
mutual benefits”
 Propositions will tell you where to look for relevant evidence
 Example: Define and ascertain the specific benefits to each
organization
 Some studies may not have propositions –this implies a topic
of “exploration”
 Note: Even exploratory studies should have both clearly-
stated purposes and clearly-stated criteria for success
Unit of Analysis
 The unit of analysis defines what a “case” is in a case study
 Example: a unit of analysis (case) may be an individual, and
the case study may be the life history of that person
 Other units of analysis include decisions, social programs,
processes, changes, methods…
 Note: It is important to clarify the definition of these cases
as they may be subjective, e.g. the beginning and end points
of a process
 What unit of analysis to use generally depends on the primary
research questions
 Once defined, the unit of analysis can still be changed if
desired, e.g. as a result of discoveries based on data
 To compare results with previous studies (or allow others to
compare results with yours), try to select a unit of analysis
that is or can be used by others
 Examples of Units of Analysis
 For a study of software adoption
 Individuals
 Development team
 Organization
 For a study of programming
 Programming episode
 Pairs of programmers
 Development team
 Organization
 For a study of software evolution
 Modification report
 File
 System
 Release
 Stable release
Linking Logic (analysis)
 Logic or reasoning to link data to propositions/and/or findings
 One of the least well developed components in case studies
 Many ways to perform this, but none as precisely defined as
the treatment/subject approach used in experiments
 Can be simply explained as interrelating themes.
 One possibility is pattern matching: describe several potential
patterns, then compare the case study data to the patterns
and see which one is closer
 Cont..
 Validity and Reliability in Case Study
 Refers to the quality of the research process mainly in data collection
and analysis
 Using the same criteria for other empirical research
 Construct Validity
 Concepts being studied are operationalized and measured correctly
 Internal Validity
 Does the data support the conclusion?
 External Validity
 Establish the domain to which a study’s findings can be generalized
 Can the result be generalized?
 Experimental Reliability
 Demonstrate that the study can be repeated with the same results
 Is the method/conduct of the research systematic and logical?
Cont…
 Criteria for High Quality Analysis
 Present all the evidence
 Develop rival hypotheses/concepts
 Address all major rival interpretations
 Address most significant aspect of the case study
 Use prior or expert knowledge
 Criteria for interpreting the findings
 The last issue in case study design is
 Setting criteria for interpreting findings
 Obviously from the context /case at hand
 And with the existing literature.

 It is nothing but planning how the findings should be

interpreted
Qualitative research in IT/CS
 Used to understand user problems for design such as in
diagnosing user problems and needs
 Used to collect inputs for artifact development
 Used in artifact evaluation- researchers qualitatively evaluate a
product by interviewing and observation
 Used to uncover non-technical factors affecting the adoption
and evolution of a new software product and other IT systems
 Used to develop theories such as HCI theory
Application of Qualitative - Example
 System Development Research Process that Nunamaker, et al
(1991) proposed five stages or activities
1. construct a conceptual framework,
2. develop a system architecture,
3. analyze and design the system,
4. build the (prototype) system, and
5. observe and evaluate the system.
 The last stage (along with the first) explicitly includes “Develop
new theories/models based on the observation and
experimentation of the system’s usage”
Points to recall
 Research Design
 Can be quantitative, qualitative, design or a combination of
these
 Should be logical for the validity and reliability of the research.
 There are different methods under each research approach
 The choice will mainly depend on the research agenda/topic
 Determining data sources and type of data to be collected is
common to all
 Deciding if sampling is required and selecting appropriate
sampling method with right number of sample is again
important in all cases.
 Case study is a good example method under Qualitative
approach