Research Design:

Planning the Research

 Planning the Research Project

 Successful research is a planned research

 Fail because plunged into activity with only
a partially thought-out plan & inconclusive
design
 Research requires a conceptualisation of
overall organization and detailed plan
 Should be no less totally visualized and
precisely detailed
Planning the Research Project
 A general strategy = research design (RD)
 RD provides /includes
 Mainly Research method/approach
 Overall structure for the procedures
 Philosophical assumption
 The data the researcher collects
 The data analyses the researcher conducts
 Must plan the overall design carefully to be successful
 Wasted effort if half-prepared (vague ideas)
 Must identify RESOURCES, PROCEDURES, and DATA
always with the goal from the very beginning
 It matters to get your proposal accepted or rejected
 …Cont’d
A research method is a strategy of inquiry which moves from
the underlying philosophical assumptions to specific research
design and data collection & analysis
 Research methodologies reflect high-level approaches to
conducting research.
 The individual steps within the methodology might vary
based on the research problem being performed.
 There are three broad approaches to be used in research design
 Quantitative.
 Qualitative.
 Design science

4
Design Science
 Is a problem-solving paradigm:
 Seeks to create innovations that define the ideas,
practices, technical capabilities, and products
through which the analysis, design, implementation,
and use of IT can be effectively and efficiently
accomplished.
 Emplyes data collection techniques from
qualiatative and quantitative methods
It is recommended to apply
Triangulation
Combining Methods, Data
Some comparisons
Quantitative Qualitative Design science
•Explanation, •Explanation, •Motivated to solve
prediction description problems
•Test/Modify Build theories Prescriptive –
theories Develop artifact
•Known Unknown Known variables
variables variables
•Large sample Small sample Small sample, case
based
•Standardized Observations, Observation,
instruments interviews interviews, etc
•Deductive Inductive Inductive
11
 …Cont’d
Quantitative Qualitative Design science
Statistical •Analytic •Replicable to similar
generalization generalization problems in similar
contexts
•Use numbers Use texts Texts, codes and
drawings
•Standard No standard Pragmatic validity
instrument to instrument,
validation triangulation
•Objective Subjective Subjective Truth ???
Truth truth
•Linear Iterative Iterative
Process
12
Data Collection (sampling and type of data)
 Data Sources (-Sampling )
 People (respondents)
 Databases
 Computer Log files
 Documents (web documents, news…, reports …)
 Devices like camera
 All require a decision whether to use as it is or sample.
 Why sample? How? How many?
 Data collection techniques
 Depends on the type of research methods
 What is Sampling?
• The act, process, or technique of selecting a
suitable sample, or a representative part of a
population for the purpose of determining
parameters or characteristics of the whole
population.
• There would be no need for statistical theory if a
census rather than a sample was always used to
obtain information about populations.
• A census may not be practical and is almost
never economical.
Sampling
 Sampling Techniques
• Probability sampling
– Each unit of the population will be
represented in the sample.
– Each member of the population has a chance
(equal in the case of random selection) of
being selected.
• Non-probability sampling
– The researcher has no way of forecasting that
each member of the population will be
represented in the sample.
– Some members of the population have little or
no chance of being selected.
 Sampling Techniques … cont’d

Probability Sampling
– Simple random sampling
– Systematic random sampling
– Stratified random sampling
– Proportional stratified sampling
– Cluster sampling
 Probability Sampling Techniques … cont’d

• Simple random sampling

– The least sophisticated of all sampling designs
– Simple random selection where every member
of the population is given an equal chance of
being selected;
– Good for homogeneous population;
– Easy when the population is small and
elements are known;
– Impractical for very large populations
 Probability Sampling Techniques… cont’d

• Systematic random sampling

– Selecting elements of the population in
predetermined sequence;
– Select every kth item on a list (k= N/n)
– Randomness element is in picking up the starting
point
 Probability Sampling Techniques… cont’d
• Stratified random sampling
– Applied when the population has different
layers (strata)
– The researcher samples from each one of the
layers (stratum) equally
– Examples
• Sampling of school children from grades 4, 5 and 6
• Sampling of customers
– Corporate customers
– Residential customers
– etc
Probability Sampling Techniques … cont’d

• Proportional stratified sampling

– When number of the elements of the strata
are different
• 2,000 corporate customers
• 400,000 residential customers
Probability Sampling Techniques … cont’d
• Cluster sampling
– Grouping the population into clusters and then
select members of clusters
Random selection
Population of clusters
C1 C2 C3 C4 C5

C5 C6 C7 C8 C8
C9 C10 C11 C12 C11
 Probability Sampling Techniques …cont’d
Population Appropriate sampling
characteristics technique
Homogeneous members •Simple random sampling
•Systematic random
sampling
Stratified population with Stratified random
approximately equal in size sampling
Stratified population, strata Proportional stratified
different in size sampling
Population with discrete Cluster sampling
clusters with similar
characteristics
 Non-Probability Sampling Techniques

• Non-probability sampling
– Convenience sampling
– Quota sampling
– Purposive sampling
Non-Probability Sampling Techniques… cont’d

• Convenience sampling
– Taking of samples that are readily available
• Ex: those that arrive on a scene by coincidence
– May be appropriate for some less demanding
research
• Purposive sampling
– Units of the sample are chosen purposively.
– Choosing people who we have decided are
“typical” of a group;
 Non-Probability Sampling Techniques… cont’d
• Quota sampling
– A variation of convenience sampling
– Elements are selected in the same proportion as in
the population but not in a random fashion
• Ex: there are equal number of Information
Science (IS) and Computer Science (CS) students
–Quota sampling would choose 20 IS and 20 CS
students without any attempt to random
selection
 Summary of Sampling Process
• Steps in the sampling process
– Defining the population
– Specifying the sampling unit
– Specifying the sampling frame
• (the means of representing the elements of
the population. Eg. Tele. Book, map, city
directory, …)
– Specifying the sampling method
– Determining the sample size
– Specifying the sampling plan
– Selecting the sample
What should be Sample Size?

How Many?
Identifying a Sufficient Sample Size
 It depends on many factors like type of research …..
 The basic rule is, the larger the sample, the better
 But such a generalised rule is not too helpful to a researcher who has a
practical decision to make about a specific research situation
 The following are guidelines for selecting a sample size:
 For small populations (with fewer than 100 people or other units), there is
little point in sampling. Survey the entire population
 If the population size is around 500, 50% of the population should be sampled
 If the population size is around 1,500, 20% should be sampled
 Beyond a certain point (at about 5,000 units or more), the population size is
almost irrelevant, and a sample size of 400 should be adequate
 Generally speaking, then, the larger the population, the smaller the percentage
(but not the smaller the number!) one needs to get a representative sample

 In DM like - ~`5000 records

 Types of Data
• Primary Data:
– Are those which are collected afresh and for the
first time and thus happen to be original in
character
• Secondary data:
– Are those which have been collected by someone
else and which have already been passed through
the statistical process
Qualitative Research in IT
 Used to understand user problems for design such in
diagnosing user problems and needs
 Used to collect inputs for artifact development
 Used in artifact evaluation- researchers qualitatively
evaluate a product by interviewing and observation
 Used to uncover non-technical factors affecting the
adoption and evolution of a new software product and
other IT systems
 Used to develop theories such as HCI theory
Application of Qualitative - Example
 System Development Research Process that Nunamaker,
et al (1991) proposed five stages or activities
1. construct a conceptual framework,
2. develop a system architecture,
3. analyze and design the system,
4. build the (prototype) system, and
5. observe and evaluate the system.
 The last stage explicitly includes “Develop new
theories/models based on the observation and
experimentation of the system’s usage”
Quantitative Data Analysis
 Laboratory experiment
 Using machine learning algorithms (for NLP, DM, IR….)
 Different parameter settings and various experiments
 Statistical - two types
1. Descriptive statistics
 Used to describe the phenomena with frequency, mean, median,
mode
2. Inferential statistics
• Is used to infer about the population from the sample data
• T-test to compare means of two groups
• Analysis of Variance (ANOVA)- (When groups more than two)
• Regression , ……
 More readings

About how to conduct interviews and questionnaire

survey
Enumerate basic differences of qualitative
research design and quantitative research
design?
Design Research
Design research basics
 Process model
 Artifact types:
 result of the research work
 Artifact structure
 content of the research approach
 Evaluation:
 evaluation criteria
 evaluation approach
Process model
 a problem-solving paradigm:
 seeks to create innovations that define the ideas, practices,
technical capabilities, and products through which the analysis,
design, implementation, and use of information systems can be
effectively and efficiently accomplished [Tsichritzis 1997;
Denning 1997]
 Design research process

knowledge
flows + operation and goal knowledge

circumscription
process
steps
Awareness of
Suggestion Development Evaluation Conclusion
problem

logical
formalism
abduction deduction

[Takeda,1990]
Artifacts
 are not exempt from natural laws or behavioral theories
 artifacts rely on existing "kernel theories" that are applied,
tested, modified, and extended through the experience,
creativity, intuition, and problem solving capabilities of
the researcher [Walls et al. 1992; Markus et al. 2002]
Design research outputs
[March & Smith, 1995]
 Constructs
 conceptual vocabulary of a problem/solution domain
 Methods
 algorithms and practices to perform a specific task
 Models
 a set of propositions or statements expressing relationships among
constructs
 abstractions and representations
 Instantiations
ut
t
 constitute the realization of constructs, models and methods in a p
working system
s ou
 implemented and prototype systems
s i
 Better theories
he
 artifact construction T
 Design research outputs

constructs
better theories
emergent theory about models
embedded phenomena
abstraction models
abstraction methods
knowledge as constructs
operational principles
better theories

abstraction

artifact as situated implementation instatiations

methods
constructs
[Purao , 2002]
Gregor and Hevner (2013)
Examples
 Provide a unifying framework
 Resolve a long-standing question
 Thoroughly explore an area
 Contradict existing knowledge
 Experimentally validate a theory
 Produce an ambitious system
 Provide empirical data
 Derive superior algorithms ( in IR, Machine
learning..
 Develop new methodology (for software develop..)
 Develop a new tool (DM/BI, KBS, Stemmer…
Artifact structure
 Structure of the artifact
 the information space the artifact spans
 basis for deducing all required information about the artifact
 determines the configurational characteristics necessary to
enable the evaluation of the artifact
i s
h es
e t
t h
of
nt
t e
on
C
Evaluation criteria
 Evaluation criteria
 the dimensions of the information space which are relevant for
determining the utility of the artifact
 can differ on the purpose of the evaluation

e s
a s
t c
e s
T
Evaluation approach
 Evaluation approach
 the procedure how to practically test an artifact
 defines all roles concerned with the assessment and the way of
handling the evaluation
 result is a decision whether or not the artifact meets the
evaluation criteria based on the available information.
od
e th
m
g
tin
s
Te
Cont...
 Quantative evaluation:
 originally developed in the natural sciences to study natural
phenomena
 approaches:
 survey methods
 laboratory experiments
 formal methods (e.g. econometrics)
 numerical methods (e.g. mathematical modeling)
Cont...
 Qualitative evaluation:
 developed in the social sciences to enable researchers to
study social and cultural phenomena
 approaches:
 action research
 case study research
 ethnography
 grounded theory
 qualitative data sources:
 observation and participant observation (fieldwork)
 interviews
 documents and texts
 the researcher’s impressions and reactions
Constructs

Structure Evaluation criteria Evaluation approach

meta-model of construct deficit ontological analysis

the vocabulary construct overload

construct redundancy

construct excess
 Methods

Structure Evaluation criteria Evaluation approach

process-based meta appropriateness laboratory research
model completeness field inquiries
intended applications consistency surveys
conditions of case studies
applicability action research
products and results of
practice descriptions
the method application
interpretative research
reference to constructs
Models

Structure Evaluation criteria Evaluation approach

 domain correcteness syntactical validation
 scope, purpose completeness integrity checking

 syntax and semantics clarity sampling using selective

 terminology flexibility matching of data to actual

 intended application simplicity external phenomena or
trusted surrogate
applicability
integration tests
implementability
risk and cost analysis

user surveys
Instantiations
Structure Evaluation criteria Evaluation approach
executable implementation functionality code inspection
in a programming language usability testing
reference to a design model
reliability code analysis
reference to a requirement
performance verification
specification
reference to the supportability
documentation
reference to quality
management documents
reference to configuration
management documents
reference to project

management documents
To Summarize- Research Design
 Must answer four fundamental questions with respect to
data – no reservation
 What are the data needed? – What data are mandatory? What
are their nature? Documentary? Statistical? Interview?
Questionnaire? …
 Where are the data located?
 How will the data be secured?
 How will the data be interpreted?
Conclusion

Good research results require a careful

design of the research methodology and
considerable evaluation efforts