Research Methodology

INF 702/780

Research Strategies
Chapters 7 - 20
Oates et al (2024)
Outline

• Research recap
• Data
• Data generation methods
• Research Strategies
6Ps of Research
Purpose: What is the reason for doing the research? Why is it important or
useful to study this? Specific research question(s) asked, and the
objectives set.
Products: What is the outcome / your contribution to knowledge? Can be
your answer to the research questions / unexpected findings.
Process: What is the sequence of activities undertaken in the research
project (the research plan)? {Identify research topic; establish conceptual
framework; select & use research strategy & data generation methods;
analyse data; draw conclusions; recognise limitations.}
Participants: Who will be involved in the research? {Interviewing / observing
– deal with them in an ethical way}
Paradigm/Philosophy: A pattern/model/shared way of thinking. Any piece of
research has an underlying philosophical paradigm = set of commonly
held beliefs and assumptions within a research community about
ontological, epistemological and methodological concerns. It constitutes a
mental model that influences and structures how members of a research
community perceive their field of study (a shared way of thinking) (E.g.,
Positivism, realism, interpretivism, pragmatism)
Presentation: How will the research and findings be disseminated and
explained to others? (In a journal paper, conference paper, book chapter,
thesis, etc.)
Data generation methods
The research process
Model of the research process (Oates, 2022)
Data Generation Methods
• The means by which you produce empirical (field) data or
evidence
• Either quantitative OR qualitative:
• Quantitative = numeric: nr of website hits; number of employees;
average age of students; annual turnover; etc.
• Qualitative = all other types of data: words; images; sounds; etc.
 Data Generation Methods
• More than one method:
• = more data
• improves quality of research
• might take longer
• might cost more

• Can compare findings of different methods

• If data shows some consistency (across methods) = your
findings are supporting with evidence = increases confidence in
your findings
 Data Generation Methods
The use of more than one data generation method to corroborate findings + enhance
their validity = METHOD TRIANGULATION

Types of triangulation:
• Method triangulation: use 2 or more data generation methods
• Strategy triangulation: use 2 or more research strategies
• Time triangulation: study takes place at 2 or more different points in time
• Space triangulation: study takes place in two or more different countries or cultures – to
overcome parochialism = a limited or narrow outlook (study based on one country or one
culture)
• Investigator triangulation: study carried out by 2 or more researchers who then compare their
findings
• Theoretical triangulation: study draws on 2 or more theories (not only 1)
Data generation methods:
Interviews
Interviews
A conversation between people
Researcher has a purpose for undertaking the interview (to gain information
from the other(s))
• Discussion is planned in some way
• Usually have an agenda (issues they want to find out about)
• Researcher steers/guides the discussion onto their topic of interest
• Tacit agreement that the researcher has the right to control the agenda and
proceeding
• Interviewee words can be treated as ‘on the record’ and used by the
researcher
Interviews
Instances for which an interview is a suitable data generation method:
• Obtain detailed information
• Ask complex or open-ended questions
• Explore emotions, experiences or feelings not easily observed/described via
pre-defined questionnaire responses
• Investigate sensitive issues or privileged information

Typical research strategies used with:

• Case studies
• Ethnographies
• Surveys
• Design and create strategy
Interviews
Also used with surveys: interview people rahter than having them
completing a questionnaire
• To ‘top-and-tail’ a survey strategy (elicit themes to be included in
questionnaire; follow-up interviews to obtain more detail about questionnaire
responses)
With design and create strategy:
• To generate requirements specification and elicit user feedback once
design is finished
One-on-one interviews OR group interviews with group of respondents (focus
groups)
Data collection methods:
Observations
Observations
Method used to find out what people ACTUALLY DO, rather than what they
report they do when you question them
Observing is ‘to watch’ and ‘to pay attention to’:
• Seeing
• Hearing
• Smelling
• Touching
• Tasting
• Noting
• Analysing
• Forming theories
• Making inferences
• Imposing meaning
Observations
Observation example: Researcher might
• look at the seeting pattern in a group meeting;
• feel the tecture or comfort of the different chairs
• listen to people’s contributions (words and toen)
• watch body language
• smell when people are becoming hot and bothered
• taste the orange juice handed rounf to help them cool down

Used with any research strategy

Observations
Used in ‘overt’ or ‘covert’ research:
• Covert research: people being observed do not know it (the setting is not
disturbed, and people act naturally) – is this ethical?
• Overt research: people know they are being observed (they can give
consent - research is more ethical)
• Problems:
• the ‘Hawthorn Effect’: people modify their behaviour because they
know they are observed
• How should they treat the researcher? = stressful
• Get used to being observed = uncomfortable; defensive
Data collection methods:
Questionnaires
Questionnaires
A pre-defined set of questions assembled in a pre-determined order
Respondents asked to answer questions
Provide researcher with data to analyse and interpret
Researcher will look at patterns and make generalisations about the
actions or views of a larger population than the sample

Strategies used with:

• Typically used with survey research strategy
• Also used with case study, action research or design and creation
Questionnaires
Can be self-administered (completed by responded) OR researcher-
administered (researcher asks question and writes down response =
kind of structured interview)
Situations suited for:
• Obtain data from many people
• Obtain relatively brief and uncontroversial information from people
• Obtain standardised data (identical questions to each respondent
by pre-defining range of answers)
• Enough time to wait for responses to come in
Questionnaires
IMPORTANT: A questionnaire is not just a list of questions with
multiple-choice answers!
• Must be carefully designed and constructed sot that valid,
reliable data can be generated
• Write the questions so that all respondents understand them in
the same way and decode the possible answers you provide in the
way you intended to
• Questions and answers must meet your research needs (be in line
with your research questions)
• Know in advance how you intend to analyse the responses
Data collection methods:
Documents
Documents
Another source of data
Two types:
• Found documents:
• already exist prior to the research
• E.g., production schedules; profit and loss accounts; internal telephone
directories; job descriptions; procedure manuals; etc.
• Researcher-generated documents:
• Put together solely for the purpose of the research task
Documents
• Examples of researcher generated documents:
• photographs taken, and field notes made by researcher undertaking an
ethnography
• models and diagrams (story boards, use cases, DFDs) designed and created for a
new artefact using the design and creation strategy
• Researcher can design document and ask someone to complete it e.g.,
someone working at helpdesk might be asked to log the different kinds of requests
handled in a month
Research Strategies
 Data Generation Methods
• Linking data generation methods with research
strategies:
• Experiment – uses observations
• Survey – uses questionnaires
• Strategy can use MORE than one method:
• Allows us to look at phenomena in different ways
• Examples:
• using action research as strategy: interview people;
observe what happens; ask people to complete a
questionnaire; and collect documents produced
• doing case study of how people interact with web-
based system: observe people using it; collect data
about their movement around the site; interview
them about the perceptions of it; ask them to
complete a questionnaire on it.
Research strategy: surveys
Objective – to understand and predict
• To obtain the same kinds of data from a large group of people (or
events) in a standardized and systematic way
• Look for patterns to generalise to larger population
• Often associated with a positivist research paradigm
• Often starts with a theoretical framework/model

Surveys can also be used to collect qualitative data

Of which the end-result is often themes/categories or even theories
Generalization less important
Survey – Data requirements

Depends on the research questions!

What data do you wish to generate?
Directly topic related or indirectly topic related?
Data requirements are often guided by a
theoretical model.
The role of theory
What is theory? An idea to explain something, or a set of guiding
principles.
A theory is a coherent collection of concepts and propositions with an
underlying world-view https://is.theorizeit.org/wiki/Main_Page.

A theory will also determine what you will see – a lens

E.g. Theory of technology acceptance (TAM)

Deductive reasoning in research

• Sometimes when using a survey research strategy, the data collection is

guided by a theoretical framework. The framework guides the formulation
of hypothesis.
• The hypotheses are tested by collecting data from a carefully selected
sample.
• The data is analyzed using statistical measures and tests.
• The hypothesis are confirmed or rejected.

Often associated with positivism

Inductive reasoning in research
I am interested in how professional programmers learn new
languages. There is some research available on that, but I want to
learn from programmers themselves.
I use survey research strategy and use snowball sampling (for
example) to get the names of several expert programmers. I then
conduct interviews and observations.
This data is recorded and analysed using thematic and/or content
analysis (data analysis is the technique of taking data, finding
patterns, and abstract those to higher level concepts) – which
eventually can lead to a theory.
Often associated with interpretivist research
Survey – data generation methods

Questionnaires
Observations
Documents
Interviews
Survey – sampling frame

A sample frame is a list or collection of the whole population of

people (or events or document) that could be included in your
survey, from which you will choose a sample.
Example, if your population is the undergraduate students at UP
the sample frame is the list of registered students in the
Peoplesoft system.
Survey – sampling technique

Probability vs nonprobability
Probability – the sample is a representative cross-section of the
overall population
Non-probability – the researcher cannot be sure that the sample is
representative of the population
 Probability sampling techniques
Random – one needs the sampling frame to pick from in a random way –
draw from a hat, use software to generate random numbers etc.
Systematic sampling – choose every 100th name or event from the
sampling frame
Stratified sampling – if population is stratified – e.g. vegetarians (20%)
and non-vegetarians (80%) you need to stratify the sample as well and
then choose in a random fashion from two groups
Cluster sampling – e.g. certain types of events or people might naturally
occur in clusters. E.g. young people naturally occur in schools – restrict
your sample to schools.
Non-probability sampling techniques

Purposive – choose instances that are likely to produce valuable

data to meet the purpose of the research
Snowball – fine one person from the target population, who
suggests other names etc
Self-selection – researcher advertise their interest in a particular
topic and need for responses and collect data from anyone who
responds.
Convenience – researchers select respondents who are
convenient for them.
Response rate and non-responses

10% response rate quite typical

You need to get as many responses as possible.
Follow-up, remind again etc.
Try to find the characteristics of those not responding to
understand whether their non-response is meaningful in its own
right or if the lack of response creates bias in your research
Sample size

If you want to generalize your findings from the sample to the

population, the sample must be of adequate size.
Accuracy level - margin of error. If it is reported that 70% of the
population think the president is doing a good job, with an
accuracy range of +- 3 percent, then the true value of people who
think that is somewhere in the range 67% to 73% of the
population. You can only get 100% accuracy if the whole
population is sampled.
Sample size

Confidence level means we are for example 95% sure that the true
population value falls within the range of values obtained from the
sample. A 0% confidence level means you have no faith at all that if you
repeated the survey that you would get the same results. A 100%
confidence level means there is no doubt at all that if you repeated the
survey you would get the same results. This is influenced by population
size and sample size.
One can calculate the accuracy level and sample size -
https://www.surveysystem.com/sscalc.htm
Sampling – qualitative survey research

No need for random sampling as the purpose is not generalization to the

population
Typical sample size – minimum of 12 (Clarke and Braun, 2013)
Survey Summary

The research paradigm, research questions and research strategy are

interrelated.
Survey research is used to obtain the same kinds of data from a large
group of people (or events) in a standardized and systematic way.
Careful sampling is important in quantitative survey research
Same research problems can be approached from different paradigms,
different strategies with differently formulated research questions.
Research strategies:
Case Studies
 Learning objectives
• What is meant by a case study research
strategy;
• Issues to address in planning and undertaking
case study research;
• How case study research might be applied in
the digital world;
• How case studies have been used in IS and
computing;
• The advantages and disadvantages of case
study research;
• Analysing and evaluating case study
research.
 Research strategy: Case Studies
• A case study:
• Focusses on ONE instance of a ‘thing’ which is
investigated (e.g., an organisation; a department, an
information system; a networking platform; a systems
developer; a development project; etc.)
• This instance is studied IN DEPTH
• The researcher uses a variety of data generation
methods (interviewing, observation, document analysis;
questionnaires; etc.) for the in-depth study.

• Objective of a case study:

• To obtain a rich, detailed insight into the ‘life’ of the
‘thing’ and its complex relationships and processes.
Research strategy: Case Studies

• Difference between a survey strategy and

a case study strategy?

• Survey strategy provides a wide but only

shallow view of lots of instances of the
phenomenon you investigate (not likely to
gain a lot of information about the context of
the phenomenon for each instance)
• Case study looks at the chosen case within
its real-life context and focusses on ALL the
factors, issues, politics, processes and
relationships that constitutes the
phenomenon.
 Research strategy: Case Studies

• The essence of a case study:

• It paints a detailed picture of a
phenomenon
• It provides insight and generates
knowledge that might also be relevant to
other situations.
• Can have an underlying philosophical
paradigm of either positivism, interpretivism
or critical research.
 Research strategy: Case Studies

Objective – to understand, describe and explain!

• To study, in depth, the “thing” that is to be investigated: an organisation, a department, an
information system, a discussion forum, a system developer, etc.
• by using different data generation methods to get a rich detailed understanding of the “life” of the
case and its complex relationships and processes.

Important questions to ask

• What type of case study?
• How to choose the case study?
• Can one generalize?
• How to report the findings?
Definition of a case study

• A case study is an empirical inquiry

that investigates a contemporary
phenomenon within its real-life
context, especially when the
boundaries between phenomenon and
context are not clearly evident. (Yin,
2017, p. 18)
Yin, R. K. (2017). Case study research and
applications: Design and methods (6th ed.).
London: Sage.
Characteristics of a Case Study

1. Focus on depth rather than breadth:

• as much detail as possible about one instance of the phenomenon under
investigation
2. Natural setting:
• the instance, or case, is examined in its natural setting.
• It existed prior to the researcher arriving on the scene
• It normally, continues to exist after the researcher has moved on.
• The researcher seeks to disturb the setting as little as possible.
Research strategy: Case Studies

3. Holistic study:
• focuses on the complexity of relationships and processes and how they are
interconnected and interrelated
• Not trying to isolate individual factors.
4. Multiple sources and methods:
• Wide range of data sources used (talk to as many people as possible to obtain multiple
perceptions).
• Both quantitative and qualitative data can be used.
• Data obtained through a large range of data generation methods: interviewing,
observation, questionnaires, and document analysis, internal briefing notes, the
researcher's own field notes and personal journal.
Types of Case Studies

Exploratory case study:

A case study that explores situations to develop an initial understanding of
a phenomenon.
Used where there is little in the literature about a topic - a real-life instance
is investigated in order to identify the topics to be covered in a research
project.
Usually conducted when a researcher has just begun an investigation and
wishes to understand the topic in general.
Types of Case Studies

Descriptive case study:

Leads to a rich, detailed analysis of a particular phenomenon and its
context (describes).
Analysis tells a story: discussion on what occurred and how different
people perceive what occurred.
Types of Case Studies

Explanatory study:
Tries to explain why events happened as they did or why particular
outcomes occurred.
Seeks to identify the multiple, often interlinked, factors that had an effect
OR
Compares what was found in the case to theories from the literature in
order to see whether one theory matches the case better than others.
The Case Study Strategy

Can use a single case or multiple cases.

The most common approach is to examine one case only.
If a multiple case study approach is adopted:
each case written up separately
the researcher looks for similarities or differences between the cases.
Outcomes from several case studies are more compelling than
conclusions drawn from just one case study.
Differences can also be useful.
The Case Study Strategy

Could use first case study to suggest a theoretical framework to explain

what has been found.
This framework can then suggest the circumstances when another case
will not be like the first.
Examine data from a second, differing case to see whether the theoretical
framework's predictions are correct.
Multiple case study approaches = more time-consuming = much more data
to analyse.
Case Study Approach

Case studies vary in their approach to time:

A historical study:
• examines what happened in the past
• ask people what they remember about earlier events
• analyse documents produced at the time (e.g., meeting minutes -
these depend on people's memories; are rarely an objective
description of what occurred)
Case Study Approach

Case studies vary in their approach to

time (contd.):
A short-term, contemporary study:
• examines what is occurring in the case
now.
• Or in multiples cases
• The researcher observes what occurs and
asks people to talk about and explain what
is going on.
Case Study Approach

Case studies vary in their

approach to time (contd.):
A longitudinal study:
• the researcher investigates the
same case over time
• anything from one month to
several years
• the processes and relationships
which occur, and which are
continuous, and those that
change, are analysed.
Selection of cases

A case study focuses on one

instance of the ‘thing’ that is
to be investigated.

Important to choose the right

instance!

You should be able to justify this

choice to the readers of your
research.
Selection of cases

Your choice of a particular case might be

based on a:
1. Typical instance:
• the chosen case is typical of many others
and can therefore stand as representative of
the whole class.
• Findings from the one case should be
generalizable to the whole class.
2. Extreme instance:
• the case is not typical (unusual or
special) but provides a contrast with the
norm (e.g., an outstanding success;
notable failure; exotic event)
Selection of cases

Your choice of a particular case might be based on (contd.):

3. Test-bed for theory:
• Choose a case that contains elements that make it suitable for
testing an existing theory.
• Investigate whether the theory holds true in the particular case,
• or whether the theory must be challenged or modified.
4. Convenience:
• People in the chosen case have agreed to give you access.
• It is convenient in terms of time and resources
• This should not be main reason why you choose the case!
Selection of cases

Your choice of a particular case might be based on:

5. Unique opportunity:
• You get a chance to study something that you
had not previously planned for, and that may
not occur again.
• Perhaps you were in the right place at the right time
• You happened to talk to the right person.
• Good researchers watch out for such
opportunities.
Generalization

What is generalization?
• the degree to which you can apply the results
of your study to a broader context
• Why is it important?
• The goal of research is to produce
knowledge that can be applied as widely as
possible.
• When are results considered generalizable?
• when the findings can be applied to most
contexts, most people, most of the time.
Generalization

Main critique on case studies:

• They produce knowledge that only relates to the case under study.
• BUT: it is possible to generate broader conclusions that are
relevant beyond the case itself = GENERALIZATIONS
• Can be made to the extent that your chosen case is typical of other
cases.
• E.g., if you have studied a small manufacturing company's use of
IT, your findings may also apply to other small manufacturing
companies.)
Generalizations

Main critique on case studies (contd.):

• It's important that you inform your readers
about:
• The extent to which you judge your chosen case
as typical of other cases in its class
• Whether you feel your chosen case is unique
• You should also give your readers sufficient detail
about the case so that they can decide if your
case is similar to one with which they are familiar.
Generalizations

Main critique on case studies (contd.):

• Cases might be similar on the basis of:
• their physical location,
• history,
• social mix,
• technical basis,
• organizational type.
• rich insight
• or a combination of these.
Generalizations

Main critique on case studies (contd.):

• Seminal work by Walsham (1995) - Cambridge
• He suggests that four main types of
generalizations are possible from case study
research:
• concepts,
• theory,
• implications and
• rich insight
• OR a combination of these.
Generalizations

Generalizations from a case study:

• A concept:
• a new idea or notion that emerges
from the analysis
• it may even require a new word to be
added to the vocabulary of the
research discipline
Generalizations

Generalizations from a case study

(contd.):
• A theory:
• a coherent collection of concepts and
propositions with an underlying
worldview.
• it might be expressed as a ‘conceptual
framework’ and is often presented via a
diagrammatic model.
Generalizations

Generalizations from a case study

(contd.):
• Implications:
• suggestions about what might happen
in other similar instances
• possibly with specific
recommendations for action.
Generalizations

Generalizations from a case study:

• Rich insight:
• ‘things’ from a case study that does not fit neatly into the other
three categories BUT which give us an important new
understanding about a situation.

* Read Bent Flyvberg’s (2004) paper on: Five

misunderstandings about case-study research – uploaded on
clickUP
Case Studies’ relationship to theory

We can use a case study to:

1. Build a new theory:
• It can lead to the development of a new concept, theory, framework or
model.
• These can then be applied by researchers to other situations, possibly via
another research strategy.
• E.g., a case study could lead to a conceptual framework that could be used to
analyse another case study, or to new hypotheses that could be explored via
experiments or a survey.
Case Studies’ relationship to theory

We can use a case study to (contd.):

2. Test an existing theory:
• take an existing theory and use a case study to see if the empirical evidence gained:
• confirms the theory,
• implies necessary modifications to it, or
• contradicts it.
• E.g.:
• there is a substantial amount in the literature about technology acceptance – under what circumstances a
new IT-based system will be accepted by the users – building on Davis's (1989) technology acceptance
model, and the extended model (TAM3) (Venkatesh & Bala, 2008).
• A case study could look at one instance of the introduction and implementation of a new technical system
and see whether the pattern of user acceptance conforms (or not) with what the theory suggests.
Case Studies’ relationship to theory
We can use a case study to (contd.):
3. Evaluate alternative theories:
• examine all the factors in a case and see which pre-existing theory or model
best matches what was found in the case.
Case Studies’ relationship to theory

The theory can also be used to SHAPE the case study:

help the researcher decide:
• which case to choose,
• what data to collect,
• which questions to ask and
• what themes to look for during data analysis.
DANGER:
• the researcher might be too rigid in sticking to the chosen theory, and so not
see other interesting and relevant issues in the case
Case Studies’ relationship to theory

SO: rather follow a grounded theory approach (developed by Glaser & Strauss –
1967):
• Enter the situation with no preconceived ideas about theory,
• collect some data
• analyse it for emerging patterns and themes, and
• start to develop a theory.
This emerging theory is used to guide what further data is collected which in turn
leads to further modification of the emerging theory.
• Only when new data seems to confirm the researcher's theory, rather
than add anything new, does the case study come to an end.
• This theory can then be compared with existing theories in the
literature.
Case Studies’ relationship to theory

The argument for the grounded theory approach:

• any explanations or models or frameworks that the researcher proposes as
outcomes from the case study are well grounded in empirical data.
• They are not being dreamt up and only afterwards checked to see if they
work in reality.
• Grounded theory has become a mainstream methodology in the IS
discipline
Underlying philosophy

Case study strategy is not associated with just one underlying

research philosophy.
It can be used in research that has an underlying philosophy of
positivism, interpretivism or critical thinking.
Each paradigm has a different set of criteria for judging what
makes a case study one of high quality.
Reporting (writing up the case study)
Unlikely that readers of your research report will
be able to visit the same case to make their
own observations and analysis.
They are dependent on your account of what
was found and how it should be interpreted.
Essential to describe (in some detail) the
nature of the case and how you arrived at
your conclusions.
Provide background information about the
case and why it was chosen, who you
interviewed or observed, what documents
you used, and the timescale of your research.
Reporting (writing up the case study)

You should explain:

• how you recorded the data,
• how you analysed it and
• how you moved between theory and empirical data.
You should recognize:
• that there may be more than one interpretation or explanation for the data that you generated.
You should explore:
• alternative interpretations and make an argument for what you consider to be the one that
best explains your data.
The need to report all of this means that your final account may be very lengthy (sometimes a problem if
your number of pages are restricted).
Research strategies: design and creation
Design and creation
Objective – to create, change and control
To create an artefact based on scientific research through
iterative cycles of design, development and evaluation
Contribution – new IT artefacts

IT application the main focus (an IT application that uses IT in a new

domain, an IT application that incorporates a new theory)
IT application the vehicle for something else (a e-learning system is
developed and focus is then on how it is used etc.
IT application is a tangible end-product but the focus is on the
development process (the contribution is a development framework and
then demonstrated by the development of an IT application)
 Normal design vs design and creation research

An academic project based on design and creation should demonstrate not only technical
skills, but also academic qualities such as analysis, explanation, argument, justification and critical
evaluation – AND IT MUST CREATE NEW KNOWLEDGE
Conducting design and creation research

It follows an iterative process following five steps:

• Awareness,
• Suggestion,
• Development (a development methodology is used – Waterfall, SSM, Agile
…),
• Evaluation (functionality, completeness, consistency, accuracy, performance,
reliability, usability, accessibility, aesthetics, entertainment, fit with
organization…..),
• Conclusion
Data generation methods in D and C
Observation,
questionnaires,
interviews….
• Awareness,
• Suggestion,
• Development (be clear which development methodology is used – Waterfall,
SSM, Agile …),
• Evaluation (functionality, completeness, consistency, accuracy, performance,
reliability, usability, accessibility, aesthetics, entertainment, fit with
organization…..),
• Conclusion

Observation,
questionnaires,
interviews….
Evaluation guidelines: Design and Creation
research

What kind of artefact was designed?

What makes this work ‘research’ and not normal ‘design’?
Which development methodology was used in the Development phase?
Did the researchers describe the data generation methods?
Did they evaluate their IT artefact? How did they do it? What evaluation
criteria did they use?
Which limitations do they recognize?
How effectively do you think it was reported?
Research strategies:
Experiments
Experiments
Objective – to understand causation and to predict

To prove or disprove a causal link between a factor(s) and an observed

outcome by controlling the factors possibly influencing an outcome
Planning and conducting experiments

Important questions to ask

• What is the hypothesis?
• What are the independent and dependent variables?
• How can the variables be controlled? (e.g. eliminate the factor from the
experiment, control groups, placebo etc.)
• Internal validity and external validity?
Planning and conducting experiments

▪ Hypothesis: a statement which has not been tested empirically yet but for
which it will be possible to set up experimental tests to reject it or accept it.
When factor A occurs, B will happen
▪ Independent (cause) and dependent variables (effect)
▪ Control other variables (e.g. eliminate the factor from the experiment, hold
the factor constant, random selection of subjects, control groups, double-
blind experiments)
▪ Observation and measurement (pre and post tests – what will be
measured and how)

95
Internal validity

Internal validity: if the measurements are indeed due to your

manipulations of the independent variable and not to any other factors.
Threats:
• Difference between experimental and control groups
• History
• Maturation
• Reactivity and experimenter effects
External validity

External validity: if your results are not unique to a particular set of

circumstances but are generalizable – best way to show that is to repeat
the experiment.
Threats:
• Over-reliance on special types of participants,
• Too few participants,
• Non-representative participants,
• Non-representative test cases.
Quasi-experiments / Field experiments

When you manipulate the independent variables but do not control the
other variables that might also have an influence – since the experiment
is conducted in the ‘field’ or natural setting.

There are many possible experimental designs but there are always pre-
and post-measurements.
Evaluation guidelines: Experiments

Was a hypothesis clearly stated in the introduction?

What type of experiment was this (true experiment, field experiment or
uncontrolled trail?)
What info is given about the dependent and independent variables?
What information is given about the participants and how they were found?
What info is given about how representative the sample is?
What info is given about apparatus/measurement instruments used to do the
measurements?
Do the authors recognize any limitations?
Assuming the statistical analysis was correct, have the researchers convinced
you that they have demonstrated cause and effect?
Research strategies:
Action research
Action research
Objective – to control and change
To investigate and participate in the improvement of a practical issue
through an iterative cycle of plan-act-reflect.
Characteristics of action research

• Focus on practical issues

• An iterative cycle of plan-act-reflect
• An emphasis on change
• Collaboration with practitioners
• Multiple data generation methods
• Action outcomes plus research outcomes
Planning and conducting AR

Checkland (1981)suggests that AR should be conceptualized as

F framework of ideas
M problem solving methodology
A An area of application

As an example (objective is to improve the use of LMS at UP)

F – Systems theory
M – Soft Systems Methodology
A – ClickUP at UP
The research process

Diagnosis: identifying the nature of the problem situation, including all interrelated factors, and
developing a working theory (F) about the situation and how it might be changed.
Planning: specifying actions that should alleviate the situation (M).
Intervention: taking action in the agreed area of application (A) in line with the plan.
Evaluation: establishing whether the theoretical effects of the action were realized, and whether they
did indeed relieve the problem(s).
Reflection: deciding what has been achieved in terms of both practical outcomes and new
knowledge, and whether an alterative action research cycle is required.

Not that easy to plan ahead…

Participation between researchers and practitioners

Participation of practitioners is important! Researchers and practitioners work together. Practitioners

must agree to the changes suggested and be committed to the research objectives and
implementing the new ideas.

Mutually acceptable research protocol – practitioners must agree with objectives, the role of the
participants and any organizational constraints.

Different from consultancy (consultants work exclusively for a client, whereas action
researchers work with both client practitioners and also need to report to a broader
research community with their findings)
 Outcomes
Outcomes relate to action (practical achievements)
AND
Outcomes relate to research(learning about the process of problem solving and acting in
a situation)

Can action research outcomes be generalized?

It is important not to make broad generalizations from one action research study.
However, you should reflect upon the extent to which your chosen problem situation is
typical of other settings, so your outcomes might be applicable elsewhere. You should
also give sufficient information about the problem such that readers can decide whether
it is applicable to other settings as well.

Models, theories and methods, developed from the action research, are all forms of
generalizations that often have wider applicability.
Alternative action research

Action research is a participatory process concerned with developing

practical knowledge in the pursuit of worthwhile human purposes. It
seeks to bring together action and reflection, theory and practice, in
participation with others, in the pursuit of practical solutions to issues of
pressing concern to people, and more generally the flourishing of
individual persons and their communities. (Reason & Bradbury, 2007, p.
4)
Evaluation guide: Action Research

• Did the work involve an iterative cycle of plan–act–reflect? How many cycles are described? Do you
think this is enough?
• Can you identify if the researchers make explicit their framework of ideas (F), methodology (M) and
area of application (A)?
• What data generation methods were used? Do you think enough methods were used and enough
data collected?
• Do the researchers discuss the extent of participation achieved, and any limitations in their claimed
outcomes caused by a lack of full participation?
• Do the researchers recognize the problems of self-delusion or group-think, and explain adequately
how they addressed them?
• What practical and research outcomes and generalizations do the researchers claim from the action
research?
• What limitations in the action research do the researchers recognize?
• Can you identify other flaws or omissions in the researchers’ reporting of the action research study?
• Overall, how effectively do you think the action research strategy has been reported and used?
Research strategies:
Ethnography
Ethnography
Objective – to describe, understand and explain
To become immersed in a culture to reach a deep understanding thereof
by
collecting data,
reflect on their own processes,
keeping field notes,
acknowledge that they might have impacted on the people of that culture,
link what they have observed to previous literature and
write up the process and findings.
Ethnography

What is it like to be these people?

Characteristics

The ethnographer spends time in the field, taking part in the life of the people there; that
is, carrying out participant observation (see Chapter 14) rather than being a detached
observer.
The ethnography does not take place in an artificial experimental setting, but in the
natural setting of the subjects, which, as far as possible, should be undisturbed by the
presence of the ethnographer.
The ethnographer becomes the research instrument, using multiple data generation
methods, such as interviews, observations and documents (Chapters 13–16) and,
especially, copious personal field notes about what they see, feel and experience.
The ethnographer tries to construct a representation of the world as perceived by the
people who live in that world. The test of success of this is whether those people
recognize the ethnographer's description of familiar features of their own culture.
The ethnographer tries to produce a holistic description of the culture, including social,
cultural and/or the economic aspects of the situation, rather than just concentrating on
one or two aspects of life in that world.
Planning and conducting Ethnography

Field notes – apart from data collection, the researcher should keep note
on metadata of the data collection methods, notes and analysis
Should be placed in a theoretical context
Reflexivity – researcher should observe others and participate with them,
but at the same time, stand back and observe themsevels observing and
participating - they must understand their own preconceptions, beliefs
etc.
Writing – creative act and always partial and possibly one-sided. This
should be recognized
Test of success – If the readers are able to understand the activities of
people in another culture and see that they make sense within the
context of that culture.
Evaluation guidelines: Ethnography

Did the research focus on lifestyles, meanings and beliefs?

What data generation methods were used? Do you think enough methods were
used and enough data collected?
How long did the researcher spend in the field? Do you think this was long
enough?
Does the work seem to belong to the holistic, semiotic or critical school?
Is the ethnography a standalone description, or is it linked to theory, other
ethnographies or issues in the researcher's own culture?
Does the paper include an account of the researcher's self?
Has the researcher acknowledged that the ethnography is a construction rather
than a literal description?
What limitations in the ethnography does the researcher recognize?
Summary

The research paradigm, research questions and research strategy are

interrelated.
Same research problems can be approached from different paradigms,
different strategies with differently formulated research questions.