CL-05: Designing Linguistic Research

Prof. Philip Thierfelder

2024-09-30

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 1 / 47

 Section 1

Fundamentals of Study Design

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 2 / 47

Purpose of Quantitative Research

Quantitative research in linguistics seeks to test hypotheses and

identify patterns in language behavior through the collection and
analysis of numerical data.
This approach relies on statistical methods to ensure that findings
are not the result of chance, but reflect reliable and generalizable
trends.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 3 / 47

Common Goals

Measure linguistic phenomena: For example, analyzing word

frequency, sentence complexity, or phonetic variation.
Compare groups or conditions: Investigating how different factors
(e.g., language proficiency, exposure to stimuli) affect language
processing or acquisition.
Establish cause-effect relationships: Using controlled experiments
to determine how specific variables influence language outcomes (e.g.,
how background noise affects speech comprehension).

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 4 / 47

Key Elements in Study Design

Defining Variables: Establishing your independent variables (what

you manipulate) and dependent variables (what you measure).
Choosing a Design: Deciding between within-subjects and
between-subjects designs based on the research question.
Sample Selection: Identifying your target population and ensuring
your sample size is adequate for statistical analysis.
Assigning Participants: A critical step in ensuring valid and reliable
results.
Proper assignment methods (random, matched) can control for
individual differences and help reduce bias.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 5 / 47

What Are Carryover Effects?

In a within-subjects design, each participant experiences multiple

conditions.
Carryover effects occur when the effects of one condition influence
performance in subsequent conditions.
This can distort the results, as differences between conditions may be
due to the order in which they are presented, rather than the
manipulation itself.
Example:
In a study on sentence complexity, participants might first read simple
sentences, which could help them process complex sentences more
easily later, not because they have better comprehension but because
they’ve become familiar with the task or content.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 6 / 47

Strategies to Reduce Carryover Effects

Filler Items: Introduce unrelated tasks or stimuli between the

experimental conditions to distract participants and reduce the
likelihood that their performance in one condition will affect the next.
Counterbalancing: Present conditions in different sequences across
participants to balance out the order effects.
Time Gaps: Insert breaks between conditions to minimize the
influence of the first condition on the next.
Washout Periods: Allow time between conditions for the effects of
the first condition to wear off.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 7 / 47

A Note About Carryover Effects

In some designs, carryover effects may be difficult or impossible to

avoid.
Controlling for Trial Number in the statistical model may help
reduce learning effects from repeated trials.
e.g.,: Reaction Time ~ Condition + Trial_Number

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 8 / 47

 Section 2

Assigning Participants to Groups

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 9 / 47

Within-Subjects Design

Each participant is exposed to both conditions (e.g., simple and

complex sentences).
This design allows for direct comparison of performance within the
same individual, controlling for individual differences.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 10 / 47

Within-Subjects Design Example

Study on Sentence Complexity and Comprehension:

Each participant reads both simple and complex sentences during the
experiment.
Condition 1: Sentences with simple syntax (e.g., “The teacher gave
the student a book.”)
Condition 2: Sentences with complex syntax (e.g., “The book that
the teacher gave to the student was new.”)
Participants are exposed to both conditions in random order to avoid
any order effects (e.g., half of the participants start with simple
sentences, and the other half start with complex sentences).

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 11 / 47

Between-Subjects Design

Participants are assigned to one condition only.

This design is useful when exposure to multiple conditions may
influence results (e.g., carryover or learning effects).

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 12 / 47

Between-Subjects Design Example

Study on Music and Language Learning: Researchers are

investigating the effects of background music on vocabulary retention.
Participants are randomly assigned to one of two groups:
Group 1: Participants learn vocabulary while listening to classical
music.
Group 2: Participants learn vocabulary in silence.
Group Assignment: Participants are randomly assigned to one of
the two groups, ensuring that individual differences (e.g., prior
musical experience, language proficiency) are equally distributed
across the groups.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 13 / 47

Matched Groups

If individual differences (e.g., age, language proficiency, cognitive

ability) may significantly influence the outcome, you should use
matched groups to ensure balance across conditions.
This can be applied in either within- or between-subjects designs.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 14 / 47

Example 1: Cognitive Load and Language
Proficiency

Research Focus:
Investigating how bilinguals with different levels of language
proficiency process words in their second language (L2).
Condition 1:
Bilinguals perform a word recognition task in their second language
under low cognitive load (e.g., no additional tasks).
Condition 2:
Bilinguals perform the same word recognition task, but under high
cognitive load (e.g., performing a secondary task like memorizing
numbers).
Why Matching?
Participants are matched on language proficiency so that all members
of each group have similar proficiency (e.g., high and low proficiency
groups).

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 15 / 47

Example 2: Comparing Bilinguals and Trilinguals

Research Focus:
Investigating whether speaking more than two languages (bilingualism
vs. trilingualism) affects cognitive flexibility or lexical access.
Key Concept:
To ensure that any performance differences are due to language
experience (bilingual vs. trilingual) and not other factors, participants
are matched on key characteristics.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 16 / 47

Example 2 Group Assignment and Matching

Group 1:
Bilinguals (speak two languages fluently).
Group 2:
Trilinguals (speak three languages fluently).
Matching Criteria:
Age
Years of formal education
General cognitive ability (e.g., using a standardized cognitive test)
Language Proficiency
Why Matching?
This ensures that any observed differences in performance are due to
the number of languages spoken, rather than other variables.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 17 / 47

Control and Comparison Groups

Provide a baseline to measure the effect of the independent variable.

Ensure that observed effects are due to the manipulation, not other
factors.
Example:
In a language learning study, the control group receives no
intervention, while the experimental group receives the new teaching
method.
Comparing control and intervention groups helps isolate the effect of
the intervention.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 18 / 47

 Section 3

Background Measures

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 19 / 47

Why Collect Background Measures?

Control for Confounding Variables:

Collecting background information helps control for factors that might
influence outcomes (e.g., age, education, language proficiency).
Enhance Generalizability:
Allows researchers to understand the characteristics of their sample and
assess how representative it is of the population.
Ensure Balanced Groups:
If using a between-subjects design, background measures help ensure
groups are balanced on key characteristics like age, gender, or
proficiency.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 20 / 47

Key Background Measures to Collect: Demographic
Information

Age: Can influence cognitive performance, processing speed, or

language acquisition.
Note: Older adults may process language differently than younger
adults.
Gender: May be relevant in specific research contexts (e.g., language
and gender studies).
Education Level: Higher education often correlates with better
performance in certain language tasks, such as reading
comprehension.
Handedness: May be important for neurophysiological studies.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 21 / 47

Key Background Measures to Collect: Language
History

First Language (L1): Important for studies involving bilinguals or

multilinguals.
Second Language (L2) Proficiency: Collecting self-reported or
standardized test scores can help categorize participants by
proficiency level.
Note: There are numerous short tests available to estimate language
level, LexChi
Age of Acquisition: When participants learned their second
language can affect their proficiency and cognitive processing.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 22 / 47

Key Background Measures to Collect: Cognitive
Measures

Memory Capacity: Working memory plays a critical role in language

processing.
Example: Participants with higher working memory capacity may
perform better in sentence processing tasks.
Attention Control: In studies involving multitasking or divided
attention, measuring attention control can be useful.
Example: In a study on language comprehension in noisy environments,
participants with higher attention control might perform better.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 23 / 47

Example: Collecting Background Measures

Study: Bilingualism and Cognitive Flexibility

Background Measures Collected:
Age: To control for cognitive aging effects.
Language Proficiency: Participants self-report proficiency in L1 and
L2, and take a standardized language proficiency test.
Years of Education: To control for general cognitive ability.
Working Memory: Assessed via a working memory span task.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 24 / 47

Why These Measures?

Age and education are controlled to ensure cognitive differences are

not due to these factors.
Language proficiency helps categorize participants into high and low
proficiency groups.
Working memory is collected to understand its role in cognitive
flexibility tasks.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 25 / 47

How to Collect Background Measures

Self-Report Questionnaires:
Language History Questionnaire: Collects information on L1, L2,
age of acquisition, and daily language usage.
Ad hoc and Standardized Tests:
Language Proficiency Tests: (e.g., TOEFL, IELTS) to objectively
assess language skills.
Cognitive Tests: (e.g., working memory tasks) to measure cognitive
factors that may influence language processing.
Interviews:
In-depth interviews can be used to gather detailed language history or
other relevant background information.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 26 / 47

 Section 4

Matching Items

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 27 / 47

Why It’s Important

Control for Confounding Variables:

Matching item properties (e.g., word length, frequency) ensures that
observed differences are due to the experimental manipulation, not
differences in the stimuli.
This should often be done with visual stimuli as well.
Isolate the Effect of the Manipulation:
By matching properties such as word length, frequency, or
concreteness, we can be confident that any observed differences are
caused by the independent variable (e.g., cognitive load), not inherent
differences in the items.
Improve Internal Validity:
Matching reduces variability in the stimuli, making it easier to detect
the true effect of the independent variable.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 28 / 47

 Section 5

Essential Components of Quantitative Research

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 29 / 47

Internal Validity

Definition:
The degree to which the observed effects can be confidently
attributed to the independent variable (i.e., the treatment or
manipulation).
Enhancing Internal Validity:
Use random assignment, control groups, and matching, etc. to
reduce potential biases and confounds.
Example:
If a study finds that reaction times differ between groups, internal
validity ensures that this difference is due to the word frequency
manipulation and not some other factor.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 30 / 47

External Validity

Definition:
The extent to which the results can be generalized beyond the specific
study sample to other populations, settings, or times.
Enhancing External Validity:
Use a diverse participant sample and replicate the study in different
contexts.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 31 / 47

Reliability

Definition:
The consistency of the measurement across different raters, times,
or instruments.
Enhancing Reliability:
Use well-validated instruments and ensure that data collection
procedures are standardized.
Example:
If multiple researchers are coding grammatical accuracy, ensure they
are trained to apply the criteria consistently.
Check for reliability across different raters with an inter-rater
reliability test.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 32 / 47

Replicability

Definition:
The ability to replicate study results with different populations,
contexts, or times.
Importance:
Replicability enhances confidence in the robustness of findings and is
key to building reliable scientific knowledge.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 33 / 47

 Section 6

Experimental Measures

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 34 / 47

Understanding Your Measurements

Using the right measure is critical for answering your research

question.
Different measures capture different aspects of cognitive processing.
Understanding the strengths and limitations of each measure helps
you draw meaningful conclusions.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 35 / 47

Example Measures

ERP (Event-Related Potentials):

Captures time-locked brain activity in response to stimuli (measured
via EEG).
Excellent temporal resolution (milliseconds).
Useful for understanding when a cognitive process occurs.
Reaction Time (RT):
Measures the speed of a response to a stimulus.
Widely used to infer cognitive processing speed.
Sensitive to decision-making and response execution.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 36 / 47

Example Measures (Continued)

Eye Fixations:
Tracks where and how long participants look at specific stimuli.
Provides insight into visual attention and information processing.
Can reveal real-time cognitive strategies during tasks.
Pupillary Response:
Measures changes in pupil size in response to stimuli.
Reflects cognitive load and emotional arousal.
Pupil dilation is often linked to effortful processing.
Accuracy:
Measures the correctness of responses.
Simple but effective for assessing task performance.
Often used in conjunction with RT to understand the speed-accuracy
tradeoff.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 37 / 47

Why Understanding Differences Matters

Temporal vs. Spatial Resolution:

ERP is ideal for understanding when processes occur but offers limited
spatial information.
Eye tracking and pupillary response provide insight into where
attention is focused but have lower temporal precision.
Processing Effort vs. Outcome:
RT and accuracy measure outcomes (speed and correctness), but
don’t reveal details about underlying cognitive processes.
ERP and pupillary response can provide information about
processing effort and cognitive load, even if the task outcome is the
same.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 38 / 47

Choosing the Right Measure

What’s Your Research Question?:

Use ERP if you’re interested in when processing occurs.
Choose eye tracking for questions about visual attention and time
course.
Use pupillary response to assess cognitive load or emotional
arousal.
Consider RT and accuracy for assessing task performance and
decision-making.
Complementary Measures:
Combining multiple measures (e.g., RT + ERP or Eye Fixations +
pupillary response) can give you a fuller picture of cognitive
processing.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 39 / 47

Practical Example

Language Processing Study:

ERP might reveal early stages of lexical access (~ 200 ms after word
onset).
RT could indicate how long it takes to make a decision about word
meaning (more reflective of conscious processes).
Accuracy could measure how often participants correctly identify word
meanings.
Eye fixations can show which words participants focus on during
reading and processing time.
Pupillary response might reveal increased cognitive load when
processing difficult words.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 40 / 47

 Section 7

Factorial Designs

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 41 / 47

Overview of Factorial Designs

Factorial Designs allow researchers to examine the effects of

multiple independent variables (IVs) simultaneously.
This design enables the study of:
Main effects: The effect of each IV on the dependent variable (DV).
Interaction effects: How IVs combine to influence the DV.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 42 / 47

Example of Factorial Design

Example:
In a language processing study, you manipulate:
Word frequency (high vs. low)
Sentence complexity (simple vs. complex)
This allows you to explore the individual and combined effects of
these variables on comprehension.
Structure:
Each IV has levels (e.g., high vs. low frequency).
A 2x2 factorial design results in 4 conditions:
High-frequency, simple sentence
High-frequency, complex sentence
Low-frequency, simple sentence
Low-frequency, complex sentence

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 43 / 47

Types of Effects in Factorial Designs

Main Effects:
The impact of a single IV on the DV, ignoring other IVs.
Example:
A main effect of word frequency: Participants process high-frequency
words faster than low-frequency words, regardless of sentence
complexity.
Interaction Effects:
Occur when the effect of one IV depends on the level of another IV.
Example:
The effect of word frequency (high vs. low) is stronger for simple
sentences but weaker for complex sentences.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 44 / 47

Complexity of Factorial Designs

Increasing Complexity:
As the number of independent variables and their levels increases, so
does the complexity of the design.
Example:
A 3 x 2 factorial design (three levels of word frequency: high,
medium, low; two levels of sentence complexity: simple, complex)
results in six conditions.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 45 / 47

Challenges and Advantages of Factorial Designs

Advantages:
Allows testing for interaction effects between variables.
Provides a more nuanced understanding of how IVs jointly impact the
DV.
Challenges:
More factors and levels increase the complexity of the design.
Don’t go crazy with complex combinations of factors!
Requires more participants and more careful data analysis to detect
effects accurately.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 46 / 47

Factorial Design Summary

Factorial designs allow for the simultaneous examination of multiple

independent variables.
Look for both main effects (impact of individual IVs) and
interaction effects (combined impact of IVs).
Factorial designs can be between-subjects, within-subjects, or
mixed.
As the number of IVs increases, the design becomes more complex,
requiring more careful planning and analysis.

Prof. Philip Thierfelder CL-05: Designing Linguistic Research 2024-09-30 47 / 47