Formulating a Research Question: Overview

Formulating Research Hypotheses

● Establishing a background
○ Choosing a topic
■ Interest
■ Feasibility
● Formulating a question
○ Refining a topic into a question
○ Characteristics of a good research question
○ Sources of ideas
● Reviewing the literature
○ Purposes of the literature review
■ Identify if the topic has been researched
■ Provide design ideas
■ Identify methodological problems
■ Identify special needs in terms of research participants or equipment
■ Provide information for the research report
○ Types of information
■ Relevant theories
■ Types of participants
■ Methods and procedures used
○ Primary versus secondary sources
■ Primary sources → researchers published their own work
■ Secondary sources → review of primary source (e.g., books, reviews)
○ Where to find information
○ Library research tools
○ Evaluating information
● Formulating hypotheses
○ Research hypotheses
○ Statistical hypotheses
● Designing the study
○ How?
○ What?
○ Where?
○ Whom?
○ When?
● Writing the proposal

What’s a good research question?

Knowledge base
● Theory-Theory map
○ State theory
 ○ Evidence
○ Competing theories?

Researchable?

Importance

Evaluating Web Pages

● Main problems with information from the web is the validity of the information

Criteria to help you evaluate validity

● Authority
○ Author and their credentials
○ Preferred domains .edu, .org, .gov
○ Qualifications of the publisher of the page
● Accuracy
○ Credentials and email address for contact
○ Purpose of the information
○ Determine if there is acknowledgment of limitations of the information
● Objectivity
○ Identify any evidence of biases
○ Advertising would suggest bias to sell something
○ Is the information traceable to factual information of references
○ Author’s opinions suggest bias
● Currency
○ Regular updates to the Web page
● Coverage
○ Being able to view the information without paying fees or having additional
software requirements
○ Exception might be scientific journals, but you should be able to access them
through the library’s website for no additional cost to you

Evaluating Research Reports

● Internal validity
● Construct validity
● Statistical validity
○ Were the data analyzed properly?
○ Did they use the right test?
○ Did they test the right groups?
● Generalization
 ○ Good external and ecological validity?
■ Realistic enough?
● Data to conclusions
● Value of the research

Formulating a Research Hypothesis

Writing the Research Proposal

Replication Research

Forms of replication
● Exact → copying the research study
● Conceptual → using the same hypothesis but not repeating the study exactly the same
○ Change types of participants, scales used, procedure, etc.

Implications of Replication Research

● Support and extend principle
● Damage or limit principle

Considerations in conducting replication research

● Importance of hypothesis
● Avoiding overduplication

Designing Research for Utilization

Knowledge Utilization
● Truth test
○ Quality and benefit for user
● Utility test
○ Does it challenge the status quo or can results be used
○ Perception of usefulness to be used in practice
■ Policy makers
■ Clinicians
■ Organizations and corporations

Design Considerations
● Independent variables
○ Policy variables - those that can be controlled
○ Estimator variables - affect behavior
 ● Dependent variables
○ Utility
● Research Population
● Research Settings

Dissemination of Research Results

Bias in the Formulation of Research Questions

Biased assumptions - Generalizations to other groups

● White males, Gay parents, Social Power

Biased theory

Avoiding bias
● Ask questions at each step

Development a Measurement Strategy: Overview

● Reliability and Validity

○ Manifest variables - what we observe
○ Hypothetical constructs - underlying dimensions
○ Reliability, validity, and measurement error
■ Reliability = consistency
● E.g., IQ
■ Validity = accuracy
● E.g., Galvanic Skin Response (GSR) and truth telling
■ Relationship between reliability and validity
■ Measurement error
● Observed score = true score + measurement error
● Random error
● Systematic error
● Assessing Reliability
○ Forms of reliability
■ Consistency across time
● Test-retest
■ Consistency across forms
● Alternate form reliability
● Inter-rater % agreement. Cohen’s Kappa
■ Consistency across items
● Split-half reliability
● Internal consistency (Cronbach’s alpha)
 ○ Choosing among forms of reliability
○ Standards for reliability
■ What form to use?
■ What’s good?
● Assessing Validity
○ Categories of validity evidence
■ Generalizability
● Groups (gender, SES, class year in school, ethnicities, etc.)
● Time (day of the week, time of the year, how many times, etc.)
● Settings
■ Content validity
● Items or content should be relevant
● Representative
■ Structural validity
● Factor structure
○ Number of factors based on theory
■ External validity
● Criterion
● Contrasted groups
● Prediction of behavior
■ Substantive validity
● The measure actually works the way it should
○ E.g., people who score high on authoritarianism should
function the way it is expected that such people behave
■ Discriminant validity
● Previous measures of validity are convergent
● Measure doesn’t measure something else
○ E.g., autism and intellectual disability
○ E.g., Shyness and anxiety
● Campbell and Fiske-Multi-trait, multi-measure
■ Differential validity
● Cross cultural context
● Assessing differential validity
● The extent of the problem
○ Relationships among the categories for validity evidence
○ Determining a measure’s degree of validity
● Modalities of Measurement
○ Self-report measures
■ Advantages:
● quicker response
■ Limitations:
● not understand scope, don’t take it seriously, socially-desirable
responses, can lie on survey, unreliable memory
○ Behavioral measures
 ■ Advantages:
● see the behavior, observable
■ Limitations:
● time-consuming, labor-intensive, costly, have to be trained, have
to be there, dependent on the situation, behavior changed
because being watched
○ Physiological measures
■ Advantages:
● concrete evidence, hard to lie
■ Limitations:
● a lot of variables (unstable, e.g., HR, sweating), inconvenient, very
expensive, constant calibration
○ Implicit measures
■ Advantages:
● can’t change response/defend against
■ Limitations:
● doesn’t always work
○ Choosing a modality
● Locating and Evaluating Measures
○ Categories of Measures
■ Manifest variables - directly observable
■ Hypothetical constructs - not directly observable
● Psychometric tests
○ Normed
● Research measures
○ Means scores
■ Developed research measures
■ Ad hoc research measures
○ Locating Measures
■ Mental Measurement Yearbook (Plake & Spies, 2007)
■ Tests in Print (Murphy, Spies, & Plate, 2006)
■ Directory of Unpublished Experimental Measures (Goldman & Mitchell,
2008)
■ Measures of Personality and Social Psychological Attitudes (Robinson,
Shaver, & Wrightsman, 1991)
○ Evaluating Measures
■ Theoretical background
■ Quality of development
■ Reliability and validity
■ Freedom from response bias
● Social desirability
● Acquiescence
■ Comparing measures
Process of Measurement

● What to measure
● Measurement strategy - what modality?
● Evaluate measures
● Apply the measures