Bridging a Gap in Coherence: The Coordination of Comprehension Processes When Viewing Visual Narratives
Abstract
:1. Introduction
1.1. Scene Perception and Event Comprehension Theory
1.2. Back-End Processes at Coherence Gaps
1.3. Current Experiments and Hypotheses
2. Experiment 1
2.1. Method
2.1.1. Participants
2.1.2. Materials
2.1.3. Procedure
We want you to identify when you think that the situation has changed in the story. Click on the picture that you judge to be the start of a new situation. How you define a change in the situation is up to you. Please keep in mind that most stories contain multiple situations that change. As such, you should be making multiple judgments when viewing a story.
2.2. Results
2.2.1. Analysis of Event Segmentation
2.2.2. Analysis of Viewing Time
2.3. Discussion
3. Experiment 2
3.1. Method
3.1.1. Participants
3.1.2. Procedure
What you will do is view picture stories and you will “think out loud” as you view them. When you think aloud, you say the thoughts that come to mind after you view each picture in the stories. It is like turning up the volume on what you are thinking in your head as you make sense of the story in the pictures.
3.1.3. Think-Aloud Coding
3.2. Results
3.2.1. Analysis of Think-Aloud Strategies
Analysis of Explanations
Exploratory Analysis of Picture Paraphrasing
3.2.2. Analysis of Event Segmentation and Think-Aloud Strategies
Analysis of Event Segmentation and Explanations
Exploratory Analysis of Event Segmentation and Picture Paraphrasing
3.3. Discussion
4. General Discussion
4.1. Theoretical Implications of This Study
4.2. Limitations
4.3. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- McCloud, S. Understanding Comics: The Invisible Art; Tundra Publishing: Northampton, MA, USA, 1993. [Google Scholar]
- Cohn, N.; Kutas, M. Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension. Neuropsychologia 2015, 77, 267–278. [Google Scholar] [CrossRef]
- Cohn, N.; Wittenberg, E. Action starring narratives and events: Structure and inference in visual narrative comprehension. J. Cogn. Psychol. 2015, 27, 812–828. [Google Scholar] [CrossRef] [PubMed]
- Hutson, J.P.; Magliano, J.P.; Loschky, L.C. Understanding moment-to-moment processing of visual narratives. Cogn. Sci. 2018, 42, 2999–3033. [Google Scholar] [CrossRef]
- Magliano, J.P.; Larson, A.M.; Higgs, K.; Loschky, L.C. The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension. Mem. Cogn. 2016, 44, 207–219. [Google Scholar] [CrossRef]
- Mayer, M. Frog Goes to Dinner; Dial Press: New York, NY, USA, 1974. [Google Scholar]
- Graesser, A.C.; Singer, M.; Trabasso, T. Constructing inferences during narrative text comprehension. Psychol. Rev. 1994, 101, 371. [Google Scholar] [CrossRef] [PubMed]
- Klin, C.M.; Myers, J.L. Reinstatement of causal information during reading. J. Exp. Psychol. Learn. Mem. Cogn. 1993, 19, 554. [Google Scholar] [CrossRef] [PubMed]
- Millis, K.K.; Golding, J.M.; Barker, G. Causal connectives increase inference generation. Discourse Process. 1995, 20, 29–49. [Google Scholar] [CrossRef]
- Myers, J.L.; Shinjo, M.; Duffy, S.A. Degree of causal relatedness and memory. J. Mem. Lang. 1987, 26, 453–465. [Google Scholar] [CrossRef]
- Trabasso, T.; Magliano, J.P. Conscious understanding during comprehension. Discourse Process. 1996, 21, 255–287. [Google Scholar] [CrossRef]
- McKoon, G.; Ratcliff, R. Inference during reading. Psychol. Rev. 1992, 99, 440. [Google Scholar] [CrossRef]
- Loschky, L.C.; Larson, A.M.; Smith, T.J.; Magliano, J.P. The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives. Top. Cogn. Sci. 2020, 12, 1. [Google Scholar] [CrossRef]
- Gernsbacher, M.A. Language Comprehension as Structure Building; Erlbaum: Mahwah, NJ, USA, 1990. [Google Scholar]
- Zwaan, R.A.; Radvansky, G.A. Situation models in language comprehension and memory. Psychol. Bull. 1998, 123, 162. [Google Scholar] [CrossRef]
- Radvansky, G.A. Across the Event Horizon. Curr. Dir. Psychol. Sci. 2012, 21, 4. [Google Scholar] [CrossRef]
- Radvansky, G.A.; Zacks, J.M. Event boundaries in memory and cognition. Curr. Opin. Behav. Sci. 2017, 17, 133–140. [Google Scholar] [CrossRef] [PubMed]
- Radvansky, G.A.; Zacks, J.M. Event Cognition; Oxford University Press: Oxford UK, 2014. [Google Scholar]
- Zacks, J.M.; Speer, N.K.; Swallow, K.M.; Braver, T.S.; Reynolds, J.R. Event perception: A mind-brain perspective. Psychol. Bull. 2007, 133, 2. [Google Scholar] [CrossRef]
- Gernsbacher, M.A. Two decades of structure building. Discourse Process. 1997, 23, 3. [Google Scholar] [CrossRef]
- Hard, B.M.; Recchia, G.; Tversky, B. The shape of action. J. Exp. Psychol. Gen. 2011, 140, 4. [Google Scholar] [CrossRef] [PubMed]
- Gernsbacher, M.A. Coherence cues mapping during comprehension. In Processing Interclausal Relationships; Psychology Press: London, UK, 1997; pp. 3–21. [Google Scholar]
- Newtson, D. Attribution and the unit of perception of ongoing behavior. J. Personal. Soc. Psychol. 1973, 28, 1. [Google Scholar] [CrossRef]
- Zacks, J.M.; Tversky, B.; Iyer, G. Perceiving, remembering, and communicating structure in events. J. Exp. Psychol. Gen. 2001, 130, 1. [Google Scholar] [CrossRef]
- Gold, D.A.; Zacks, J.M.; Flores, S. Effects of cues to event segmentation on subsequent memory. Cogn. Res. Princ. Implic. 2017, 2, 1. [Google Scholar] [CrossRef]
- Newtson, D.; Engquist, G.A.; Bois, J. The objective basis of behavior units. J. Personal. Soc. Psychol. 1977, 35, 847. [Google Scholar] [CrossRef]
- Zacks, J.M.; Kumar, S.; Abrams, R.A.; Mehta, R. Using movement and intentions to understand human activity. Cognition 2009, 112, 201–216. [Google Scholar] [CrossRef] [PubMed]
- Huff, M.; Meitz, T.G.; Papenmeier, F. Changes in situation models modulate processes of event perception in audiovisual narratives. J. Exp. Psychol. Learn. Mem. Cogn. 2014, 40, 1377. [Google Scholar] [CrossRef] [PubMed]
- Kurby, C.A.; Zacks, J.M. Starting from scratch and building brick by brick in comprehension. Mem. Cogn. 2012, 40, 5. [Google Scholar] [CrossRef] [PubMed]
- Speer, N.K.; Reynolds, J.R.; Swallow, K.M.; Zacks, J.M. Reading stories activates neural representations of visual and motor experiences. Psychol. Sci. 2009, 20, 989–999. [Google Scholar] [CrossRef]
- Zacks, J.M.; Speer, N.K.; Reynolds, J.R. Segmentation in reading and film comprehension. J. Exp. Psychol. Gen. 2009, 138, 2. [Google Scholar] [CrossRef]
- Cohn, N.; Bender, P. Drawing the line between constituent structure and coherence relations in visual narratives. J. Exp. Psychol. Learn. Mem. Cogn. 2017, 43, 289. [Google Scholar] [CrossRef]
- Magliano, J.P.; Zwaan, R.A.; Graesser, A.C. The role of situational continuity in narrative understanding. In The Construction of Mental Representations during Reading; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 1999; pp. 219–245. [Google Scholar]
- Brich, I.R.; Papenmeier, F.; Huff, M.; Merkt, M. Construction or updating? Event model processes during visual narrative comprehension. Psychon. Bull. Rev. 2024, 1–10. [Google Scholar] [CrossRef]
- Papenmeier, F.; Brockhoff, A.; Huff, M. Filling the gap despite full attention: The role of fast backward inferences for event completion. Cogn. Res. Princ. Implic. 2019, 4, 3. [Google Scholar] [CrossRef]
- Magliano, J.P.; Graesser, A.C. A three-pronged method for studying inference generation in literary text. Poetics 1991, 20, 193–232. [Google Scholar] [CrossRef]
- Huff, M.; Rosenfelder, D.; Oberbeck, M.; Merkt, M.; Papenmeier, F.; Meitz, T.G. Cross-codal integration of bridging-event information in narrative understanding. Mem. Cogn. 2020, 48, 942–956. [Google Scholar] [CrossRef] [PubMed]
- Bach, M. The Freiburg Visual Acuity Test-variability unchanged by post-hoc re-analysis. Graefe’s Arch. Clin. Exp. Ophthalmol. 2006, 245, 965–971. [Google Scholar] [CrossRef]
- Kumle, L.; Võ, M.L.-H.; Draschkow, D. Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behav. Res. Methods 2021, 53, 6. [Google Scholar] [CrossRef] [PubMed]
- Mayer, M. A Boy, a Dog, and a Frog; Dial Books for Young Readers; Dial Press: New York, NY, USA, 1967. [Google Scholar]
- Mayer, M. Frog on His Own; Dial Press: New York, NY, USA, 1973. [Google Scholar]
- Mayer, M. One Frog Too Many; Dial Press: New York, NY, USA, 1975. [Google Scholar]
- Mayer, M. Frog, Where Are You? Dial Books for Young Readers; Dial Press: New York, NY, USA, 1980. [Google Scholar]
- Mayer, M.; Mayer, M. A Boy, a Dog, a Frog and a Friend; Dial Press: New York, NY, USA, 1971. [Google Scholar]
- Speer, N.K.; Swallow, K.M.; Zacks, J.M. Activation of human motion processing areas during event perception. Cogn. Affect. Behav. Neurosci. 2003, 3, 4. [Google Scholar] [CrossRef] [PubMed]
- Zacks, J.M.; Braver, T.S.; Sheridan, M.A.; Donaldson, D.I.; Snyder, A.Z.; Ollinger, J.M.; Buckner, R.L.; Raichle, M.E. Human brain activity time-locked to perceptual event boundaries. Nat. Neurosci. 2001, 4, 6. [Google Scholar] [CrossRef] [PubMed]
- Magliano, J.P.; Kopp, K.; McNerney, M.W.; Radvansky, G.A.; Zacks, J.M. Aging and perceived event structure as a function of modality. Aging Neuropsychol. Cogn. 2012, 19, 264–282. [Google Scholar] [CrossRef]
- Bates, D.M. lme4: Mixed-Effects Modeling with R; Springer: New York, NY, USA, 2010. [Google Scholar]
- Lüdecke, D.; Ben-Shachar, M.S.; Patil, I.; Waggoner, P.; Makowski, D. Performance: An R package for assessment, comparison and testing of statistical models. J. Open Source Softw. 2021, 6. [Google Scholar] [CrossRef]
- Bates, D.M.; Kliegl, R.; Vasishth, S.; Baayen, H. Parsimonious Mixed Models. arXiv 2018, arXiv:1506.04967. [Google Scholar] [CrossRef]
- Matuschek, H.; Kliegl, R.; Vasishth, S.; Baayen, H.; Bates, D.M. Balancing Type I error and power in linear mixed models. J. Mem. Lang. 2017, 94, 305–315. [Google Scholar] [CrossRef]
- Barr, D.J.; Levy, R.; Scheepers, C.; Tily, H.J. Random effects structure for confirmatory hypothesis testing: Keep it maximal. J. Mem. Lang. 2013, 68, 3. [Google Scholar] [CrossRef]
- Robin, X.; Turck, N.; Hainard, A.; Tiberti, N.; Lisacek, F.; Sanchez, J.C.; Müller, M. pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform. 2011, 12, 1–8. [Google Scholar] [CrossRef] [PubMed]
- Wagenmakers, E.-J.; Farrell, S. AIC model selection using Akaike weights. Psychon. Bull. Rev. 2004, 11, 1. [Google Scholar] [CrossRef] [PubMed]
- Lo, S.; Andrews, S. To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. Front. Psychol. 2015, 6, 1171. [Google Scholar] [CrossRef] [PubMed]
- Brysbaert, M.; Stevens, M. Power analysis and effect size in mixed effects models: A tutorial. J. Cogn. 2018, 1, 9. [Google Scholar] [CrossRef]
- Green, P.; MacLeod, C.J. SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods Ecol. Evol. 2016, 7, 4. [Google Scholar] [CrossRef]
- Cohn, N.; Magliano, J.P. Editors’ introduction and review: Visual narrative research: An emerging field in cognitive science. Top. Cogn. Sci. 2020, 12, 197–223. [Google Scholar] [CrossRef]
- Öncel, P.; Hu, S.; Ness-Maddox, H.; Allen, L.K.; Magliano, J.P. Exploring the affordances of text and picture stories. Discourse Process. 2024, 61, 203–221. [Google Scholar] [CrossRef]
- Magliano, J.P.; Dijkstra, K.; Zwaan, R.A. Generating predictive inferences while viewing a movie. Discourse Process. 1996, 22, 199–224. [Google Scholar] [CrossRef]
- Clark, H.H. Inferences in comprehension. In Basic Processes in Reading: Perception and Comprehension; Routledge: Oxford, UK, 1977; pp. 243–263. [Google Scholar]
- Suh, S.Y.; Trabasso, T. Inferences during reading: Converging evidence from discourse analysis, talk-aloud protocols, and recognition priming. J. Mem. Lang. 1993, 32, 279–300. [Google Scholar] [CrossRef]
- Loschky, L.C.; Larson, A.M.; Magliano, J.P.; Smith, T.J. What would Jaws do? The tyranny of film and the relationship between gaze and higher-level narrative film comprehension. PLoS ONE 2015, 10, e0142474. [Google Scholar] [CrossRef]
- Magliano, J.P.; Graesser, A.C.; Eymard, L.A.; Haberlandt, K.; Gholson, B. Locus of interpretive and inference processes during text comprehension: A comparison of gaze durations and word reading times. J. Exp. Psychol. Learn. Mem. Cogn. 1993, 19, 704. [Google Scholar] [CrossRef]
- Delogu, F.; Drenhaus, H.; Crocker, M.W. On the predictability of event boundaries in discourse: An ERP investigation. Mem. Cogn. 2018, 46, 315–325. [Google Scholar] [CrossRef]
- Calvo, M.G. Relative contribution of vocabulary knowledge and working memory span to elaborative inferences in reading. Learn. Individ. Differ. 2005, 15, 53–65. [Google Scholar] [CrossRef]
- Gernsbacher, M.A.; Varner, K.R.; Faust, M.E. Investigating differences in general comprehension skill. J. Exp. Psychol. Learn. Mem. Cogn. 1990, 16, 430. [Google Scholar] [CrossRef]
- McCarthy, K.S.; Goldman, S.R. Constructing interpretive inferences about literary text: The role of domain-specific knowledge. Learn. Instr. 2019, 60, 245–251. [Google Scholar] [CrossRef]
- Pitts, B.L.; Smith, M.E.; Newberry, K.M.; Bailey, H.R. Semantic knowledge attenuates age-related differences in event segmentation and episodic memory. Mem. Cogn. 2022, 50, 3. [Google Scholar] [CrossRef] [PubMed]
- Singer, M.; Andruslak, P.; Reisdorf, P.; Black, N.L. Individual differences in bridging inference processes. Mem. Cogn. 1992, 20, 539–548. [Google Scholar] [CrossRef] [PubMed]
- Singer, M.; Ritchot, K.F. The role of working memory capacity and knowledge access in text inference processing. Mem. Cogn. 1996, 24, 733–743. [Google Scholar] [CrossRef]
- Whitney, P.; Ritchie, B.G.; Clark, M.B. Working-memory capacity and the use of elaborative inferences in text comprehension. Discourse Process. 1991, 14, 133–145. [Google Scholar] [CrossRef]
- Cohn, N. Visual narrative comprehension: Universal or not? Psychon. Bull. Rev. 2020, 27, 266–285. [Google Scholar] [CrossRef]
- Cohn, N.; Taylor-Weiner, A.; Grossman, S. Framing attention in Japanese and American comics: Cross-cultural differences in attentional structure. Front. Psychol. 2012, 3, 349. [Google Scholar] [CrossRef] [PubMed]
- Cohn, N.; Maher, S. The notion of the motion: The neurocognition of motion lines in visual narratives. Brain Res. 2015, 1601, 73–84. [Google Scholar] [CrossRef] [PubMed]
- Cohn, N. The Patterns of Comics: Visual Languages of Comics from Asia, Europe, and North America; Bloomsbury Publishing: London, UK, 2023. [Google Scholar]
- Nisbett, R.E.; Peng, K.; Choi, I.; Norenzayan, A. Culture and systems of thought: Holistic versus analytic cognition. Psychol. Rev. 2001, 108, 291. [Google Scholar] [CrossRef] [PubMed]
- Nisbett, R.E.; Masuda, T. Culture and point of view. In Biological and Cultural Bases of Human Inference; Psychology Press: London, UK, 2013; pp. 49–70. [Google Scholar]
- Li, L.; Asano, A.; Asano, C.M.; Okajima, K. Statistical quantification of the effects of viewing distance on texture perception. JOSA A 2013, 30, 1394–1403. [Google Scholar] [CrossRef]
- Magliano, J.P.; Millis, K.K. Assessing reading skill with a think-aloud procedure and latent semantic analysis. Cogn. Instr. 2003, 21, 251–283. [Google Scholar] [CrossRef]
Strategy | Description | Example |
---|---|---|
Explanations | Statements that provide reasons why events happened. These could come from prior events or from prior knowledge | The musicians are unhappy because the frog jumped into the saxophone |
He tipped the saxophone upside down because it was plugged | ||
Predictions | Statements that reflect the anticipation of future events | The frog is going to cause some trouble |
Associations | Statements about the setting or the character that are not explicitly conveyed in a picture | This is the woods next to the boy’s house. |
The boy must be about 10 years old. | ||
Picture Paraphrasing | ||
Narrative Descriptions | Statements that specify the actions that are conveyed in a picture | He is looking inside the saxophone The bandmates are annoyed |
Picture Descriptions | Statements that describe objects and their spatial arrangement in a picture | The trumpet player has his finger over his mouth The drum is below the man |
Metacognitive Statements | Statements that reflect participants’ understanding of the story | I do not know what is going on |
Evaluations | Statements of whether the participant does or does not like the content of the story | Very cute story |
Errors | Statements that do not correctly identify the story content. Most of these were misidentifications of characters in the study | The frog is trying to get into the beehive. |
Other statements | Statements that could not be coded as any of the other categories in the study | Yeah, I think that’s it for this picture. |
Bridging Action Presence | ||||
---|---|---|---|---|
Absent | Present | |||
Mean | SD | Mean | SD | |
Explanations | 1.98 | 1.63 | 1.58 | 1.44 |
Predictions | 0.08 | 0.35 | 0.08 | 0.32 |
Associations | 0.10 | 0.40 | 0.11 | 0.43 |
Picture Paraphrasing | 3.26 | 2.25 | 3.33 | 2.05 |
Narrative Descriptions | 2.47 | 1.65 | 2.68 | 1.67 |
Picture Descriptions | 0.79 | 1.35 | 0.65 | 1.10 |
Metacognitive Statements | 0.15 | 0.44 | 0.08 | 0.31 |
Evaluations | 0.03 | 0.23 | 0.03 | 0.23 |
Errors | 0.02 | 0.12 | 0.01 | 0.11 |
Other statements | 0.17 | 0.47 | 0.19 | 0.51 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Smith, M.E.; Hutson, J.P.; Newell, M.; Wing-Paul, D.; McCarthy, K.S.; Loschky, L.C.; Magliano, J.P. Bridging a Gap in Coherence: The Coordination of Comprehension Processes When Viewing Visual Narratives. Vision 2024, 8, 50. https://doi.org/10.3390/vision8030050
Smith ME, Hutson JP, Newell M, Wing-Paul D, McCarthy KS, Loschky LC, Magliano JP. Bridging a Gap in Coherence: The Coordination of Comprehension Processes When Viewing Visual Narratives. Vision. 2024; 8(3):50. https://doi.org/10.3390/vision8030050
Chicago/Turabian StyleSmith, Maverick E., John P. Hutson, Mi’Kayla Newell, Dimitri Wing-Paul, Kathryn S. McCarthy, Lester C. Loschky, and Joseph P. Magliano. 2024. "Bridging a Gap in Coherence: The Coordination of Comprehension Processes When Viewing Visual Narratives" Vision 8, no. 3: 50. https://doi.org/10.3390/vision8030050