Chapter 3 (Study 2). Language and vision in conceptual processing: Multilevel analysis and statistical power
Research has suggested that conceptual processing depends on both language-based and vision-based information. We tested this interplay at three levels of the experimental structure: individuals, words and tasks. To this end, we drew on three existing, large data sets that implemented the paradigms of semantic priming, semantic decision and lexical decision. We extended these data sets with measures of language-based and vision-based information, and analysed how the latter variables interacted with participants’ vocabulary size and gender, and also with presentation speed in the semantic priming study. We performed the analysis using mixed-effects models that included a comprehensive array of fixed effects—including covariates—and random effects. First, we found that language-based information was more important than vision-based information. Second, in the semantic priming study—whose task required distinguishing between words and nonwords—, both language-based and vision-based information were more influential when words were presented faster. Third, a ‘task-relevance advantage’ was identified in higher-vocabulary participants. Specifically, in lexical decision, higher-vocabulary participants were more sensitive to language-based information than lower-vocabulary participants. In contrast, in semantic decision, higher-vocabulary participants were more sensitive to word concreteness. Fourth, we demonstrated the influence of the analytical method on the results. These findings support the interplay between language and vision in conceptual processing, and demonstrate the influence of measurement instruments on the results. Last, we estimated the sample size required to reliably investigate various effects. We found that 300 participants were sufficient to examine the effect of language-based information contained in words, whereas more than 1,000 participants were necessary to examine the effect of vision-based information and the interactions of both former variables with vocabulary size, gender and presentation speed. In conclusion, this power analysis suggests that larger sample sizes are necessary to investigate perceptual simulation and individual differences in conceptual processing.
Over the last two decades, research in the cognitive sciences has suggested that conceptual processing depends on both language and embodiment systems. That is, understanding words involves—on the one hand—lexical and semantic associations of an amodal kind, and—on the other hand—modality-specific associations within perceptual, motor, affective and social domains (Barsalou et al., 2008; Connell, 2019; Davis & Yee, 2021; Khatin-Zadeh et al., 2021; Vigliocco et al., 2009). Studies addressing these systems have found that the language system is overall more prevalent in word processing, producing larger effect sizes (Banks et al., 2021; Kiela & Bottou, 2014; Lam et al., 2015; Louwerse et al., 2015; Pecher et al., 1998; Petilli et al., 2021). More intricately, the roles of language and embodiment are modulated by the characteristics of individuals, words and tasks. For instance, people’s individual experience with language is associated with differential effects relating to phonological, lexical and semantic features of words (Jared & O’Donnell, 2017; Pexman & Yap, 2018; Yap et al., 2009, 2012, 2017). Similarly, physical expertise and perceptual biases are associated with differences in the mental simulation of meaning (Beilock et al., 2008; Calvo-Merino et al., 2005; Vukovic & Williams, 2015). Furthermore, the embodiment system is especially suited for the processing of concrete concepts—e.g., red, building (C. R. Jones et al., 2022; Kousta et al., 2011; Ponari, Norbury, Rotaru, et al., 2018; cf. Borghi et al., 2022). Embodied information also becomes more important in the following conditions: (I) later in the time courses of word recognition (Bernabeu et al., 2017; Louwerse & Hutchinson, 2012; cf. Petilli et al., 2021) and property generation (Santos et al., 2011; Simmons et al., 2008), (II) when participants produce slower responses (Louwerse & Connell, 2011), and (III) in tasks that elicit deeper semantic processing—e.g., semantic decision, as opposed to lexical decision (Ostarek & Huettig, 2017; Petilli et al., 2021). Last, research in computational linguistics has provided further support for the complementarity of language and embodied information, by revealing increased predictive performance when models are provided with perceptual information on top of text-based information (Frome et al., 2013; Roads & Love, 2020).
In spite of the amount of evidence demonstrating the interplay between language and embodiment, there are four reasons to continue testing the interplay theory. First, the coexistence of several systems in a scientific theory must be thoroughly justified due to the value of simplicity (Gallese & Lakoff, 2005; Tillman et al., 2015). This scrutiny is particularly necessary because the language system has consistently produced larger effect sizes than the embodiment system (Banks et al., 2021; Kiela & Bottou, 2014; Lam et al., 2015; Louwerse et al., 2015; Pecher et al., 1998; Petilli et al., 2021). Therefore, it should be ruled out that the language system can suffice in all contexts.
Second, it is important to examine both language and embodiment across various levels of the experimental structure—namely, individuals (i.e., due to individual differences such as vocabulary size), words (i.e., lexical and semantic variables) and tasks (i.e., experimental conditions affecting, for instance, processing speed). Some studies have approached this comprehensive structure but there is still room to widen the scope. One of the findings revealed by cross-level analyses is the influence of word processing tasks on the importance of modality-specific information. For instance, Connell and Lynott (2014a) found that the vision-based information in words is important both for word identification (i.e., lexical decision) and for reading aloud (i.e., naming). In contrast, the auditory information in words is important for reading aloud but not so much for word identification. Another finding from cross-level research is a ‘task-relevance advantage’ for individuals that have a greater linguistic experience. Specifically, Pexman and Yap (2018) found that higher-vocabulary individuals were more sensitive to task-relevant information, such as word concreteness in the semantic decision task. Furthermore, regarding embodiment, research has revealed that individuals who are briefly exposed to a certain sport develop neural activity that allows them to mentally simulate sport-specific actions during language processing (Beilock et al., 2008). While these works have covered a large swathe of the present topic, one question remains unanswered: how does an individual’s linguistic experience relate to their sensivity to both linguistic and embodied information in words?
Third, there is some inconclusive evidence. First, some findings have suggested that higher-vocabulary participants are more sensitive to language-based information—as reflected in greater semantic priming (Yap et al., 2017)—, whereas other findings have suggested the opposite (Yap et al., 2009). Second, some studies have suggested that the language system is activated before the embodiment system (Lam et al., 2015; Louwerse & Connell, 2011), whereas a recent study suggested that this pattern does not hold in the lexical decision task (Petilli et al., 2021). Third, some evidence has suggested that female participants draw on the language system more prominently than males (Burman et al., 2008; Hutchinson & Louwerse, 2013; Jung et al., 2019; Ullman et al., 2008), whereas other research has suggested that this difference is negligible in the general population (Wallentin, 2020).
Fourth, some of the previous studies could have been affected by the scarcity of statistical power that has been identified in cognitive psychology and neuroscience (Lynott et al., 2014; Marek et al., 2022; Montero-Melis et al., 2022). Problematically, low-powered studies present more errors in the estimation of effect sizes and \(p\) values (Heyman et al., 2018; Loken & Gelman, 2017; Vasishth, Mertzen, et al., 2018). The current studies address these four key issues.
Banks, B., Wingfield, C., & Connell, L. (2021). Linguistic distributional knowledge and sensorimotor grounding both contribute to semantic category production.
Cognitive Science,
45(10), e13055.
https://doi.org/10.1111/cogs.13055
Barsalou, L. W., Santos, A., Simmons, W. K., & Wilson, C. D. (2008). Language and simulation in conceptual processing. In
Symbols and Embodiment.
Oxford University Press.
https://doi.org/10.1093/acprof:oso/9780199217274.003.0013
Beilock, S. L., Lyons, I. M., Mattarella-Micke, A., Nusbaum, H. C., & Small, S. L. (2008). Sports experience changes the neural processing of action language.
Proceedings of the National Academy of Sciences,
105(36), 13269–13273.
https://doi.org/10.1073/pnas.0803424105
Bernabeu, P., Willems, R. M., & Louwerse, M. M. (2017). Modality switch effects emerge early and increase throughout conceptual processing:
Evidence from
ERPs. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. J. Davelaar (Eds.),
Proceedings of the 39th Annual Conference of the Cognitive Science Society (pp. 1629–1634).
Cognitive Science Society.
https://doi.org/10.31234/osf.io/a5pcz
Borghi, A. M., Shaki, S., & Fischer, M. H. (2022). Abstract concepts: External influences, internal constraints, and methodological issues.
Psychological Research.
https://doi.org/10.1007/s00426-022-01698-4
Burman, D., Bitan, T., & Both, J. (2008). Sex differences in neural processing of language among children.
Neuropsychologia,
46, 5, 1349–1362.
https://doi.org/10.1016/j.neuropsychologia.2007.12.021
Calvo-Merino, B., Glaser, D. E., Grèzes, J., Passingham, R. E., & Haggard, P. (2005). Action observation and acquired motor skills:
An FMRI study with expert dancers.
Cerebral Cortex,
15(8), 1243–1249.
https://doi.org/10.1093/cercor/bhi007
Connell, L. (2019). What have labels ever done for us?
The linguistic shortcut in conceptual processing.
Language, Cognition and Neuroscience,
34(10), 1308–1318.
https://doi.org/10.1080/23273798.2018.1471512
Connell, L., & Lynott, D. (2014a). I see/hear what you mean:
Semantic activation in visual word recognition depends on perceptual attention.
Journal of Experimental Psychology: General,
143(2), 527–533.
https://doi.org/10.1037/a0034626
Davis, C. P., & Yee, E. (2021). Building semantic memory from embodied and distributional language experience.
WIREs Cognitive Science,
12(5), e1555.
https://doi.org/10.1002/wcs.1555
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., & Mikolov, T. (2013).
DeViSE:
A deep visual-semantic embedding model. In C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.),
Advances in neural information processing systems (Vol. 26).
Curran Associates, Inc. https://proceedings.neurips.cc/paper/2013/file/7cce53cf90577442771720a370c3c723-Paper.pdf
Gallese, V., & Lakoff, G. (2005). The
Brain’s concepts: The role of the
Sensory-motor system in conceptual knowledge.
Cognitive Neuropsychology,
22(3-4), 455–479.
https://doi.org/10.1080/02643290442000310
Heyman, T., Bruninx, A., Hutchison, K. A., & Storms, G. (2018). The (un)reliability of item-level semantic priming effects.
Behavior Research Methods,
50(6), 2173–2183.
https://doi.org/10.3758/s13428-018-1040-9
Hutchinson, S., & Louwerse, M. M. (2013). Language statistics and individual differences in processing primary metaphors.
Cognitive Linguistics,
24(4), 667–687.
https://doi.org/10.1515/cog-2013-0023
Jared, D., & O’Donnell, K. (2017). Skilled adult readers activate the meanings of high-frequency words using phonology:
Evidence from eye tracking.
Memory & Cognition,
45(2), 334–346.
https://doi.org/10.3758/s13421-016-0661-4
Jones, C. R., Chang, T. A., Coulson, S., Michaelov, J. A., Trott, S., & Bergen, B. (2022). Distrubutional semantics still can’t account for affordances.
Proceedings of the Annual Meeting of the Cognitive Science Society,
44(44).
https://escholarship.org/uc/item/44z7r3j3
Jung, M., Mody, M., Fujioka, T., Kimura, Y., Okazawa, H., & Kosaka, H. (2019). Sex differences in white matter pathways related to language ability.
Frontiers in Human Neuroscience,
13, 898.
https://doi.org/10.3389/fnins.2019.00898
Khatin-Zadeh, O., Eskandari, Z., Cervera-Torres, S., Ruiz Fernández, S., Farzi, R., & Marmolejo-Ramos, F. (2021). The strong versions of embodied cognition:
Three challenges faced.
Psychology & Neuroscience,
14(1), 16–33.
https://doi.org/10.1037/pne0000252
Kiela, D., & Bottou, L. (2014). Learning image embeddings using convolutional neural networks for improved multi-modal semantics.
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP, 36–45.
https://doi.org/10.3115/v1/D14-1005
Kousta, S.-T., Vigliocco, G., Vinson, D. P., Andrews, M., & Del Campo, E. (2011). The representation of abstract words:
Why emotion matters.
Journal of Experimental Psychology: General,
140, 14–34.
https://doi.org/10.1037/a0021446
Lam, K. J., Dijkstra, T., & Rueschemeyer, S. A. (2015). Feature activation during word recognition: Action, visual, and associative-semantic priming effects.
Frontiers in Psychology,
6, 659.
https://doi.org/10.3389/fpsyg.2015.00659
Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis.
Science,
355(6325), 584–585.
https://doi.org/10.1126/science.aal3618
Louwerse, M. M., & Connell, L. (2011). A taste of words: Linguistic context and perceptual simulation predict the modality of words.
Cognitive Science,
35(2), 381–398.
https://doi.org/10.1111/j.1551-6709.2010.01157.x
Louwerse, M. M., & Hutchinson, S. (2012). Neurological evidence linguistic processes precede perceptual simulation in conceptual processing.
Frontiers in Psychology,
3, 385.
https://doi.org/10.3389/fpsyg.2012.00385
Louwerse, M. M., Hutchinson, S., Tillman, R., & Recchia, G. (2015). Effect size matters:
The role of language statistics and perceptual simulation in conceptual processing.
Language, Cognition and Neuroscience,
30(4), 430–447.
https://doi.org/10.1080/23273798.2014.981552
Lynott, D., Corker, K. S., Wortman, J., Connell, L., Donnellan, M. B., Lucas, R. E., & O’Brien, K. (2014). Replication of
“Experiencing physical warmth promotes interpersonal warmth” by
Williams and
Bargh (2008).
Social Psychology,
45(3), 216–222.
https://doi.org/10.1027/1864-9335/a000187
Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Hendrickson, T. J., Malone, S. M., Kandala, S., Feczko, E., Miranda-Dominguez, O., Graham, A. M., Earl, E. A., Perrone, A. J., Cordova, M., Doyle, O., … Dosenbach, N. U. F. (2022). Reproducible brain-wide association studies require thousands of individuals.
Nature, 1–7.
https://doi.org/10.1038/s41586-022-04492-9
Montero-Melis, G., van Paridon, J., Ostarek, M., & Bylund, E. (2022). No evidence for embodiment:
The motor system is not needed to keep action verbs in working memory.
Cortex,
150, 108–125.
https://doi.org/10.1016/j.cortex.2022.02.006
Ostarek, M., & Huettig, F. (2017). A task-dependent causal role for low-level visual processes in spoken word comprehension.
Journal of Experimental Psychology: Learning, Memory, and Cognition,
43(8), 1215–1224.
https://doi.org/10.1037/xlm0000375
Pecher, D., Zeelenberg, R., & Raaijmakers, J. G. W. (1998). Does pizza prime coin?
Perceptual priming in lexical decision and pronunciation.
Journal of Memory and Language,
38(4), 401–418.
https://doi.org/10.1006/jmla.1997.2557
Petilli, M. A., Günther, F., Vergallito, A., Ciapparelli, M., & Marelli, M. (2021). Data-driven computational models reveal perceptual simulation in word processing.
Journal of Memory and Language,
117, 104194.
https://doi.org/10.1016/j.jml.2020.104194
Pexman, P. M., & Yap, M. J. (2018). Individual differences in semantic processing:
Insights from the
Calgary semantic decision project.
Journal of Experimental Psychology: Learning, Memory, and Cognition,
44(7), 1091–1112.
https://doi.org/10.1037/xlm0000499
Ponari, M., Norbury, C. F., Rotaru, A., Lenci, A., & Vigliocco, G. (2018). Learning abstract words and concepts:
Insights from developmental language disorder.
Philosophical Transactions of the Royal Society B: Biological Sciences,
373, 20170140.
https://doi.org/10.1098/rstb.2017.0140
Roads, B. D., & Love, B. C. (2020). Learning as the unsupervised alignment of conceptual systems.
Nature Machine Intelligence,
2(1, 1), 76–82.
https://doi.org/10.1038/s42256-019-0132-2
Santos, A., Chaigneau, S. E., Simmons, W. K., & Barsalou, L. W. (2011). Property generation reflects word association and situated simulation.
Language and Cognition,
3(1), 83–119.
https://doi.org/10.1515/langcog.2011.004
Simmons, W. K., Hamann, S. B., Harenski, C. L., Hu, X. P., & Barsalou, L. W. (2008).
fMRI evidence for word association and situated simulation in conceptual processing.
Journal of Physiology-Paris,
102(1), 106–119.
https://doi.org/10.1016/j.jphysparis.2008.03.014
Tillman, R., Hutchinson, S., & Louwerse, M. M. (2015). How sharp is
Occam’s razor?
Language statistics in cognitive processing. In D. C. Noelle, R. Dale, A. S. Warlaumont, J. Yoshimi, T. Matlock, C. D. Jennings, & P. P. Maglio (Eds.),
Proceedings of the 37th Annual Conference of the Cognitive Science Society (pp. 2404–2409).
Cognitive Science Society.
https://cogsci.mindmodeling.org/2015/papers/0413/paper0413.pdf
Ullman, M. T., Miranda, R. A., & Travers, M. L. (2008). Sex differences in the neurocognition of language. In J. B. Becker, K. J. Berkley, N. Geary, E. Hampson, J. Herman, & E. Young (Eds.), Sex on the brain: From genes to behavior (pp. 291–309). Oxford University Press.
Vasishth, S., Mertzen, D., Jäger, L. A., & Gelman, A. (2018). The statistical significance filter leads to overoptimistic expectations of replicability.
Journal of Memory and Language,
103, 151–175.
https://doi.org/10.1016/j.jml.2018.07.004
Vigliocco, G., Meteyard, L., Andrews, M., & Kousta, S. (2009).
Toward a theory of semantic representation.
1(2), 219–247.
https://doi.org/10.1515/LANGCOG.2009.011
Vukovic, N., & Williams, J. N. (2015). Individual differences in spatial cognition influence mental simulation of language.
Cognition,
142, 110–122.
https://doi.org/10.1016/j.cognition.2015.05.017
Wallentin, M. (2020). Chapter 6 -
Gender differences in language are small but matter for disorders. In R. Lanzenberger, G. S. Kranz, & I. Savic (Eds.),
Handbook of Clinical Neurology (Vol. 175, pp. 81–102).
Elsevier.
https://doi.org/10.1016/B978-0-444-64123-6.00007-2
Yap, M. J., Balota, D. A., Sibley, D. E., & Ratcliff, R. (2012). Individual differences in visual word recognition:
Insights from the
English Lexicon Project.
Journal of Experimental Psychology: Human Perception and Performance,
38, 1, 53–79.
https://doi.org/10.1037/a0024177
Yap, M. J., Hutchison, K. A., & Tan, L. C. (2017). Individual differences in semantic priming performance: Insights from the semantic priming project. In M. N. Jones (Ed.), Frontiers of cognitive psychology. Big data in cognitive science (pp. 203–226). Routledge/Taylor & Francis Group.
Yap, M. J., Tse, C.-S., & Balota, D. A. (2009). Individual differences in the joint effects of semantic priming and word frequency revealed by
RT distributional analyses:
The role of lexical integrity.
Journal of Memory and Language,
61(3), 303–325.
https://doi.org/10.1016/j.jml.2009.07.001