Time Scales and Moments of Listening Effort Revealed in Pupillometry

Abstract

This article offers a collection of observations that highlight the value of time course data in pupillometry and points out ways in which these observations create deeper understanding of listening effort. The main message is that listening effort should be considered on a moment-to-moment basis rather than as a singular amount. A review of various studies and the reanalysis of data reveal distinct signatures of effort before a stimulus, during a stimulus, in the moments after a stimulus, and changes over whole experimental testing sessions. Collectively these observations motivate questions that extend beyond the “amount” of effort, toward understanding how long the effort lasts, and how precisely someone can allocate effort at specific points in time or reduce effort at other times. Apparent disagreements between studies are reconsidered as informative lessons about stimulus selection and the nature of pupil dilation as a reflection of decision making rather than the difficulty of sensory encoding.

Keywords

listening effort - pupillometry - speech perception

Note

This paper has been submitted as a preprint on PsyArXiv, available at https://psyarxiv.com/wpquj.

Publication History

Article published online:
04 April 2023

• References

• Ackermann, L., Hepach, R., & Mani, N. (2020). Children learn words easier when they are interested in the category to which the word belongs. Developmental Science, 23(3), e12915
• Alhanbali, S., Dawes, P., Lloyd, S., & Munro, K. J. (2017). Self-reported listening-related effort and fatigue in hearing-impaired adults. Ear and Hearing, 38, e39–e48
• Alhanbali, S., Munro, K. J., Dawes, P., Carolan, P. J., & Millman, R. E. (2021). Dimensions of self-reported listening effort and fatigue on a digits-in-noise task, and association with baseline pupil size and performance accuracy. International Journal of Audiology, 60(10), 762–772
• Aston-Jones, G., & Cohen, J. D. (2005). An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annual Review of Neuroscience, 28, 403–450
• Ayasse, N. D., & Wingfield, A. (2020). Anticipatory baseline pupil diameter is sensitive to differences in hearing thresholds. Frontiers in Psychology, 10, 2947
• Ayasse, N. D., Hodson, A. J., & Wingfield, A. (2021). The principle of least effort and comprehension of spoken sentences by younger and older adults. Frontiers in Psychology, 12, 629464
• Bala, A. D. S., Whitchurch, E. A., & Takahashi, T. T. (2020). Human auditory detection and discrimination measured with the pupil dilation response. Journal of the Association for Research in Otolaryngology : JARO, 21(1), 43–59
• Beatty, J. (1982a). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276–292.
• Beatty, J. (1982b). Phasic not tonic pupillary responses vary with auditory vigilance performance. Psychophysiology, 19(2), 167–172
• Bianchi, F., Wendt, D., Wassard, C., Maas, P., Lunner, T., Rosenbom, T., & Holmberg, M. (2019). Benefit of higher maximum force output on listening effort in bone-anchored hearing system users: a pupillometry study. Ear and Hearing, 40(5), 1220–1232
• Bilger, R. C., Nuetzel, J. M., Rabinowitz, W. M., & Rzeczkowski, C. (1984). Standardization of a test of speech perception in noise. Journal of Speech and Hearing Research, 27(1), 32–48
• Bönitz, H., Lunner, T., Finke, M., Fiedler, L., Lyxell, B., Riis, S. K., Ng, E., Valdes, A. L., Büchner, A., & Wendt, D. (2021). How do we allocate our resources when listening and memorizing speech in noise? A pupillometry study. Ear and Hearing, 42(4), 846–859
• Borghini, G., & Hazan, V. (2020). Effects of acoustic and semantic cues on listening effort during native and non-native speech perception. The Journal of the Acoustical Society of America, 147(6), 3783
• Boswijk, V., Loerts, H., & Hilton, N. (2020). Salience is in the eye of the beholder: increased pupil size reflects acoustically salient variables. Ampersand (Oxford, UK), 7, 100061
• Bradshaw J. (1968) Pupil size and problem solving, Quarterly Journal of Experimental Psychology, 20:2, 116–122, DOI: 10.1080/14640746808400139
• Brown, V. A., McLaughlin, D. J., Strand, J. F., & Van Engen, K. J. (2020). Rapid adaptation to fully intelligible nonnative-accented speech reduces listening effort. Quarterly Journal of Experimental Psychology, 73(9), 1431–1443
• Cavanagh, J. F., Wiecki, T. V., Kochar, A., & Frank, M. J. (2014). Eye tracking and pupillometry are indicators of dissociable latent decision processes. Journal of Experimental Psychology. General, 143(4), 1476–1488
• Colby, S., & McMurray, B. (2021). Cognitive and physiological measures of listening effort during degraded speech perception: relating dual-task and pupillometry paradigms. Journal of Speech, Language, and Hearing Research: JSLHR, 64(9), 3627–3652
• de Gee, J. W., Colizoli, O., Kloosterman, N. A., Knapen, T., Nieuwenhuis, S., & Donner, T. H. (2017). Dynamic modulation of decision biases by brainstem arousal systems. eLife, 6, e23232
• de Gee, J. W., Tsetsos, K., Schwabe, L., Urai, A. E., McCormick, D., McGinley, M. J., & Donner, T. H. (2020). Pupil-linked phasic arousal predicts a reduction of choice bias across species and decision domains. eLife, 9, e54014
• de Gee, J. W., Mridha, Z., Hudson, M., Shi, Y., Ramsaywak, H., Smith, S., Karediya, N., Thompson, M. K., Jaspe, K., Zhang, W., & McGinley, M. (2022). Mice regulate their attentional intensity and arousal to exploit increases in task utility. bioRxiv. Preprint accessed April 9, 2022 at: https://doi.org/10.1101/2022.03.04.482962
• Demberg, V., & Sayeed, A. (2016). The frequency of rapid pupil dilations as a measure of linguistic processing difficulty. PLoS One, 11(1), e0146194
• Eckstein, M. K., Guerra-Carrillo, B., Miller Singley, A. T., & Bunge, S. A. (2017). Beyond eye gaze: What else can eye tracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience, 25, 69–91
• Engelhardt, P. E., Ferreira, F., & Patsenko, E. G. (2010). Pupillometry reveals processing load during spoken language comprehension. Quarterly Journal of Experimental Psychology, 63(4), 639–645
• Francis, A. L., Tigchelaar, L. J., Zhang, R., & Zekveld, A. A. (2018). Effects of second language proficiency and linguistic uncertainty on recognition of speech in native and nonnative competing speech. Journal of Speech, Language, and Hearing Research: JSLHR, 61(7), 1815–1830
• Geacintov, T., & Peavler, W. S. (1974). Pupillography in industrial fatigue assessment. The Journal of Applied Psychology, 59(2), 213–216
• Glennon, E., Carcea, I., Martins, A. R. O., Multani, J., Shehu, I., Svirsky, M. A., & Froemke, R. C. (2019). Locus coeruleus activation accelerates perceptual learning. Brain Research, 1709, 39–49
• Hornsby, B. W., & Kipp, A. M. (2016). Subjective ratings of fatigue and vigor in adults with hearing loss are driven by perceived hearing difficulties not degree of hearing loss. Ear and Hearing, 37(1), e1–e10
• Hornsby BWY, Camarata S, Cho SJ, Davis H, et al. (2021). Development and validation of the Vanderbilt Fatigue Scale for Adults (VFS-A).Psychological Assessment, 33(8), 77788. https://doi.org/10.1037/pas0001021
• Hughes, S. E., Hutchings, H. A., Rapport, F. L., McMahon, C. M., & Boisvert, I. (2018). Social connectedness and perceived listening effort in adult cochlear implant users: a grounded theory to establish content validity for a new patient-reported outcome measure. Ear and Hearing, 39(5), 922–934
• Kadem, M., Herrmann, B., Rodd, J. M., & Johnsrude, I. S. (2020). Pupil dilation is sensitive to semantic ambiguity and acoustic degradation. Trends in Hearing, 24, 2331216520964068
• Kahneman, D. (1973). Attention and Effort. Prentice Hall, Englewood Cliffs, NJ
• Kaandorp, M. W., Smits, C., Merkus, P., Festen, J. M., & Goverts, S. T. (2017). Lexical-access ability and cognitive predictors of speech recognition in noise in adult cochlear implant users. Trends in Hearing, 21, 2331216517743887
• Klingner, J., Tversky, B., & Hanrahan, P. (2011). Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology, 48(3), 323–332
• Koelewijn, T., de Kluiver, H., Shinn-Cunningham, B. G., Zekveld, A. A., & Kramer, S. E. (2015). The pupil response reveals increased listening effort when it is difficult to focus attention. Hearing Research, 323, 81–90
• Koelewijn, T., Zekveld, A. A., Festen, J. M., & Kramer, S. E. (2012). Pupil dilation uncovers extra listening effort in the presence of a single-talker masker. Ear and Hearing, 33(2), 291–300
• Koelewijn, T., Zekveld, A. A., Lunner, T., & Kramer, S. E. (2021). The effect of monetary reward on listening effort and sentence recognition. Hearing Research, 406, 108255
• Książek, P., Zekveld, A. A., Wendt, D., Fiedler, L., Lunner, T., & Kramer, S. E. (2021). Effect of speech-to-noise ratio and luminance on a range of current and potential pupil response measures to assess listening effort. Trends in Hearing, 25, 23312165211009351
• Kucewicz, M. T., Dolezal, J., Kremen, V., Berry, B. M., Miller, L. R., Magee, A. L., Fabian, V., & Worrell, G. A. (2018). Pupil size reflects successful encoding and recall of memory in humans. Scientific Reports, 8(1), 4949
• Kuchinke, L., Võ, M. L., Hofmann, M., & Jacobs, A. M. (2007). Pupillary responses during lexical decisions vary with word frequency but not emotional valence. International Journal of Psychophysiology, 65(2), 132–140
• Kuchinsky, S. E., Ahlstrom, J. B., Vaden, K. I., Jr., Cute, S. L., Humes, L. E., Dubno, J. R., & Eckert, M. A. (2013). Pupil size varies with word listening and response selection difficulty in older adults with hearing loss. Psychophysiology, 50(1), 23–34
• Lemke, U., & Besser, J. (2016). Cognitive load and listening effort: concepts and age-related considerations. Ear and Hearing, 37(Suppl 1), 77S–84S
• Lempert, K. M., Chen, Y. L., & Fleming, S. M. (2015). Relating pupil dilation and metacognitive confidence during auditory decision-making. PLoS One, 10(5), e0126588
• Liao, H. I., Kidani, S., Yoneya, M., Kashino, M., & Furukawa, S. (2016). Correspondences among pupillary dilation response, subjective salience of sounds, and loudness. Psychonomic Bulletin & Review, 23(2), 412–425
• Lim, S. J., Carter, Y. D., Njoroge, J. M., Shinn-Cunningham, B. G., & Perrachione, T. K. (2021). Talker discontinuity disrupts attention to speech: evidence from EEG and pupillometry. Brain and Language, 221, 104996
• Lõo, K., van Rij, J., Järvikivi, J., & Baayen, R. H. (2016). Individual differences in pupil dilation during naming task. In: A. Papafragou, D. Grodner, D. Mirman, J. C. Trueswell (Eds.), Proceedings of the 38th Annual Conference of the Cognitive Science Society (550–555). Austin, TX: Cognitive Science Society
• Lowenstein, O., Feinberg, R., & Loewenfelt, I. (1963). Pupillary movements during acute and chronic fatigue. Investigative Ophthalmology & Visual Science, 2, 138–157.
• Mathôt S, & Vilotijević A. (2022). Methods in cognitive pupillometry: Design, preprocessing, and statistical analysis. Behavior Research Methods, 10.3758/s13428-022-01957-7. Advance online publication
• McCloy, D. R., Larson, E. D., Lau, B., & Lee, A. K. (2016). Temporal alignment of pupillary response with stimulus events via deconvolution. The Journal of the Acoustical Society of America, 139(3), EL57–EL62
• McCloy, D. R., Lau, B. K., Larson, E., Pratt, K. A. I., & Lee, A. K. C. (2017). Pupillometry shows the effort of auditory attention switching. The Journal of the Acoustical Society of America, 141(4), 2440
• McGarrigle R. (2015). Listening-related effort and fatigue in young adults and school-aged children. Doctoral dissertation, University of Manchester. van den Brink, R.L., Murphy, P. R., & Nieuwenhuis, S. (2016). Pupil Diameter Tracks Lapses of Attention. PloS One, 11(10), e0165274. https://doi.org/10.1371/journal.pone.0165274
• McGarrigle, R., Dawes, P., Stewart, A. J., Kuchinsky, S. E., & Munro, K. J. (2017). Pupillometry reveals changes in physiological arousal during a sustained listening task. Psychophysiology, 54, 193–203
• McGarrigle, R., Knight, S., Rakusen, L., Geller, J., & Mattys, S. (2021a). Older adults show a more sustained pattern of effortful listening than young adults. Psychology and Aging, 36(4), 504–519
• McGarrigle, R., Rakusen, L., & Mattys, S. (2021b). Effortful listening under the microscope: examining relations between pupillometric and subjective markers of effort and tiredness from listening. Psychophysiology, 58, e13703
• McGinley, M. J., David, S. V., & McCormick, D. A. (2015). Cortical membrane potential signature of optimal states for sensory signal detection. Neuron, 87(1), 179–192
• McHaney, J. R., Tessmer, R., Roark, C. L., & Chandrasekaran, B. (2021). Working memory relates to individual differences in speech category learning: insights from computational modeling and pupillometry. Brain and Language, 222, 105010
• McLaughlin, D. J., & Van Engen, K. J. (2020). Task-evoked pupil response for accurately recognized accented speech. The Journal of the Acoustical Society of America, 147, EL151–EL156.
• Miles, K., McMahon, C., Boisvert, I., Ibrahim, R., de Lissa, P., Graham, P., & Lyxell, B. (2017). Objective assessment of listening effort: coregistration of pupillometry and EEG. Trends in Hearing, 21, 2331216517706396
• Milne, A., Zhao, S., Tampakaki, C., Bury, G., & Chait, M. (2021). Sustained pupil responses are modulated by predictability of auditory sequences. The Journal of Neuroscience : The Official Journal of the Society for Neuroscience, 41(28), 6116–6127
• DeRoy Milvae, K., Kuchinsky, S. E., Stakhovskaya, O. A., & Goupell, M. J. (2021). Dichotic listening performance and effort as a function of spectral resolution and interaural symmetry. The Journal of the Acoustical Society of America, 150(2), 920
• Mridha, Z., de Gee, J. W., Shi, Y., Alkashgari, R., Williams, J., Suminski, A., Ward, M. P., Zhang, W., & McGinley, M. J. (2021). Graded recruitment of pupil-linked neuromodulation by parametric stimulation of the vagus nerve. Nature Communications, 12(1), 1539
• Papesh, M. H., & Goldinger, S. D. (2012). Pupil-BLAH-metry: cognitive effort in speech planning reflected by pupil dilation. Attention, Perception & Psychophysics, 74(4), 754–765
• Papesh, M. H., Goldinger, S. D., & Hout, M. C. (2012). Memory strength and specificity revealed by pupillometry. International Journal of Psychophysiology, 83(1), 56–64
• Parthasarathy, A., Hancock, K. E., Bennett, K., DeGruttola, V., & Polley, D. B. (2020). Bottom-up and top-down neural signatures of disordered multi-talker speech perception in adults with normal hearing. eLife, 9, e51419
• Peelle, J. E., & Van Engen, K. J. (2021). Time stand still: effects of temporal window selection on eye tracking analysis. Collabra. Psychology, 7, 25961
• Pichora-Fuller, M. K., Kramer, S. E., Eckert, M. A., Edwards, B., Hornsby, B. W., Humes, L. E., Lemke, U., Lunner, T., Matthen, M., Mackersie, C. L., Naylor, G., Phillips, N. A., Richter, M., Rudner, M., Sommers, M. S., Tremblay, K. L., & Wingfield, A. (2016). Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear and Hearing, 37(Suppl 1), 5S–27S
• Piquado, T., Isaacowitz, D., & Wingfield, A. (2010). Pupillometry as a measure of cognitive effort in younger and older adults. Psychophysiology, 47(3), 560–569
• Porretta, V., & Tucker, B. V. (2019). Eyes wide open: pupillary response to a foreign accent varying in intelligibility. Frontiers in Communication, 4, 8
• Reimer, J., McGinley, M. J., Liu, Y., Rodenkirch, C., Wang, Q., McCormick, D. A., & Tolias, A. S. (2016). Pupil fluctuations track rapid changes in adrenergic and cholinergic activity in cortex. Nature Communications, 7, 13289
• Russo, F. Y., Hoen, M., Karoui, C., Demarcy, T., Ardoint, M., Tuset, M. P., De Seta, D., Sterkers, O., Lahlou, G., & Mosnier, I. (2020). Pupillometry assessment of speech recognition and listening experience in adult cochlear implant patients. Frontiers in Neuroscience, 14, 556675
• Satterthwaite, T. D., Green, L., Myerson, J., Parker, J., Ramaratnam, M., & Buckner, R. L. (2007). Dissociable but inter-related systems of cognitive control and reward during decision making: evidence from pupillometry and event-related fMRI. NeuroImage, 37(3), 1017–1031
• Scheepers, C., Mohr, S., Fischer, M. H., & Roberts, A. M. (2013). Listening to Limericks: a pupillometry investigation of perceivers' expectancy. PLoS One, 8(9), e74986
• Schmidtke, J. (2014). Second language experience modulates word retrieval effort in bilinguals: evidence from pupillometry. Frontiers in Psychology, 5, 137
• Schmidtke, J. (2017). Pupillometry in linguistic research: an introduction and review for second language researchers. Studies in Second Language Acquisition, 40(3), 529–549
• Seifi Ala, T., Graversen, C., Wendt, D., Alickovic, E., Whitmer, W. M., & Lunner, T. (2020). An exploratory study of EEG alpha oscillation and pupil dilation in hearing-aid users during effortful listening to continuous speech. PLoS One, 15(7), e0235782
• Smith ML, Winn MB. (2022). The difference between using context to predict versus using context to repair: a study of listening effort. Presentation at the 182^nd meeting of the Acoustical Society of America, Denver, CO.
• Steinhauer, S. R., Bradley, M. M., Siegle, G. J., Roecklein, K. A., & Dix, A. (2022). Publication guidelines and recommendations for pupillary measurement in psychophysiological studies. Psychophysiology, 59(4), e14035
• Strand, J. F., & Brown, V. A. (2022, May 12). Spread the word: enhancing replicability of speech research through stimulus sharing. Accessed March 10, 2023 at: https://doi.org/10.31234/osf.io/amevw
• Tamási, K., McKean, C., Gafos, A., Fritzsche, T., & Höhle, B. (2017). Pupillometry registers toddlers' sensitivity to degrees of mispronunciation. Journal of Experimental Child Psychology, 153, 140–148
• van Rij, J. (2012). Pronoun Processing: Computational, Behavioral, and Psychophysiological Studies in Children and Adults (Doctoral Thesis). University of Groningen, Germany
• van Rij, J., Hendriks, P., van Rijn, H., Baayen, R. H., & Wood, S. N. (2019). Analyzing the time course of pupillometric data. Trends in Hearing, 23, 2331216519832483
• Wagner, A. E., Nagels, L., Toffanin, P., Opie, J. M., & Başkent, D. (2019). Individual variations in effort: assessing pupillometry for the hearing impaired. Trends in Hearing, 23, 2331216519845596
• Wang, Y., Naylor, G., Kramer, S. E., Zekveld, A. A., Wendt, D., Ohlenforst, B., & Lunner, T. (2018). Relations between self-reported daily-life fatigue, hearing status, and pupil dilation during a speech perception in noise task. Ear and Hearing, 39(3), 573–582
• Wierda, S. M., van Rijn, H., Taatgen, N. A., & Martens, S. (2012). Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution. Proc. Natl. Acad. Sci. USA, 109(22), 8456–8460
• Wendt, D., Dau, T., & Hjortkjær, J. (2016). Impact of background noise and sentence complexity on processing demands during sentence comprehension. Frontiers in Psychology, 7, 345
• Wendt, D., Koelewijn, T., Książek, P., Kramer, S. E., & Lunner, T. (2018). Toward a more comprehensive understanding of the impact of masker type and signal-to-noise ratio on the pupillary response while performing a speech-in-noise test. Hearing Research, 369, 67–78
• Wetzel N, Einhauser W, Widmann A. (2020). Picture evoked changes in pupil size predict learning success in children. Journal of Experimental Child Psychology, 192, 104787. https://doi.org/10.1016/j.jecp.2019.104787
• Widmann, A., Schröger, E., & Wetzel, N. (2018). Emotion lies in the eye of the listener: Emotional arousal to novel sounds is reflected in the sympathetic contribution to the pupil dilation response and the P3. Biological Psychology, 133, 10–17
• Winn, M. B. (2016). Rapid release from listening effort resulting from semantic context, and effects of spectral degradation and cochlear implants. Trends in Hearing, 20, 1–17
• Winn, M. B., Edwards, J. R., & Litovsky, R. Y. (2015). The impact of auditory spectral resolution on listening effort revealed by pupil dilation. Ear and Hearing, 36, e153–e165
• Winn, M. B., & Moore, A. N. (2018). Pupillometry reveals that context benefit in speech perception can be disrupted by later-occurring sounds, especially in listeners with cochlear implants. Trends in Hearing, 22, 2331216518808962
• Winn, M. B., & Teece, K. H. (2021a). Listening effort is not the same as speech intelligibility score. Trends in Hearing, 25, 23312165211027688
• Winn, M. B., & Teece, K. H. (2021b). Slower speaking rate reduces listening effort among listeners with cochlear implants. Ear and Hearing, 42(3), 584–595
• Winn, MB, Teece, KH. (2022). Effortful listening despite correct responses: the cost of mental repair in sentence recognition by listeners with cochlear implants. Journal of Speech, Language, and Hearing Research, 65(10), 3966–3980, DOI: 10.1044/2022_JSLHR-21-00631
• Winn, M. B., Wendt, D., Koelewijn, T., & Kuchinsky, S. E. (2018). Best practices and advice for using pupillometry to measure listening effort: an introduction for those who want to get started. Trends in Hearing, 22, 2331216518800869
• Zekveld, A. A., Festen, J. M., & Kramer, S. E. (2013). Task difficulty differentially affects two measures of processing load: the pupil response during sentence processing and delayed cued recall of the sentences. Journal of Speech, Language, and Hearing Research: JSLHR, 56(4), 1156–1165
• Zekveld, A. A., Heslenfeld, D. J., Johnsrude, I. S., Versfeld, N. J., & Kramer, S. E. (2014). The eye as a window to the listening brain: neural correlates of pupil size as a measure of cognitive listening load. NeuroImage, 101, 76–86
• Zekveld, A. A., Koelewijn, T., & Kramer, S. E. (2018). The pupil dilation response to auditory stimuli: current state of knowledge. Trends in Hearing, 22, 2331216518777174
• Zekveld, A. A., Kramer, S. E., & Festen, J. M. (2010). Pupil response as an indication of effortful listening: the influence of sentence intelligibility. Ear and Hearing, 31(4), 480–490
• Zekveld, A. A., Kramer, S. E., & Festen, J. M. (2011). Cognitive load during speech perception in noise: the influence of age, hearing loss, and cognition on the pupil response. Ear and Hearing, 32(4), 498–510
• Zhang, Y., Lehmann, A., & Deroche, M. (2021). Disentangling listening effort and memory load beyond behavioural evidence: pupillary response to listening effort during a concurrent memory task. PLoS One, 16(3), e0233251
• Zhao, S., Bury, G., Milne, A., & Chait, M. (2019). Pupillometry as an objective measure of sustained attention in young and older listeners. Trends in Hearing, 23, 2331216519887815