The study of human-human communication and the development of computational models for human-agent communication have diverged significantly throughout the last decade. Yet, despite frequently made claims of “super-human performance” in, e.g., speech recognition or image processing, so far, no system is able to lead a half-decent coherent conversation with a human. In this paper, we argue that we must start to re-consider the hallmarks of cooperative communication and the core capabilities that we have developed for it, and which conversational agents (...) need to be equipped with: incremental joint co-construction and mentalizing. We base our argument on a vast body of work on human-human communication and its psychological processes that we reason to be relevant and necessary to take into account when modeling human-agent communication. We contrast those with current conceptualizations of human-agent interaction and formulate suggestions for the development of future systems. (shrink)

When young children learn to use language, they start to use their hands in co‐verbal gesturing. There are, however, considerable differences between children, and it is not completely understood what these individual differences are due to. We studied how children at 4 years of age employ speech and iconic gestures to convey meaning in different kinds of spatial event descriptions, and how this relates to their cognitive abilities. Focusing on spontaneous illustrations of actions, we applied a semantic feature (SF) analysis (...) to characterize combinations of speech and gesture meaning and related them to the child's visual‐spatial abilities or abstract/concrete reasoning abilities (measured using the standardized SON‐R test). Results show that children with higher cognitive abilities convey significantly more meaning via gesture and less solely via speech. These findings suggest that young children's use of cospeech representational gesturing is positively related to their mental representation and reasoning abilities. (shrink)

The gesture-as-simulated-action framework explains the occurrence of iconic gestures. Accordingly, simulated visual imagery gives rise to observer-viewpoint, whereas simulated motor imagery gives rise to character-viewpoint gestures. Because little is known about whether this relationship is either the product of becoming a competent speaker in different communicative tasks or exists from an early age, we investigated 4-year-olds. In the first session, 55 children performed three different communicative tasks. In the second session, we administered a SON-R non-verbal intelligence test to assess children’s (...) cognitive abilities. Results revealed a strong association between the frequency with which children used observer-viewpoints gestures and their scores in visual processing abilities. Because the association between gestures and visuospatial cognitive abilities was found across three communicative tasks, we can propose that it is a general ability rather than a phenomenon of task-specific discourse competence. We discuss these findings in terms of gestural threshold and cognitive offloading. (shrink)