This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how (...) the multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

Musical performance is a multimodal experience, for performers and listeners alike. This paper reports on a pilot study which constitutes the first step toward a comprehensive approach to the experience of music as performed. We aim at bridging the gap between qualitative and quantitative approaches, by combining methods for data collection. The purpose is to build a data corpus containing multimodal measures linked to high-level subjective observations. This will allow for a systematic inclusion of the knowledge of music (...) professionals in an analytic framework, which synthesizes methods across established research disciplines. We outline the methods we are currently developing for the creation of a multimodal data corpus dedicated to the analysis and exploration of instrumental music performance from the perspective of embodied music cognition. This will enable the study of the multiple facets of instrumental music performance in great detail, as well as lead to the development of music creation techniques that take advantage of the cross-modal relationships and higher-level qualities emerging from the analysis of this multi-layered, multimodal corpus. The results of the pilot project suggest that qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Furthermore, the results indicate several directions for further development, regarding observational movement analysis, and computational analysis of coarticulation, chunking, and movement qualities in musical performance. We argue that the development of methods for combining qualitative and quantitative data are required to fully understand expressive musical performance, especially in a broader scenario in which arts, humanities, and science are increasingly entangled. The future work in the project will therefore entail an increasingly multimodal analysis, aiming to become as holistic as is music in performance. (shrink)

Taste is a central sense for humans and animals, and it has been largely studied either from physiological and neurological approaches or from socio-cultural ones. This paper adopts another view, focused on the activity of tasting rather than on the sense of taste, approached within the perspective of ethnomethodology and multimodal conversation analysis. This view addresses the activity of tasting as it is interactionally organized in specific social settings, observed in a naturalistic way, on the basis of video recordings. (...) Focusing on video recorded improvised tastings of cheese in gourmet shop encounters, the paper offers a systematic analysis of the way in which tasting is orderly achieved in an intersubjective way. It follows the various steps characterizing tasting, from the invitation to taste, to the grasping of a bit to taste, which is put in the mouth, chewed, and swallowed; it details how an interactional moment offering the taster a priviledged, individual, focused space in which to devote exclusive attention to the object tasted is actively tailored by all parties. By contrast, the completion of tasting is marked by a return to mutual gaze, the animation of facial expressions and nods, and the final production of a judgment of taste. By offering a systematic reconstruction of how these tasting moments are organized, the paper invites to a multimodal approach of sensoriality in social interaction. (shrink)

Recently, it has been observed that the usual type-theoretic restrictions are not enough to block certain paradoxes involving two or more predicates. In particular, when we have a self-referential language containing modal predicates, new paradoxes might appear even if there are type restrictions for the principles governing those predicates. In this article we consider two type-theoretic solutions to multimodal paradoxes. The first one adds types for each of the modal predicates. We argue that there are a number of problems (...) with most versions of this approach. The second one, which we favour, represents modal notions by using the truth predicate together with the corresponding modal operator. This way of doing things is not only useful because it avoids multimodal paradoxes, but also because it preserves the expressive capacity of the language. As an example of the sort of theory we have in mind, we provide a type-theoretic axiomatization that combines truth with necessity and knowledge. (shrink)

In the process of communication between people, everyone will have emotions, and different emotions will have different effects on communication. With the help of external performance information accompanied by emotional expression, such as emotional speech signals or facial expressions, people can easily communicate with each other and understand each other. Emotion recognition is an important network of affective computers and research centers for signal processing, pattern detection, artificial intelligence, and human-computer interaction. Emotions convey important information in human communication and (...) communication. Since the end of the last century, people have started the research on emotion recognition, especially how to correctly judge the emotion type has invested a lot of time and energy. In this paper, multi-modal emotion recognition is introduced to recognize facial expressions and speech, and conduct research on adaptive higher education management. Language and expression are the most direct ways for people to express their emotions. After obtaining the framework of the dual-modal emotion recognition system, the BOW model is used to identify the characteristic movement of local areas or key points. The recognition rates of emotion recognition for 1,000 audios of anger, disgust, fear, happiness, sadness and surprise are: 97.3, 83.75, 64.87, 89.87, 84.12, and 86.68%, respectively. (shrink)

This article starts with the environmental changes in human cognition, analyzes the virtual as the main feature of visual perception under digital technology, and explores the transition from passive to active human cognitive activities. With the diversified understanding of visual information, human contradiction of memory also began to become prominent. Aiming at the problem that the existing multimodal TV media recognition methods have low recognition rate of unknown application layer protocols, an adaptive clustering method for identifying unknown application layer (...) protocols is proposed. This method clusters application layer protocols based on similarity of the load characteristics of network stream application layer protocol data. The method divides the similarity calculation in the clustering algorithm to improve the clustering efficiency of the algorithm. Experimental results show that the proposed method can efficiently and accurately recognize unknown visual communication. This article proposes that, in the interactive multimodal visual information transmission, human visual perception experience has changed, the diversity of visual information content expression makes the aesthetic subject more personalized and stylized. (shrink)

The question, “What norms are appropriate for the evaluation of the probative merits of visual arguments?” underlies the investigation of this paper. The notions of argument and of multimodal visual argument employed in the study are explained. Then four multimodal visual arguments are analyzed and their probative merits assessed. It turns out to be possible to judge these qualities using the same criteria that apply to verbally expressed arguments. Since the sample is small and not claimed to be (...) representative, this finding can at best be regarded as suggestive for the probative assessment of multimodal visual arguments in general. (shrink)

In academic oral genres such as conference presentations, speakers resort to more than words to convey meaning. Research also suggests that persuasion, an important element of the communicative purpose of conference presentations, is frequently achieved through a combination of semiotic modes. Therefore, a skilful orchestration of these modes can be considered key to achieving effective communication in this genre. However, our understanding of persuasion has often focused on specific elements of the message considered in isolation and mainly from the linguistic (...) perspective. Relatively little attention has been paid to the overall persuasive effect achieved by the complex multimodal ensemble. This study approaches the analysis of persuasive strategies in conference presentations combining multimodal discourse analysis and ethnographic methods. It focuses on a particular attention-getting technique: enactment of characters, or acting the part of a person that is being referred to. Our analysis shows how it is achieved through the orchestration of different modes such as words, intonation, gestures, head movements, gaze and facial expression. (shrink)

In this paper I contend that abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc. but also kinesthetic experiences and other feelings such as pain, and thus all sensory modalities. The kinesthetic aspects simply explain abductive reasoning is basically manipulative, both linguistic and non linguistic signs have an internal semiotic life, as particular configurations of neural networks and chemical distributions (and in terms (...) of their transformations) at the level of human brains, and as somatic expressions, but can also be delegated to many external objects and devices, for example written texts, diagrams, artifacts, etc. Still in this case unconscious aspects takepart of the abductive procedure, which acquires the features of a kind of "thinking through doing". (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

This study explores multimodal metaphors and metonymies in Faust, a German Expressionist silent fiction movie by Murnau. The article combines principles of psychocinematics, an interdisciplinary scientific field of enquiry, with the multimodal metaphor and expressive movement model, which looks into the temporal dynamics of metaphoric meaning-making by movie watchers. It is shown that interrelating both film-analytic approaches provides a deeper and more comprehensive insight into how figurative thought influences psycho-cognitive processes in the moviegoer’s mind as they dynamically unfold (...) in their cinematic contexts. Evidence is provided that Kappelhoff and co-workers’ proposal can be complemented and enriched by psychocinematics findings on the allocation of visual attention and eye fixation, motion and image frame sensing, and mental activity underlying emotional and diagetic experiences. (shrink)

According to the shared signal hypothesis the impact of facial expressions on emotion processing partially depends on whether the gaze is directed toward or away from the observer. In autism spectrum disorder several aspects of face processing have been found to be atypical, including attention to eye gaze and the identification of emotional expressions. However, there is little research on how gaze direction affects emotional expression processing in typically developing individuals and in those with ASD. This question is investigated (...) here in two multimodal experiments. Experiment 1 required processing eye gaze direction while faces differed in emotional expression. Forty-seven children participated. Their Autism Diagnostic Observation Schedule scores ranged from 0 to 6 in the experiment. Event-related potentials were sensitive to gaze direction and emotion, but emotion processing did not depend on gaze direction. However, for angry faces the gaze direction effect on the N170 amplitude, as typically observed in TD individuals, diminished with increasing ADOS score. For neutral expressions this correlation was not significant. Experiment 2 required explicit emotion classifications in a facial emotion composite task while eye gaze was manipulated incidentally. A group of 22 children with ASD was compared to a propensity score-matched group of TD children. The same comparison was carried out for a subgroup of nine children with ASD who were less trained in social cognition, according to clinician’s report. The ASD group performed overall worse in emotion recognition than the TD group, independently of emotion or gaze direction. However, for disgust expressions, eye tracking data revealed that TD children fixated relatively longer on the eyes of the stimulus face with a direct gaze as compared with averted gaze. In children with ASD we observed no such modulation of fixation behavior as a function of gaze direction. Overall, the present findings from ERPs and eye tracking confirm the hypothesis of an impaired sensitivity to gaze direction in children with ASD or elevated autistic traits, at least for specific emotions. Therefore, we conclude that multimodal investigations of the interaction between emotional processing and stimulus gaze direction are promising to understand the characteristics of individuals differing along the autism trait dimension. (shrink)

This book envisions a relational and multimodal conceptualisation of higher education, which advocates for knowledge to be analyzed under three relational dimensions: social, digital and environmental. The volume draws on interdisciplinary approaches grounded in Peirceian semiotics, in exploring an integration of these dimensions in higher education theory and practice. The book situates learning in an awareness of the environment, grounded in principles of interconnectedness and flattened hierarchies. The volume features practical case studies through dialogues with higher education teachers, presenting (...) different ways of practicing multimodal and relational higher education which prioritize and embrace expression and creation through more than one dominant modality of communication. The book highlights how a relational multimodal paradigm can serve as a way forward for universities to address global challenges concerning social, technological, and environmental futures through a social justice lens. This innovative book will be of interest to students and scholars in semiotics, multimodality, and higher education, as well as stakeholders in these fields. (shrink)

The conceptual metaphor “THOUGHT IS FOOD” is exemplified in many verbal expressions. Nevertheless, how food metaphors are realized through the actual dining experience remains unexplored. Based on a food design event called EATAIPEI that took place in the London Design Festival in 2015, one aimed at promoting Taipei as World Design Capital 2016, this article analyzes how the multimodal metaphors of food were creatively represented and elaborated within it. This study proposes an analytical framework that combines insights from cognitive (...) linguistics with an intercultural performative view of food presentation and food metaphor. More specifically, it treats complex metaphors as resulting from the interactions of primary metaphors, proposition schemas and intercultural performance. The multimodal metaphors of food are complex metaphors composed of combined sets of primary metaphors, enriched by schema propositions and embedded within food performance. The experience of eating in the event reactivates the sensorimotor experiences that underpin the complex metaphors. (shrink)

This paper is a cognitive semantic analysis of a CCTV educational commercial, which is one of a series designed and produced in preparation for, and in celebration of, the Beijing 2008 Olympic Games. Called the “Beijing Opera Episode”, this TV commercial converges on the theme: “To mount the stage of the world, and to put on a show of China”. That is, China sees her hosting of the 2008 Olympics by Beijing as a great opportunity for her to step onto (...) the international stage to perform a show of China. This theme, which alludes to China's globalization and retention of cultural identity, is conceptualized and expressed in terms of a central metaphor of theatrical performance: hosting the beijing olympics is performing beijing opera on an international stage. This study first analyzes in detail how this central metaphor is manifested multimodally through visual and aural as well as verbal discourse, thus examining it as a multimodal metaphor. The study then applies a decompositional approach, based on the distinction between primary and complex metaphors, to analyzing the central metaphor as a metaphorical compound with its internal structures and components, which is nevertheless built on a more general cognitive foundation. The study also offers a linguistic perspective on a cultural model underlying the central metaphor, which is arguably a Chinese cultural model for understanding various aspects and events of life in general. At the core of this cultural model is the widespread life is a stage metaphor, which however has a specific manifestation within the Chinese cultural context. (shrink)

The analysis of legal communication has almost exclusively been the domain of discourse analysts focusing on the ways that the linguistic system is used to realise legal meanings. Multimodal discourse analysis, where visual forms in combination with traditional linguistic expressions co-occur, is now also an area of expanding interest. Taking a Systemic Functional Linguistics “social semiotic” perspective, this paper applies and critiques an analytical framework that has been used for examining intersemiotic complementarity in various types of page-based multimodal (...) texts by analysing a cartoon satirizing the demands that verdict deliberations can make on a citizen jury seated in a Crown Court. While the multimodal analysis reveals that a straightforward application of this analytical framework is useful in and of itself, it is argued that the framework needs to be extended to also account for further and more complex layers of represented meaning. The analysis is also situated in the wider legal social semiotic, and draws attention to international debates on the efficacy of the jury system itself. (shrink)

Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than (...) facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition. (shrink)

As humans interact in the world, they often orient one another's attention to objects through the use of spoken demonstrative expressions and head and/or hand movements to point to the objects. Although indicating behaviors have frequently been studied in lab settings, we know surprisingly little about how demonstratives and pointing are used to coordinate attention in large-scale space and in natural contexts. This study investigates how speakers of Quiahije Chatino, an indigenous language of Mexico, use demonstratives and pointing to give (...) directions to named places in large-scale space across multiple scales. The results show that the use and coordination of demonstratives and pointing change as the scale of search space for the target grows. At larger scales, demonstratives and pointing are more likely to occur together, and the two signals appear to manage different aspects of the search for the target: demonstratives orient attention primarily to the gesturing body, while pointing provides cues for narrowing the search space. These findings underscore the distinct contributions of speech and gesture to the linguistic composite, while illustrating the dynamic nature of their interplay.s in Spanish and Quiahije Chatino are provided as appendices.Se incluyen como apéndices resúmenes en español y en el chatino de San Juan Quiahije. SonGktyiCreCinH, ngyaqCskaEktyiCnoEndaHsonBnaFngaJnoIngyaqCloEktyiCreC, ngyaqCranFchaqExlyaKqoEchaqFjnyaJnoAndywiqArenqAKchinAKyqyaC. (shrink)

Many nonhuman animals produce facial expressions which sometimes bear clear resemblance to the facial expressions seen in humans. An understanding of this evolutionary continuity between species, and how this relates to social and ecological variables, can help elucidate the meaning, function, and evolution of facial expression. This aim, however, requires researchers to overcome the theoretical and methodological differences in how human and nonhuman facial expressions are approached. Here, we review the literature relating to nonhuman facial expressions and suggest future (...) directions that could facilitate a better understanding of facial expression within an evolutionary context. (shrink)

Adult-infant early dyadic interactions have been extensively explored by developmental psychologists. Around the age of 2 months, infants already demonstrate complex, delicate and very sensitive behaviors that seem to express their ability to interact and share emotions with their caregivers. This paper presents 3 pilot studies of parent-infant dyadic interaction in various set-ups. The first two present longitudinal data collected on two infants aged between 1 and 6 months and their mothers. We analyzed the development of coordination between them, at (...) the motor and at the vocal level. The 3rd pilot study aims to explore interpersonal coordination in both vocal behavior and motor activity for one infant and his mother at 2, 4 and 6 months. These pilot studies however leave a number of questions open concerning developmental changes and infants’ progressive mastery of interaction. We identify areas worth examining and try to tease out specific issues that may help develop new methodological pathways for the study of early naturalistic social interaction. We assume that a continuous, rather than discrete, approach would better capture the changes taking place in the various communicative modalities, while also displaying each dyad’s specificity and the narrative dimension of social engagement between infants and caregivers. (shrink)

One of the recent areas of discussion in aesthetics and the visual arts is the tension between the so-called “ocularcentric” tradition, on the one hand, and the tendency to move in a multisensory, multimodal direction, on the other. My aim in this paper is to bring out this tension by tracing it in a number of moments; firstly, in the late 19th-early 20th century discussion, concerning the “total art work” and the contribution of synaesthesia; secondly, the reaction to what (...) Clement Greenberg called the modernist “reduction to the visual”, expressed in a variety of multimodal art forms in the latter half of the 20th century. The weight of my paper is given to two contemporary discussions with significant repercussions in aesthetics: a. the opposition between “disembodiment” and “embodied immersion” in the area of new media; and b. the debate between modularity and plasticity of perception in neuroscience. The conclusion to which I am led is that the possibilities of sensory integration offered by new media and of sensory plasticity - offered by synaesthesia - suggest new creative ways in which our senses may interact and enhance our experience of the world, as well as elucidate artistic creativity and aesthetic appreciation. (shrink)

XML documents and, more generally, semistructured data, can be seen as labelled graphs. In this paper we set a correspondence between such graphs and the models of a language of hybrid multimodal logic. This allows us to characterize a schema for semistructured data as a formula of hybrid multimodal logic, and instances of the schema as models of this formula. We also investigate how to express in such a logic integrity constraints on semistructured data, in particular some classes (...) of constraints widely considered in the literature. The contribution of this work is twofold: 1) We generalize the notion of schema, by proposing a definition of schema were references are “well typed”. 2) We formalize semistructured data, schema for semistructured data and integrity constraints in a unique framework, namely hybrid multimodal logic. (shrink)

This paper examines the forms and functions of religious Internet memes that relate to Covid-19, with a view to identifying the conceptual metaphors that underlie the creation of the memes. The data, which consist of thirty religious Internet memes shared in the Nigerian WhatsApp space, are analyzed qualitatively using the categorization of religious Internet memes, and the concept of multimodal metaphors. The memes contain (non-)linguistic metaphors such as the picture of Biblical Noah’s ark and expressions such as Noah’s family (...) was on lockdown, which reveal underlying conceptual metaphors such as LOCKDOWN IS A GODLY INSTRUCTION and COVID-19 IS A WAR. The memes are used to allay the fears of people in the face of the disease, and encourage adherence to lockdown orders, amongst others. The study concludes that the forms and functions of these religious memes assist in revealing the multimodal conceptual metaphors underlying the memes. (shrink)

The human experience is shaped by information from different perceptual channels, but it is still debated whether and how differential experience influences language use. To address this, we compared congenitally blind, blindfolded, and sighted people's descriptions of the same motion events experienced auditorily by all participants (i.e., via sound alone) and conveyed in speech and gesture. Comparison of blind and sighted participants to blindfolded participants helped us disentangle the effects of a lifetime experience of being blind versus the task-specific effects (...) of experiencing a motion event by sound alone. Compared to sighted people, blind people's speech focused more on path and less on manner of motion, and encoded paths in a more segmented fashion using more landmarks and path verbs. Gestures followed the speech, such that blind people pointed to landmarks more and depicted manner less than sighted people. This suggests that visual experience affects how people express spatial events in the multimodal language and that blindness may enhance sensitivity to paths of motion due to changes in event construal. These findings have implications for the claims that language processes are deeply rooted in our sensory experiences. (shrink)

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic expressions are addressed: proximal (...) (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance. (shrink)

In this contribution we will approach language and images in different modalities: speech and face, anticipated and imagined movements, illusions on the sound by the image. It will be the opportunity for us to revisit the Gestalt concepts which were considered obsolete since structuralism in Humanities. As instantiated by Gilbert Durand in The Anthropological Structures of the Imaginary (1999, French 1st ed. 1960), we shall recall that Gestalt is not—even implicitly—an exclusively static approach to cognition. On the contrary we will (...) emphasize that forms can emerge from movements and stabilize in memory, thence morphology. And for speech, we will consider phonology as the stabilized outcome from audible and visible gestures of the mouth—caught in the time-course of a flow of some four in six syllables by second—via the perception of motor coordinations: which is necessary to master the most common expression of language between humans. We will adopt here also for the perceptual flows of spoken language, Multimodal Scene Analysis, the ongoing legacy of Gestalt Theory, including one of our main research topics: the gestural expression of the face and the hand for Deaf people, who practise as an augment French LPC (Langue française Parlée Complétée), adapted in 1967 from Cued Speech by its originator Dr Richard Cornett. (shrink)

Emotion recognition is an important part of building an intelligent human-computer interaction system and plays an important role in human-computer interaction. Often, people express their feelings through a variety of symbols, such as words and facial expressions. A business ecosystem is an economic community based on interacting organizations and individuals. Over time, they develop their capabilities and roles together and tend to develop themselves in the direction of one or more central enterprises. This paper aims to study a multimodal (...) ER method based on attention mechanism. It analyzes the current emotional state of consumers and the development direction of enterprises through multi-modal ER of human emotions and analysis of market trends, so as to provide the most appropriate response or plan. This paper firstly describes the related methods of multimodal ER and deep learning in detail, and briefly outlines the meaning of enterprise strategy in the business ecosystem. Then, two datasets, CMU-MOSI and CMU-MOSEI, are selected to design the scheme for multimodal ER based on self-attention mechanism. Through the comparative analysis of the accuracy of single-modal and multi-modal ER, the self-attention mechanism is applied in the experiment. The experimental results show that the average recognition accuracy of happy under multimodal ER reaches 91.5%. (shrink)

The expression ‘Fake News’ inside Internet memes engenders significant online virulence, possibly heralding an iconoclastic emergence of weaponized propaganda for assaulting agencies reared on public trust. Internet memes are multimodal artifacts featuring ideological singularities designed for ‘flash’ consumption, often composed by numerous voices echoing popular, online culture. This study proposes that ‘Fake News’ Internet memes are weaponized iconoclastic multimodal propaganda discourse and attempts to delineate them as such by asking: What power relations and ideologies do Internet memes (...) featuring the expression ‘fake news’ harbor? How might those manifestations qualify as WIMP discourse? A multimodal critical discourse analysis of a small pool of ‘fake news’ Internet memes drawn from four popular social media websites revealed what agencies were often targeted and from what political canons they likely emerged. Findings indicate that many Internet memes featuring ‘fake news’ are specifically directed, revealing an underlying hazard that WIMP discourse could diminish democratic processes while influencing online trajectories of public discourse. (shrink)

In contrast to symbol-manipulation approaches, Cognitive Linguistics offers a modal rather than an amodal account of meaning in language. From this perspective, the meanings attached to linguistic expressions, in the form of conceptualisations, have various properties in common with visual forms of representation. This makes Cognitive Linguistics a potentially useful framework for identifying and analysing language-image relations in multimodal texts. In this paper, we investigate language-image relations with a specific focus on intersemiotic convergence. Analogous with research on gesture, we (...) extend the notion of co-text images and argue that images and language usages which are proximal to one another in a multimodal text can be expected to exhibit the same or consistent construals of the target scene. We outline some of the dimensions of conceptualisation along which intersemiotic convergence may be enacted in texts, including event-structure, viewpoint, distribution of attention and metaphor. We take as illustrative data photographs and their captions in online news texts covering a range of topics including immigration, political protests, and inter-state conflict. Our analysis suggests the utility of Cognitive Linguistics in allowing new potential sites of intersemiotic convergence to be identified and in proffering an account of language-image relations that is based in language cognition. (shrink)

This paper explores how Wundt’s and Bühler’s respective conceptions of gestural expression have implications for how each conceives of what, in broad terms, may be understood as a ‘grammar of gestures’: that is, the rules for the formation and performance of gestures with and without speech. Unlike previous scholarship that has looked at the relationship of Wundt and Bühler, the aim here will be to give particular attention to the relevance of their respective accounts for current philosophical and linguistic (...) research on gesture. Building off of Bühler, we can offer an alternative to the psychologistic and solipsistic model of gestural expression that can be found in Wundt and that seems to be resurgent today in work by McNeil and others. Via Bühler’s notions of diacrisis and of the underlying functions that guide human communication, we propose an understanding of the relationship between gestures and speech that aims to render accurately both the highly contextualised and the highly structured conditions of their mutual employment. (shrink)

We discuss here one of our projects, aimed at developing an automatic facial expression interpreter, mainly in terms of signaled emotions. We present some of the relevant findings on facial expressions from cognitive science and psychology that can be understood by and be useful to researchers in Human-Computer Interaction and Artificial Intelligence. We then give an overview of HCI applications involving automated facial expression recognition, we survey some of the latest progresses in this area reached by various approaches (...) in computer vision, and we describe the design of our facial expression recognizer. We also give some background knowledge about our motivation for understanding facial expressions and we propose an architecture for a multimodal intelligent interface capable of recognizing and adapting to computer users¿ affective states. Finally, we discuss current interdisciplinary issues and research questions which will need to be addressed for further progress to be made in the promising area of computational facial expression recognition. (shrink)

Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at (...) a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine and a subject-independent model based on long short-term memory network. In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals dataset and the Multimodal Database for Affect Recognition and Implicit Tagging dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated. (shrink)

This study focuses on a series of videos aimed at teacher recruitment in Singapore and how they are used as an ideological tool for persuasion. By adopting a multimodal critical discourse analysis approach to focus on affect, it examines how these videos create and promulgate the ideology of an ideal teacher as one who is caring, encouraging and supportive of students. The analysis shows how affect is not only embodied in and performed by the primary protagonists in the video (...) narratives through their action, facial expression, posture and speech. It is also evoked through various secondary meaning-making modes, such as focus, angle, lighting, background music and setting, through which the narratives unfold. More importantly, it demonstrates how affect is used not only as a means to arouse and engage viewers’ sensibilities but also as a persuasive strategy to manufacture and manage particular social and economic realities in contemporary society. (shrink)

While influential works since the 1970s have widely assumed that imitation is an innate skill in both human and non-human primate neonates, recent empirical studies and meta-analyses have challenged this view, indicating other forms of reward-based learning as relevant factors in the development of social behavior. The visual input translation into matching motor output that underlies imitation abilities instead seems to develop along with social interactions and sensorimotor experience during infancy and childhood. Recently, a new visual stream has been identified (...) in both human and non-human primate brains, updating the dual visual stream model. This third pathway is thought to be specialized for dynamics aspects of social perceptions such as eye-gaze, facial expression and crucially for audio-visual integration of speech. Here, we review empirical studies addressing an understudied but crucial aspect of speech and communication, namely the processing of visual orofacial cues and its integration with vocal auditory cues. Along this review, we offer new insights from our understanding of speech as the product of evolution and development of a rhythmic and multimodal organization of sensorimotor brain networks, supporting volitional motor control of the upper vocal tract and audio-visual voices-faces integration. (shrink)

This study investigates the influences of action research on primary school English instruction from five dimensions in the classroom, viz., types of questions, language errors, gestures, facial expressions, and interpersonal distance. Four English teachers’ 9 real classroom teaching videos before and after action research are collected and annotated by using ELAN software. The results show that primary school English teachers in Chinese rural areas prefer closed questions to open questions; They make some language errors; Deictic gestures are the most common (...) gestures used, while metaphoric gestures, beat gestures and iconic gestures are rare; Teachers have the same preference for three types of facial expressions, and teachers’ serious expressions accounts for most of the time; They seldom keep an intimate distance or personal distance from their students. Second, Action research is effective to motivate teachers in rural areas, who make great progress in all five dimensions after AR: more open questions are asked; pragmatic errors and grammatical errors are reduced; deictic gestures increase; apathetic decrease, and more intimate distance is exhibited in the interpersonal distance dimension. Third, teacher’s English teaching competence is partly transferrable to her future professional development, and this is also the long-lasting effect of AR. (shrink)

The business of a sentence is not only to describe some state of affairs but also to perform other kinds of speech acts like ordering, suggesting, asking, etc. Understanding the kind of action performed by a speaker who utters a sentence is a multimodal process which involves the computing of verbal and non-verbal information. This work aims at investigating if the understanding of a speech act is affected by the gender of the actor that produces the utterance in combination (...) with a certain facial expression. Experimental data collected show that, as compared to men, women are less likely to be perceived as performers of orders and are more likely to be perceived as performers of questions. This result reveals a gender bias which reflects a process of women’s subordination according to which women are hardly considered as holding the hierarchical social position required for the correct execution of an order. (shrink)

This contribution investigates the use of the Czech particle jako in naturally occurring conversations. Inspired by interactional research on unfinished or suspended utterances and on turn-final conjunctions and particles, the analysis aims to trace the possible development of jako from conjunction to a tag-like particle that can be exploited for mobilizing affiliative responses. Traditionally, jako has been described as conjunction used for comparing two elements or for providing a specification of a first element [“X like Y”]. In spoken Czech, however, (...) jako can be flexibly positioned within a speaking turn and does not seem to operate as a coordinating or hypotactic conjunction. As a result, prior studies have described jako as a polyfunctional particle. This article will try to shed light on the meaning of jako in spoken discourse by focusing on its apparent fuzzy or “filler” uses, i.e., when it is found in a mid-turn position in multi-unit turns and in the immediate vicinity of hesitations, pauses, and turn suspensions. Based on examples from mundane, video-recorded conversations and on a sequential and multimodal approach to social interaction, the analyses will first show that jako frequently frames discursive objects that co-participants should respond to. By using jako before a pause and concurrently adopting specific embodied displays, participants can more explicitly seek to mobilize responsive action. Moreover, as jako tends to cluster in multi-unit turns involving the formulation of subjective experience or stance, it can be shown to be specifically designed for mobilizing affiliative responses. Finally, it will be argued that the potential of jako to open up interactive turn spaces can be linked to the fundamental comparative semantics of the original conjunction. (shrink)

_This study probes into a comparative multimodal discourse analysis (MDA) of two influential Pakistani rap music videos ‘The Sibbi Song’ by Abid Brohi and ‘Karachi Chal’ performed by Young Stunners, a rap duo consisting of Talha Anjum and Talhah Yunus to explore their portrayal of societal hierarchies and individual identity. Drawing upon the theoretical framework of MDA the analysis has discovered the linguistic, visual, aural, spatial, and gestural elements of each video. The first video portrays the protagonist’s journey from (...) innocence to success, while the other highlights urban working-class struggles through nuanced examination. Major findings reveal themes of resilience, social change, and identity negotiation of urban life and cultural expression in Pakistan’s diverse societal fabric which individuals from distinct cultural backgrounds often use to express their norms and values through various forms of visual arts and entertainment. Furthermore, the global phenomenon of rap music has gained significant traction, offering a platform for contemporary rap music artists in Pakistan to articulate their unique identities, confrontations, and triumphs as a cultural shift towards self-expression._. (shrink)

Brand image constitutes one of the most salient, over-defined, heavily explored and multifariously operationalized conceptual constructs in marketing theory and practice. In this Chapter, definitions of brand image that have been offered by marketing scholars will be critically addressed in the context of a culturally oriented discussion, informed by the semiotic notion of iconicity. This cultural bend, in conjunction with the concept’s semiotic contextualization, are expected both to dispel terminological confusions in the either inter-changeable or fuzzily differentiated employment of such (...) terms as brand image, symbol, icon, as well as to address the function of brand image at a deeper level than a mere construct to be operationalized in quantitative studies of purchase drivers. This shift in focus is coupled with a critical turn from the cogito-centric view of the consuming subject through the cognitivist lens of the AI metaphor as decision-making centre at the origin of largely conscious meaning-making, in favor of a psychoanalytically informed approach that considers figurativity as an essential process whereby brand image is formed. In these terms, brand image will be intimately linked to brand images as figurative multimodal expressive units and rhetorical tropes, as figurative syntax, that are responsible for shaping an idiolectal brand language, as well as to distinctive levels of iconicity as textual condition of possibility of a brand language. In order to understand the role of iconicity as fundamental condition of brand textuality, rather than just a procedure for spawning brand images, the discussion is contextualized in a wider framework involving the culturally situated source of brand images, how they become correlated with brand image concepts and how correlations between brand images and brand image result in brand knowledge structures (Keller 1998). This opening up of the discussion on iconicity is enacted against the background of what I call the Brand Imaginarium which involves: (i) a critical engagement with the dominant cognitivist perspective in branding research that prioritizes individual memory in brand knowledge formation, through a cultural branding lens that involves two additional types of memory, viz. communicative and cultural (ii) a critical engagement with the cognitivist perspective on brand knowledge formation that prioritizes conscious processing of stimuli (as ‘brute facts’, rather than as already semiotized expressive units) in a cognitive mechanism from which the faculty of imagination has been expelled, by restoring the importance of imagination in brand knowledge formation, and, concomitantly, by showing that the highly figurative language of brands may not be researched thoroughly unless imagination is posited anew as processing correlate (iii) the adoption of an expansionist approach to the role of the imaginary in brand knowledge formation, from cognitive (or psychic) faculty, to a more sociologically inclined process of inter-subjective mirroring, and concomitantly as imaginary social significations (Castoriades 1985) that are shared by culturally conditioned and habituated subjects that engage in meaningful cultural practices, rather than individual processing monads. (shrink)

Two eye-tracking experiments investigated how and when pointing gestures and location descriptions affect target identification. The experiments investigated the effect of gestures and referring expressions on the time course of fixations to the target, using videos of human gestures and human voice, and animated gestures and synthesized speech. Ambiguous, yet informative pointing gestures elicited attention and facilitated target identification, akin to verbal location descriptions. Moreover, target identification was superior when both pointing gestures and verbal location descriptions were used. These findings (...) suggest that gesture not only operates as a context to verbal descriptions, or that verbal descriptions operate as a context to gesture, but that they complement one another in reference resolution. (shrink)

The juxtaposition between selfie and quotation is an emerging user-generated Facebook content. This exploratory study is the first to show how Facebook users self-represent themselves through Relational Processes, based on the intertextuality between these verbal and visual modes. Relational Processes refer to the process of characterizing, identifying or describing a person or entity. In this study, 132 Facebook quotations were categorized into three types of Relational Processes based on the users’ selfies. The clauses were restated into I-expressions to center the (...) users’ multimodal experiential and egocentric positioning. The findings revealed that Attributive Relational Processes, such as “I am happy,” predominated in the self-quotation juxtaposition. These relational processes are then discussed based on the potential social projections within the Philippine cultural circuit, namely: the Filipino culture of “inggit” (envy); crab mentality; being social climbers; and the neoliberal and neocolonial status of the Philippines as channeled through the Western-driven quotations. Overall, we argue that the juxtaposition of these two semiotic modes can lead to the creation of a new persona of the Facebook users, enhancing their digital self-remodeling and self-representation. (shrink)

Humans typically combine linguistic and nonlinguistic information to comprehend emotions. We adopted an emotion identification Stroop task to investigate how different channels interact in emotion communication. In experiment 1, synonyms of “happy” and “sad” were spoken with happy and sad prosody. Participants had more difficulty ignoring prosody than ignoring verbal content. In experiment 2, synonyms of “happy” and “sad” were spoken with happy and sad prosody, while happy or sad faces were displayed. Accuracy was lower when two channels expressed an (...) emotion that was incongruent with the channel participants had to focus on, compared with the cross-channel congruence condition. When participants were required to focus on verbal content, accuracy was significantly lower also when prosody was incongruent with verbal content and face. This suggests that prosody biases emotional verbal content processing, even when conflicting with verbal content and face simultaneously. Implications for multimodal communication and language evolution studies are discussed. (shrink)

Storytelling pivots around stance seen as a window unto emotion: storytellers project a stance expressing their emotion toward the events and recipients preferably mirror that stance by affiliating with the storyteller’s stance. Whether the recipient’s affiliative stance is at the same time expressive of his/her emotional resonance with the storyteller and of emotional contagion is a question that has recently attracted intriguing research in Physiological Interaction Research. Connecting to this line of inquiry, this paper concerns itself with storytellings of sadness/distress. (...) Its aim is to identify factors that facilitate emotion contagion in storytellings of sadness/distress and factors that impede it. Given the complexity and novelty of this question, this study is designed as a pilot study to scour the terrain and sketch out an interim roadmap before a larger study is undertaken. The data base is small, comprising two storytellings of sadness/distress. The methodology used to address the above research question is expansive: it includes CA methods to transcribe and analyze interactionally relevant aspects of the storytelling interaction; it draws on psychophysiological measures to establish whether and to what degree emotional resonance between co-participants is achieved. In discussing possible reasons why resonance is (not or not fully) achieved, the paper embarks on an extended analysis of the storytellers’ multimodal storytelling performance (reenactments, prosody, gaze, gesture) and considers factors lying beyond the storyteller’s control, including relevance, participation framework, personality, and susceptibility to emotion contagion. (shrink)

This article attempts to evaluate emotive meanings across languages and cultures expressed and elicited semiotically from viewers. It investigates the challenges of subtitling emotive feelings in the American filmHomeless to Harvard(2003) into Arabic. It adopts Paul Thibault’s (2000. The multimodal transcription of a television advertisement: Theory and practice. In Anthony Baldry (ed.),Multimodality and multimediality in the distance learning age, 311–385. Campobasso: Palladino Editore) method of multimodal transcription and Feng and O’Halloran’s (2013. The multimodal representation of emotion in (...) film: Integrating cognitive and semiotic approaches.Semiotica197(1/4). 79–100) framework of the multimodal representation of emotion to formulate strategies for subtitling emotion from English into Arabic. Additionally, Feng and O’Halloran’s (2013) framework is adapted to show how stylistic choices (e.g., cinematography, music, soundtracks, etc.) and semiotic expressions can elicit emotion from viewers. The social semiotic model is employed to investigate how emotive representation is realized through verbal and non-verbal items. The findings showed that the filmmakers had properly used stylistic choices and various semiotic techniques to elicit emotion from the viewer derived from the emotion of the film’s heroine. (shrink)

Polarity Items are linguistic expressions known for being a ‘lexically controlled’ phenomenon. In this paper we show how their behavior can be implemented in a deductive system. Further- more, we point out some possible directions to recast the deductive solution into a Tree Ad- joining Grammar system. In particular, we suggest to compare the proof system developed for Multimodal Categorial Grammar (Moot & Puite, 1999) with the Partial Proof Trees proposed in (Joshi & Kulick, 1997).

We introduce a multimodal framework of deontic action logic which encodes the interaction between two fundamental procedures in normative reasoning: conceptual classification and deontic classification. The expressive power of the framework is noteworthy, since it combines insights from agency logic and dynamic logic, allowing for a representation of many kinds of normative conflicts. We provide a semantic characterization for three axiomatic systems of increasing strength, showing how our approach can be modularly extended in order to get different levels of (...) analysis of normative reasoning. Finally, we discuss ways in which the framework can be used to capture other formalisms proposed in the literature, as well as to model searching problems in Artificial Intelligence. (shrink)

Animation is widely acknowledged for dynamically visualizing information and has been increasingly used in educational context. However, the growing presence of educational animation has not been accompanied by well-informed studies that focus on the semiotic features of animation. An emerging perspective influenced by Social Semiotics and Systemic Functional Linguistics greatly complements the current trend of animation studies in the field of science education. Studies taking that perspective model animation as stratified systems (consisting of an expression plane and a content (...) plane) of meaning-making options. Building on their work, this paper further develops the systems of animation on the two planes. Besides the concept of stratification, this paper also introduces a metafunctional perspective for the modeling of animation, focusing on building the ideational meaning system and textual meaning system for animation on the content plane based on an examination of online science animations. It is found that educational animation is not homogenous: animation can construe different human experience and can be used to fulfill different functions. The paper closes with a discussion of the advantages of adopting a stratified metafunctional model in studying animation and the implications for future studies in the field of multimodal semiotics and science education. (shrink)