The article investigates a multimodal practice for self-selecting observed in a video-taped corpus of work meetings: the use of pointing gestures predicting possible turn completions and projecting the emergence of possible next speakers. This practice is analyzed in various sequential positions, namely at turn beginnings and at pre-beginnings. It displays recipients' practical online turn parsing, and their orientation to transition spaces, and to TCU, completions in a visible, recognizable, public way. It shows the emergent and progressive establishment of speakership, (...) exploiting both systematic features of turn-taking and specific features of the interactional space. The article explores the emic, locally situated and contingent definition of speakership by considering not only where pointing gestures begin but also where they end: pointing does not only end in pre-completion positions, projecting turn end, but can also persist through sequences, showing speaker's orientations to her turn's sequential implicativeness. Thus, by observing a particular multimodal practice within a specific interactional setting, the article explores participants' orientations to the rights and obligations associated with talk-related categories such as `non current speaker', `possible next speaker', `incipient speaker' and `current speaker'. (shrink)

This article examines the interaction between grammar and multimodal resources by analyzing reported speech in Korean multiparty face-to-face interaction. The operation of two relevancy rules — minimization and recognition in interaction — is examined together with how the absence or presence of grammar is complemented by multimodal resources of various sorts. For the analysis, three categories are posited depending on who the quoted character is in the talk. In quoting oneself or co-participants in the talk, syntactic (...) resources, prosody, sequential organization and embodied participant frameworks collaborate in the interactional management of the quote without overt realization of grammatical resources traditionally proposed for building quotations in Korean. The multimodal character of quoted speech is most weakly realized when quoting a non-present character, not only because of the limitation in body and spatial resources available in the local context to recover various referential terms, but more importantly, because the interactional project that the parties are engaged in makes a distancing device such as the quotative particle -ko relevant. This study provides evidence for the intricate interaction between multimodal resources and grammar through investigation of an important realm of human interaction, that of reported speech. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the (...) multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

Firstly, this paper aims to analyze how verbal and visual modes contribute to build two basic components of cartoons: the referenced situation and the fictional situation. Secondly, it aims to unravel the semiotics of this discursive genre by offering a fine-grained picture of modal variations and continuities of the rhetorical resources deployed, by means of which the fictional situation is displayed. The corpus is composed of 50 multimodal cartoons chosen by cartoonists of 22 nationalities as the most representative (...) of their work in the frame of a wide questionnaire. Category analysis and non-parametric statistics were applied. We identified and fleshed out four different types of modal functioning of resources: visual preference, verbal preference, modal flexibility, and genuine multimodal functioning. Even if visual and multimodal metaphor were often in charge of transferring meaning from fiction to the referenced situation, results show that other resources such as antithesis, irony, and syllepsis can become the key resources of cartoons. Findings highlight the multilayered functioning of cartoons, among which plastic features can play a significant role, and provide further evidence in favor of the hyperdetermination of humor hypothesis proposed by Attardo. (shrink)

Irregular self-selection is a demonstration of active involvement in interaction. English as a foreign language learners’ talk-in-interaction is one of such cases. Yet, little research has explored when, how, and why learners implement this action. The aim of this article is to address these issues in Chinese postgraduate EFL learners’ conversations from the perspective of multimodal interaction. To this end, we provide descriptive statistics and use multimodal conversation analysis to investigate the detailed process of irregular self-selection. The results (...) show the interactional sensitivity of learners, and all the successful irregular self-selection can be divided into the following three types: turn interruption, turn competition TC), and turn holding abortion. Learners implement this action by using multimodal resources, including lexis, syntax, pitch reset, intensity enhancement, gaze, and so forth. However, their body movements lack diversity, causing behaviors to be constrained and inactive. The main purpose of irregular self-selection is to provide knowledge that contributes to topical development. This study reveals that Chinese postgraduate EFL learners are interactionally competent members. They are able to achieve communicative goals but in a low diversity of body movements. The findings help to understand the detailed process of speakership claiming in EFL learners’ conversations. (shrink)

Research has shown that suicide rate in Australia is on the rise and that most people who die by suicide are not in contact with mental health services. They most likely communicate their suicidal thoughts to family members or close friends, whose responses may sound unhelpful and/or dismissive, thus reinforcing suicidal ideation. This national emergency has been tackled via a social media campaign, #YouCan Talk, launched by a government-supported digital platform, BeyondBlue. This article adopts a multimodal discourse analysis approach (...) to investigate how peer support is encouraged and articulated in the context of mental health discourse for suicide prevention. The two case studies selected for analysis from the BeyondBlue platform are the #YouCanTalk social media campaign, designed to teach carers to identify severe depression and effectively respond to suicide warning signs and a sample of posts from a thread for peer support in a monitored online forum devoted to help carers who seek peer advice. Unlike previous research, the article focuses on how pronouns and verbs index interpersonal relations in a systemic-functional perspective as well as other multimodal resources, such as visuals, layout and hyperlinking, to understand how identities are entextualised by both professional health providers and peer carers in digital platforms that address mental health issues. (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

Although multimodal input has the potential to lead to more sound learning outcomes, it carries the risk of causing cognitive overload, making it difficult to determine the exact effects of multimodal input on the second language phrase learning. This study tests the efficacy of multimodal input on L2 phrase learning. It adopts a mixed-method approach by utilizing both quantitative and qualitative data. The experimental design is a 2 × 3 mixed model, with a group [the experimental group (...) and the control group ] as the between-subject factor and time as the within-subject factor. A total of 66 participants were divided into two groups. All materials incorporated three aspects of phrase knowledge, but the materials of the CG were unimodal in that they were offered only on paper, and of the EG were multimodal in that they included pictures, audio recordings, and video clips. After the treatment, a questionnaire and a semi-structured interview were given to the EG learners to explore their perceptions of using multimodal materials to learn L2 phrases. The results indicate that both groups had significant gains in learning phrases, but students with the multimodal input achieved significantly better results than those with the unimodal input. Moreover, the EG students had a generally positive attitude toward the use of multimodal resources. This study validates the efficacy of multimodal input on the acquisition of English phrases and shows that cognitive overload was avoided by sequencing the information. (shrink)

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillatory actions—language, facial, and gestural behaviors—produced during a route communication task. The focus was the temporal relationship between matching behaviors in the interlocutors (e.g., facial (...) behavior in one interlocutor vs. the same facial behavior in the other interlocutor). Cross-recurrence analysis revealed that within each category tested (language, facial, gestural), interlocutors synchronized matching behaviors, at temporal lags short enough to provide imitation of one interlocutor by the other, from one conversational turn to the next. Both social and cognitive variables predicted the degree of temporal organization. These findings suggest that the temporal structure of matching behaviors provides low-level and low-cost resources for human interaction. (shrink)

Dancers and musicians are experts in spatial and temporal processing which allows them to coordinate movement with music. This high-level processing has been associated with structural and functional adaptation of the brain for high performance sensorimotor integration. For these integration processes, adaptation does not only take place in primary and secondary sensory and motor areas but tertiary brain areas, such as the lateral prefrontal cortex (lPFC) and the intraparietal sulcus (IPS), also provide vital resources for highly specialized performance. Here, (...) we review evidence for the role of these brain areas in multimodal training protocols and integrate these findings into a new model of sensorimotor processing in complex motor learning. (shrink)

Multimodal discourse analysis offers a novel lens for the study of legal discourse. Within the closing arguments of a trial, prosecution and defense attorneys utilize various multimodal resources to present evidence, articulate opinions, and adopt stances to achieve their communicative goals. This research focuses on the closing arguments in the criminal trial concerning the death of George Floyd to analyze the multimodal representation of the closing arguments delivered by both prosecution and defense attorneys. It employs the (...) analytical framework of the attitude system and paralinguistic typology proposed by Martin and colleagues, and uses the UAM Corpus Tool and Praat to qualitatively annotate and quantify the linguistic and paralinguistic information. The findings reveal that these attorneys employ different multimodal resources, including language and paralanguage, to convey their attitudes towards the case’s participants. This leads to the construction of competing narratives, shedding light on how justice is seen to be done. (shrink)

Our brains make up a series of signs and are engaged in making or manifesting or reacting to a series of signs: through this semiotic activity they are at the same time engaged in “being minds” and so in thinking intelligently. An important effect of this semiotic activity of brains is a continuous process of “externalization of the mind” that exhibits a new cognitive perspective on the mechanisms underling the semiotic emergence of abductive processes of meaning formation. To illustrate this (...) process I will take advantage of the analysis of some aspects of the cognitive interplay between internal and external representations. I consider this interplay critical in analyzing the relation between meaningful semiotic internal resources and devices and their dynamical interactions with the externalized semiotic materiality suitably stocked in the environment. Hence, minds are material, “extended” and artificial in themselves. A considerable part of human abductive thinking is occurring through an activity consisting in a kind of reification in the external environment and a subsequent re-projection and reinterpretation through new configurations of neural networks and chemical processes. I also illustrate how this activity takes advantage of hybrid representations and how it can nicely account for various processes of creative and selective abduction, bringing up the question of how multimodal aspects involving a full range of sensory modalities are important in hypothetical reasoning. (shrink)

This article explores the socio-discursive competence of young children in Galician pre-schools. In particular, it deals with the way in which children – aged from 2;10 to 4;05 years – combine embodied actions and verbal resources to co-narrate stories with peers and adults. Using an audiovisual corpus of naturally occurring interactions, we have conducted a qualitative and multimodal analysis, observing how children react to diverse footings and negotiate participation frameworks in multiparty interactions. The findings suggest three progressive stages (...) in the development of socio-discursive competence in which children use different strategies: the stage of interactive proto-narrations, the stage of collaborative narrations and the stage of polyphonic monologue narrations. Our detailed examination reveals a progressive increase in the diversity of the participation frameworks represented by children, as well as in the complexity of the multimodal resources they use, and a rise in children’s grammatical repertoire linked to an improvement in children’s pragmatic skills. (shrink)

Studies of talk-and-bodily-conduct-in-interaction have inspired new insights into the way in which language, interaction and cognition might be articulated. More particularly, they have shown that participants mutually orient to the finely tuned multimodal details by which talk and action in interaction are sequentially organized. This article deals with this form of ‘participants’ multimodal online analysis’ by focusing on a particular phenomenon - the methodical practices and resources by which the end of a turn and of an activity (...) phase is projected and collectively achieved - in a specific videorecorded setting - a meeting in an architect’s office. It aims at questioning both how these local orientations are systematically displayed and exploited by the participants for the sequential organization of their activity and how they can be demonstrably observed by the analyst. (shrink)

It is argued in this paper that a multimodal analysis of turn-taking, one of the core areas of conversation analytic research, is needed and has to integrate gaze as one of the most central resources for allocating turns, and that new technologies are available that can provide a solid and reliable empirical foundation for this analysis. On the basis of eye-tracking data of spontaneous conversations, it is shown that gaze is the most ubiquitous next-speaker-selection technique. It can function (...) alone or enhance other techniques. I also discuss the interrelationship between the strength for sequential projection and the choice of next-speaker-selection techniques by a current speaker. The appropriate consideration of gaze leads to a revision of the turn-taking model in that it reduces the domain of self-selection and expands that of the current-speaker-selects-next sub-rule. It also has consequences for the analysis of “simultaneous starts”. (shrink)

This paper uses van Leeuwen’s Authority Legitimation framework to examine government posters published in the 1950s–1980s in Hong Kong, which serve as a means of shaping public opinion and legitimate social action. Martin and White’s Appraisal framework is also applied to provide the study with relevant analytical tools by which to construct evaluatively coherent authorial reading positions propagated by the government in the posters as well as aligning viewers with these desired positions. The government posters being studied are concerned with (...) various aspects of social practices in the domain of public health communication. This paper argues that the representations of these social practices are realized as texts – both visual and verbal – in the posters, as part of a broader recontextualization process where legitimate ways of doing things as stipulated in the government guidelines are recontextualized in these posters and conveyed to members of the public. The aim is two-fold: to provide a systematic analysis of how visual and verbal resources construe evaluation in the government posters and the consequent legitimation discourse as viewers are being persuaded to align with legitimate social practices; to highlight various forms of legitimation, namely, role model authority legitimation, impersonal authority legitimation, and conformity authority legitimation, that are realized in these posters through visual-verbal articulation. (shrink)

The paper examines selected aspects of the defence closing argument in a highly publicised criminal trial to illustrate the orchestration of various semiotic resources in legal persuasion and to explain their role in the creation of meaning. The study demonstrates that closing arguments are multimodal performances whose persuasiveness results from the combination of modes (speech, image, video, gaze, gesture, posture, proxemics) which contextualise and strengthen one another, rather than language alone. Drawing on earlier research into multimodality, courtroom rhetoric (...) and proximity in disciplinary genres, the analysis centres on the ways in which the defence counsel controls the rhetorical features of his narrative and constructs himself and the audience as people with similar understandings and goals. The study specifically demonstrates how the counsel constructs the proximity of commitment, the proximity of membership and the proximity of experience. It explores such facets of proximity as: organisation, argument structure, credibility, stance and engagement, and identifies key rhetorical strategies used to achieve the intended communicative effect. The analysis clearly shows that the persuasiveness of the counsel’s performance depends on the synchronisation of a range of meaning-making resources, which, if used in isolation, would result in a much less engaging argument. (shrink)

ABSTRACT Digital literature presents new ways of constructing aesthetic and emotional experience for child readers. This article reflects on emotion and affect in digital literature through the analysis of three literary apps for children. To deal with the complex phenomenon of the affective and emotional experience in digital reading, an interdisciplinary theoretical framework is proposed, relating Merleau-Ponty’s phenomenology to Kress and van Leeuwen’s multimodal social semiotics. The corpus includes the apps Little Red Riding Hood, by producer Nosy Crow, The (...) Monster at the End of This Book, by Jon Stone and Michael Smollin, and Hat Monkey, by Chris Haughton. It discusses how different semiotic resources are used in the representation of emotions and in suggesting emotional tension to the child reader, with embodiment and the intrinsic relationship between reader’s bodies and their emotional experience as a central part of apps’ textual construction. (shrink)

Open peer commentary on the article “A Cybernetic Approach to Contextual Teaching and Learning” by Philip Baron. Upshot: Universities in South Africa and elsewhere are being challenged to prepare graduates for the world of work and to recognize the multitude of resources that students bring with them to the higher-education experience. A multimodal approach to pedagogy may assist in both of these regards.

Face-to-face interaction is a primordial site for human activity and intersubjectivity. Empirical studies have shown how people reflexively exhibit a face orientation and work to establish a formation in which everyone is facing each other in local participation frameworks. The Face has also been described by, e.g., Levinas as the basis for a first ethical philosophy. Humans have established these Face-formations when interacting since time immemorial, but what happens when one of the participants is present through a telepresence robot? Based (...) on ethnomethodology, Peircean/goodwinian semiotics, multimodal conversation analysis and video data from a Danish residential rehabilitation center, the article shows the ways in which participants manage to interactively, cooperatively, and moment by moment achieve an F-formation in situ. The article contributes a detailed analysis and discussion of the kind of participant a telepresence robot is, in and through situated interactions: I propose that we term this participant the RoboDoc, given that it is an assemblage of a doctor who controls a robot. By focusing on the affordances of mobility, the article contributes to a renewed understanding of the importance and relevance of establishing Face-orientations in an increasingly technofied telepresence world. (shrink)

Journal Name: Semiotica Issue: Ahead of print.

As an intercultural modern art form, web-based ink and wash cartoons are significant tools to communicate cultural identities in the Chinese context because of their entertaining form, thought-provoking content, and profound cultural connotation. Against this background, the present study investigates the multimodal appraisal systems of 96 web-based ink and wash cartoons, focusing on attitudinal meanings and explicating how the attitudinal resources contribute to the communication of Chinese cultural identities. The analysis of 96 web-based ink and wash cartoons shows (...) that the cultural identities of the Chinese dream, Confucianism, collectivism, and optimism are communicated through the interplay between visual and verbal semiotic resources. The analytical results reveal a series of generic features of the web-based ink and wash cartoons that contribute to the promotion of cultural identities, including the popular theme, minimalist style, positive attitude prosody, and the use of culture-specific metaphor and ideation. These features underpin the strategies for promoting Chinese cultural identities and the shift in national branding policy. The study hopes to provide clues for government, social institutions, and individuals to promote cultural identities effectively through social media. (shrink)

In the era of eLearning 4.0, many researchers have suggested that multimodal input helps to enhance second language vocabulary learning. However, previous studies on the effects of multimodal teaching have failed to yield definitive conclusions. Furthermore, only few studies on the multimodal input of vocabulary learning have aimed at junior high school students and have focused on explicit vocabulary instruction in class. To explore the effects of multimodal input on English as a foreign language learners’ vocabulary (...) learning and summarize effective methods, this study adopts a mixed-method approach. Based on dual coding theory and cognitive load theory, the teaching materials in this study were designed using the resources provided by the multimodal corpus iWeb and other websites. A total of 60 junior high school students who learned EFL and had a similar English proficiency level were divided into an experimental group and a control group. Target words were selected through questionnaire I. During the experiment, the CG learned from monomodal materials while the EG received multimodal input, and an immediate post-test was delivered to the two groups. Questionnaire II was distributed in the EG, and five students of the EG were randomly selected for an interview. One week later, a delayed post-test was conducted on the EG and CG. The results showed that the EG performed better in the post-test but did worse than the CG in the delayed post-test. The results of the questionnaire and the interview suggest that students held both positive and negative attitudes toward the multimodal input approach in vocabulary learning. The study concludes with some implications for choosing a multimodal input approach in vocabulary learning, along with a number of suggestions on how to optimize its positive influence and minimize its negative effects. (shrink)

This book contributes to the analysis of film from a multimodal and textual perspective by extending formal semantics into the realm of multimodal discourse analysis. It accounts for both the inferential as well as intersemiotic meaning making processes in filmic discourse and therefore addresses one of the main questions that have been asked within film theory and multimodal analysis: How do we understand film and multimodal texts? The book offers an analytical answer to this question by (...) providing a systematic tool for the description of this comprehension process. It aims to advance knowledge of the various resources in filmic texts, the ways the resources work together in constructing meaning and the ways people understand this meaning construction. This new approach to film interpretation is thus able to remodel and improve the classical paradigm of film text analysis. (shrink)

This article, taking a social semiotic approach, analyses two pieces of music written, shared and exalted by two pre-1945 European fascist movements – the German NSDAP and the British Union of Fascists. These movements, both political and cultural, employed mythologies of unity, common identity and purpose in order to elide the realities of social distinction and political–economic inequalities between bourgeois and proletarian groups in capitalist societies. Visually and inter-personally, the fascist cultural project communicated a machine-like certainty about a vision for (...) a new society based on discipline, conformity and the might of the nation. In this article, we are interested in the ways that these very same discourses are also communicated through sound and music in two songs: The Horst Wessel Lied and the BUF marching song, two songs that used the same melody. We analyse the discourses communicated by the semiotic choices made in melody, arrangements, sound qualities, rhythms as well as in lyrics. The article first identifies some of the underlying semiotic resources for meaning making in sound and then shows how these are used in order to communicate specific ideas, values and attitudes. (shrink)

This study examined the disability support offices websites of twelve US higher education institutions anchored in multimodal discourse analysis and genre analysis to examine how semiotic resources are deployed to describe DSO services on their websites and to determine the discursive functions of advertisement they perform. The DSO websites were within four clicks from HEI homepages but had inconsistent navigation paths, making it difficult to reach DSO websites. DSO websites were foregrounding promoting and branding the institutions rather than (...) presenting the information about the services offered. This is achieved by using multimodal promotional rhetoric such as: situating accessibility as central commodifiable attribute, promoting the value of accessibility, establishing the superiority of the institution, constructing images of students with disabilities as empowered but dependent upon the DSO, and situating students within a college community. Implications for DSO websites functioning as advertisements are also discussed. (shrink)

The article presents an analysis of the ways in which knowledge is displayed, contested and renegotiated in the 2007 French presidential debate between Ségolène Royal and Nicolas Sarkozy. Knowledge displays can be achieved through a series of ‘neutral’ resources, such as informing, explanation or comment, or through face-damaging resources, such as questioning an unknowledgeable interlocutor to prove his inferior epistemic status and boost one’s own. The article focuses on this latter type of knowledge display where a knowledgeable participant (...) engages in question–answer sequences with an unknowledgeable respondent in front of a third party. The article also undertakes an analysis of the multimodal strategies employed by the participant to discredit the opponent. The article intends to contribute to the existent literature on epistemic stance by offering a prototypical example of incongruence between the epistemic status of the questioner and the epistemic stance he adopts. (shrink)

Combining textual, visual, and ethnographic approaches to discourse, this article examines a variety of resources employed in the narrative construction of Washington, DC’s Chinatown in a billboard advertisement that de-ethnicizes the neighborhood. Analysis of the linguistic resources of narrative structure, comparative reference, and lexical cohesion reveals how the gentrification of Chinatown is constructed as a positive transformation driven by a corporation. Further, the visual juxtaposition of text with photos and graphics appropriates the community voice and infuses it with (...) corporate identity. This ideological multimodal construction of the transformation of Chinatown is finally actualized in its durable material form and strategic spatial emplacement. Incorporating ethnographic observation and an interview, this article illustrates how the symbolic power of narrative in place-making is interdependent on the economic power of its producer to propagate ideological discourse in the material world. (shrink)

The article examines the role of discourse in organisational sensemaking. By building links between the theorising undertaken within organisational studies and the empirical analysis of multimodal social interaction, it argues for a relational view of sensemaking and investigates how sense is made in and through social interaction in real organisational situations where language use intertwines with embodied actions and the manipulation of artefacts. In particular, the article studies the use of discourse technologies of textual artefacts in sensemaking processes. The (...) data come from training workshops of a Finnish workplace organisation, conducted in order for the employees to delineate the history and future of their organisation with the help of writable papers. The results show how the papers exert agency in the situation by facilitating three specific discursive practices and by enabling and restricting the actions employed in constructing a shared understanding of the organisational reality. (shrink)

In his PhD proposal, now published as Seeing Sociological, Garfinkel [2006] formulated action in terms of a mutually constitutive structure—the Noesis-Noema Structures. This structure can be traced to Aaron Gurwitsch’s gestalt psychology and Law of Good Gestalt which theorises how participants prioritise functional Gestalts over other possible meanings of what is perceivable in their surroundings. While Gurwitsch illustrated his theory using images, in this paper we revisit Gurwitsch’s theory in light of the advances in recording real-time interaction to consider Gestalt (...) in spatio-temporality of real-time interaction. We consider the Law of Good Gestalt in terms of the dimensions of time and space, and postulate two analytical principles—the Principle of Good Momentary Gestalt and the Principle of Good Temporal Gestalt—for analysing a multi-angle video segment of a monologue taken from a training event. The analysis examines how the monologue was embedded in a multi-layer projection structure, so that during the time of the monologue, the trainer and trainees can be seen as achieving a transition between one activity to another while sustaining the frame of the training event. Through this, the analysis highlights the multi-layered structure of participants’ field of perception that constitutes their experience of the social activity, and explores a “method” to reconstruct such a structured field of perception through re-coupling meanings to the assemblages of multimodal resources recoverable on video. (shrink)

This paper presents an analysis of the use of humorous stickers within a WhatsApp chat group. With the rapid growth of digital communication, particularly on mobile devices, WhatsApp has become a popular platform for various forms of interaction, including voice and video calls, as well as text and voice messaging. In this context, humour is a prevalent phenomenon, especially in groups where participants have established familiarity and trust. The prevalence of humour in digital communication is largely attributed to the use (...) of multimodal resources, such as stickers – digital images that can include text and are used to convey emotions, feelings, and humour. The study’s initial hypothesis posits that stickers serve as a means of humour contextualization, helping to emphasize and enhance the intended humour in messages. Additionally, stickers are believed to aid in reducing misunderstandings during communication and mitigating potential threats to the participants’ social image. For this case study, the researchers employed a qualitative approach to analyse conversations within a Spanish WhatsApp group comprising numerous participants, resulting in a corpus comprising 44,362 words and 273 stickers. From this data, they selected 8 humorous conversational sequences that exemplify the primary uses and functions of stickers in this digital context. (shrink)

ABSTRACT This article aims to analyze the process of mutual understanding from the perspective of Discursive Neurolinguistics. As a methodology, a case study was carried out on a child with a linguistic profile of verbal restriction, which concerns the dimension of utterances involving words and their morphology. Data interpretation was based on the Bakhtinian perspective, which addresses ‘understanding’ as an active-responsive process and meaning as emerging from the relationship between utterances. The results showed that the interlocutors’ common ground allowed the (...) joint construction of meaning even in the face of verbal restrictions, and aspects such as nonverbal elements and the recognition of the child’s active-responsive role also favored mutual understanding. As a conclusion, the attention to different multimodal resources as legitimate modes of expression allows children with verbal restrictions to fulfill the role of “speakers” regardless of their modes of expression. (shrink)

The paper contributes to previous studies of identity as locally and interactionally produced by pointing to some of the multimodal resources employed by participants to achieving, challenge and manage the professional identity of ‘leader’ in different workplace settings. We examine professional identity work in sequential environments where it provides a resource for handling the resistance to improvement displayed by another participant. We show how leader identity work gets embedded within a four-part structure of: identifying a problem, proposing improvements, (...) misaligning with others’ proposals, and managing the misalignment. The paper is based on video ethnographic fieldwork and recordings of face-to-face institutional interaction at two different institutional settings in Denmark. The study provides insight about the role of identity work in the interactional achievement of institutional procedures. (shrink)

In general, understanding requires cognitive and linguistic skills, encompasses cultural, social, contextual and individual aspects, and is characterised by gradualness and dynamicity. In this study, the intertwined set of relevant components involved in the complex process of understanding space deixis will be analysed in the specific context of face-to-face interaction. In everyday conversation, this process is unavoidably mutual and may include misunderstanding (which often opens up a way to understanding), repairs, reformulations and negotiation cycles, all of which eventually lead to (...) understanding and mutual comprehension or to communication failure. Many scholars have already pointed out several elements that influence interactants’ understanding in a positive or negative way: on the one hand, shared physical context, cotext, common ground, shared knowledge and avoidance devices; on the other, ambiguity, speakers’ failures, and interactants’ asymmetric features with regard to language and culture. The interactional context, the cotext and the use of multimodal resources (resorted to as co-speech or to substitute language) play a significant role in the dynamic process of understanding spatial deixis, often helping to overcome the difficulties related to peripheral cases of spatial deictics and other causes of misunderstandings, which can either work as a starting point for understanding or lead to communication failure. (shrink)

This article delves into two intrinsic tensions present in humor and cartoons: funniness-seriousness and repetition-novelty. The focus is on the perspectives of cartoonists regarding the semiotic and rhetorical resources they put into play to create humor and satire. In this way, the study echoes a key question in semiotics, and particularly in multimodal studies: how image and words interplay in meaning-making. With that aim, a questionnaire was designed and filled out by 100 cartoonists from 22 different countries. The (...) responses were analyzed applying mixed methods (content analysis, non-parametric statistical techniques, and network analysis). Based on the findings, four distinct profiles of cartoonists were identified, ranging from those who prioritize multimodality (the largest group) to those who favor the verbal mode. Cartoonists who prioritize the visual mode and skeptics to humor analysis fall in between. The article also discusses the various rhetorical resources that cartoonists declare to use to create humor, with irony and sarcasm being the most frequent ones, while puns are the least common. The results are examined in relation to the cartoonists’ conceptions of humor, satire, and the cartoon genre, as well as the different dimensions they considered when discussing their humor style. The study contributes to gaining a deeper understanding of the ways cartoonists use image and writing to create humor and satire and sheds light on an under-researched area in this field. (shrink)

A variety of semiotic resources makes the construction of scientific knowledge possible and meaning-making resources are conveyed by certain semiotic modes. Next, numerous studies have demonstrated the pedagogical importance of gestures in the demonstration of scientific knowledge in the classroom. Drawing on social semiotic systemic functional theory and legitimation code theory, this study explores the types and the role of gestures in the semiotic construction of scientific knowledge in pedagogic semiosis and their pedagogical values for the meaning-making of (...) science content. With this aim, a gesture observation protocol (GOP) is developed to annotate and analyze data. Two science lectures, which are broadcast remotely for middle school students including gifted students, are observed. Systemic Functional Multimodal Discourse Analysis (SF-MDA) method is employed to analyze and evaluate data. Results demonstrated that the deployment of gestures that contributes to the construction of scientific knowledge is low and deployed gestures mostly do not have pedagogical value. It is recommended that teachers should be more conscious and educated about deploying gestures to present scientific knowledge efficiently. (shrink)

Using multimodal conversation analysis, we investigate how novices learning the “inner body” acting technique in the context of a community theater project share their experiences of the bodily exercises through verbal and embodied conduct. We focus on how verbal description and bodily enactment of the experience mutually elaborate each other, and how the experienced sensorimotor and affective qualities are made to be witnessed and recognized by the others. Participants describe their experiences without naming qualities. Instead, a display of the (...) experienced qualities is made accessible to others through coordinating the unfolding talk and bodily conduct. In particular, we show how grammatical and action projection is fulfilled by interconnected verbal and embodied conduct, with body movement and posture giving off ineffable experiential qualities. The moving body appears both as a source of the experience and as a resource for depicting perceived qualities to others; additional resources (non-specific person reference and gaze aversion) contribute to organizing the subjective and intersubjective layers of the reflection of the experiences. The study contributes to and extends recent research on sensoriality in interaction by focusing on phenomena of proprioception and interoception. The data are two cases drawn from 60 h of video-recordings made in the context of a devised community theater project. The data are in Finnish with English translations. (shrink)

This study investigates moments in which one participant in an interaction embodies that he is “doing thinking,” a display that is commonly referred to as “thinking face. ” From an interactional perspective, it is assumed that embodied displays of “doing thinking” are a recurring social practice and serve interactive functions. While previous studies have examined thinking faces primarily in word searches and storytelling, the present study focuses on argumentative activities, in which children engage in processes of joint decision-making. The paper (...) has two interrelated aims. The first aim is to describe how multiple modalities—beyond the face—are temporally coordinated to create multimodal gestalts of “doing thinking.” It is shown that thinking displays not only involve dynamic imaginative gaze but also stylized bodily postures. The second aim is to generate knowledge about the functions of thinking displays in children's argumentative activities. The analysis describes how both speakers and recipients use thinking displays in different turn positions and align them with verbal talk or silence. The data for this study comprise video recordings of decision-making processes in groups of older children. Drawing on a multimodal approach to situated interaction, it will be proposed that embodied displays of “doing thinking” provide a resource to shape participation frameworks, mark epistemic stances and create epistemic ecologies for collaborative reasoning. By investigating thinking displays in a particular conversational activity, the study sheds light on the diversity and context-sensitive functionality of thinking displays. It also contributes to recent research on children's collaborative reasoning as an embodied discursive practice. (shrink)

Notre travail s’inscrit dans les ancrages théoriques de l’ethnométhodologie et de l’analyse conversationnelle. Notre intérêt est avant tout d’analyser les pratiques interactionnelles entre joueurs en situation de jeux vidéo ; et d’autre part, de comparer différents jeux sur le plan de leur organisation interactionnelle ainsi que leur structure séquentielle et multimodale. Á partir des pratiques de joueurs, nous questionnons les problèmes et contraintes rencontrés, d’une part lors des analyses de ces données en situation de jeux vidéo, et d’autre part lors (...) de présentations académiques de nos recherches. Une étude en cours, sur les séquences d’encouragement et d’instruction dans les interactions entre joueurs, nous permet de montrer qu’il est indispensable de contextualiser le type de jeu joué, ses règles, ses options spécifiques parfois, les relations entre les joueurs, les rôles de chacun à l’instant de l’extrait, et aussi les actions en cours dans l’univers du jeu lui-même qui donneraient du sens à l’analyse. Ces contextualisations et prises en compte des ressources multimodales permettent d’expliquer et de décrire plus clairement et objectivement le phénomène linguistique, l’action ou encore la problématique analysée. (shrink)

In efforts to achieve the Sustainable Development Goals, social entrepreneurship has gained popularity as a vehicle for positive change in developing countries. The multiplicity of stakeholders, diverging sociocultural contexts and the hybrid mission complicate the process of legitimacy construction for social entrepreneurs as a basis for the acquisition of scarce resources. This study investigates how social entrepreneurs operating in Sub-Saharan Africa and Asia tackle this challenge of bridging conflicting directions in discursive interaction with their European funders. We conduct a (...) multimodal discourse analysis to uncover discursive strategies for legitimacy construction by combining linguistic data from interviews with visual data from social media accounts. Legitimacy construction, and thus resource acquisition, centers on three aspects which interlink pragmatic, cognitive and moral legitimacy: developing innovative solutions, mobilizing private-sector efficiency, and contributing to local empowerment. Presenting these aspects as mutually reinforcing overcomes contradictions between social and business logics and provides an expanded space for legitimacy construction of social entrepreneurship. Discursively constructing legitimacy around a ‘glorified version of social entrepreneurship’ mobilizes resources but downplays the risk of being an entrepreneur in the Global South, contributing to increasing corporatization of social purpose organizations. (shrink)

This paper provides a critical investigation into the discourse of egg freezing on Xiaohongshu, a prominent social media platform in China, to uncover the complexities of female identity, reproductive autonomy, and consumer narratives within this context. Employing Multimodal Critical Discourse Analysis (MCDA), it examines the discursive construction of self by single Chinese women who share their egg freezing experiences online. This study scrutinizes the semiotic resources utilized in these narratives, including textual and visual elements, to reveal how individuals (...) craft their identities in relation to emerging reproductive technologies. The analysis identifies dominant themes in the discourse, such as ‘negotiation of motherhood’, ‘women’s empowerment’, and ‘technologization of egg freezing’, juxtaposing these against the backdrop of China’s market socialism and shifting social attitudes. By engaging with the posts of Xiaohongshu bloggers, the research dissects how these women articulate their journeys, challenging traditional paradigms and contributing to ongoing debates about women’s rights and feminist ideologies in the context of advanced reproductive technologies. (shrink)

This paper proposes a multimodal ethnosemiotic conceptual framework for culturally themed selective Web archiving, taking as a practical example the curation of the London French Special Collection in the UK Web Archive. Its focus on a particular ‘community’ is presented as advantageous in overcoming the sheer scale of data available on the Web; yet, it is argued that these ethnographic boundaries may be flawed if they do not map onto the collective self-perception of the London French. The approach establishes (...) several theoretical meeting points between Pierre Bourdieu’s ethnography and Gunther Kress’s multimodal social semiotics, notably, the foregrounding of practice and the meaning-making potentialities of the everyday; the implications of language and categorisation; the interplay between subject and object; evolving notions of agency, authorship and audience; together with social engagement, and the archive as dynamic process and product. The curation rationale proposed stems from Bourdieu’s three-stage field analysis model, which places a strong emphasis on habitus, considered to be most accurately presented through blogs, yet necessitates its contextualisation within the broader field, through institutional websites, for example, whilst advocating a reflexive awareness of the researcher/curator’s role. This, alongside the Kressian acknowledgement of the inherent multimodality of on-line resources, lends itself convincingly to selection and valuation strategies, whilst the discussion of language, genre, authorship and audience is relevant to the potential cataloguing of Web objects. By conceptualising the culturally themed selective Web-archiving process within the ethnosemiotic framework constructed, concrete recommendations emerge regarding curation, classification and crowd-sourcing. (shrink)

This research investigates how multimodality is applied in logos to build a heritage brand identity for Chinese destinations. As a historical country, China is known for its huge reserves of cultural and natural heritage, which ideally offers abundant resources for developing its tourism industry. These can be taken advantage of in branding its destinations. Kress and van Leeuwen’s Visual Grammar was adopted in this study to conduct a qualitative analysis of the images and words of destination logos collected from (...) 34 provincial-level administrative regions in China. The findings indicate that pictorial and written elements are skillfully used in the logos through the manifestation of heritage, capturing features that convey significant connotations to promote a positive brand identity of the Chinese destinations. This research adds to the existing literature on destination branding through heritage commodification by embodying evocative multimodal features in destination logos, as well as promoting the semiotic approach as an effective tool in constructing the heritage brand identity of Chinese destinations. (shrink)

Drawing on Arabic data sets, this study examines reported speech in naturally occurring conversations. Building on earlier work in discourse analysis, the study demonstrates how reported speech is a multiparty social field in which much of the reporting involves not only speech but also intricate forms of voice patterns and embodied reenactments. The study argues that speakers create an integrated complex of reporting, including multimodal utterances that go beyond the stream of speech to include relevant nonlinguistic sounds and embodied (...) gestures. Analysis also shows that participants engage in verbal and nonverbal forms of stance toward the reported activities. The speaker makes certain these stance displays are available to the addressees to achieve co-participation. In contrast to studies which restrict analysis to linguistic phenomena, this study approaches reported speech as a fine performance in which different kinds of semiotic resources are brought together. (shrink)