This paper provides a categorization of cross-modal experiences. There are myriad forms. Doing so allows us to think clearly about the nature of different cross-modal experiences and allows us to clearly formulate competing hypotheses about the kind of experiences involved in different cross-modal phenomena.

For more than a century now, researchers have acknowledged the existence of seemingly arbitrary crossmodal congruency effects between dimensions of sensory stimuli in the general population. Such phenomena, known by a variety of terms including 'crossmodal correspondences', involve individual stimulus properties, rely on a crossmodal mapping of unisensory features, and appear to be shared by the majority of individuals. In other words, members of the general population share underlying preferences for specific pairings across the senses. Crossmodal correspondences between complementary (...) sensory cues have often been referred to as synesthetic correspondences but, we would argue, differ from full-blown synesthetic experiences in a number of important ways, including the fact that there are no idiosyncratic concurrent sensations. Recent psychophysical evidence suggests that such crossmodal correspondences can modulate multisensory integration by helping to resolve the crossmodal binding problem. Here, we propose a model to account for the effects of crossmodal correspondences between complementary auditory and visual cues and critically review their relation to full-blown synesthesia. (shrink)

The representational theory of phenomenal experience is often stated in terms of a supervenience thesis: Byrne recently characterises it as the thesis that “there can be no difference in phenomenal character without a difference in content”, while according to Tye, “[a]t a minimum, the thesis is one of supervenience: necessarily, experiences that are alike in their representational contents are alike in their phenomenal character.” Consequently, much of the debate over whether representationalism is true centres on purported counter-examples – that is (...) to say, purported failures of supervenience. The refutation of putative counter-examples has been, it seems to me, by and large successful. But there is a certain class of these for which the representationalist response has been something less than completely convincing. These are the cross-modality cases. I will explain what I mean, and then argue that the response in question is not only unconvincing but actually undermines the representationalist position. (shrink)

Forensic authorship analysis is based on two assumptions: that every individual has a unique idiolect, and that features characteristic of that idiolect will recur with a relatively stable frequency. Yet, a speaker’s language can change with age, affective states, according to audience, or genre. Thus, studies on authorship analysis should adopt the theory that while some linguistic parameters of an idiolect can remain stable, others can change depending on various circumstances. This investigation, which takes a constructional and functional-based approach to (...) discourse-level phenomena in idiolectal stability, analyzes cross-genre data produced by nine Mexican participants throughout a twelve-year time span. This study contributes to a greater understanding of the linguistic elements that survive genre effects and are potentially useful in both investigative and evidential forensic linguistic work. We provide a detailed description of linguistic features, their specific values, and context-dependent interpretation, keeping in mind the context of expert linguistic testimony, with its preference for methods which “employ linguistically motivated analyses in combination with quantitative tools” (Solan & Tiersma, 2004, p.463). Our findings show that idiolectal style tends to remain stable across genres and communication modes in epistemic modality constructions. Epistemic markers —specifically, markers indicating low commitment by the speaker (e.g., _no sé_ ‘I don’t know’) or expressing indirectness when introducing the illocutionary force (e.g., _la verdad [es que]_ ‘the truth [is that]’)— display idiolectal stability, as these markers seem to be the most effective in terms of allowing speakers to strategically manifest the extent of their knowledge regarding what is said. (shrink)

The loss or absence of vision is probably one of the most incapacitating events that can befall a human being. The importance of vision for humans is also reflected in brain anatomy as approximately one third of the human brain is devoted to vision. It is therefore unsurprising that throughout history many attempts have been undertaken to develop devices aiming at substituting for a missing visual capacity. In this review, we present two concepts that have been prevalent over the last (...) two decades. The first concept is sensory substitution, which refers to the use of another sensory modality to perform a task that is normally primarily sub-served by the lost sense. The second concept is cross-modal plasticity, which occurs when loss of input in one sensory modality leads to reorganization in brain representation of other sensory modalities. Both phenomena are training-dependent. We also briefly describe the history of blindness from ancient times to modernity, and then proceed to address themeansthat have been used to help blind individuals, with an emphasis on modern technologies, invasive (various type of surgical implants) and non-invasive devices. With the advent of brain imaging, it has become possible to peer into the neural substrates of sensory substitution and highlight the magnitude of the plastic processes that lead to a rewired brain. Finally, we will address the important question of the value and practicality of the available technologies and future directions. (shrink)

Philosophers and scientists have studied sensory perception and, in particular, vision for many years. Increasingly, however, they have become interested in the nonvisual senses in greater detail and the problem of individuating the senses in a more general way. The Aristotelian view is that there are only five external senses—smell, taste, hearing, touch, and vision. This has, by many counts, been extended to include internal senses, such as balance, proprioception, and kinesthesis; pain; and potentially other human and nonhuman senses. This (...) “multisensory turn” has been driven partly by developments in contemporary psychology and neuroscience, which have revealed a host of complex interrelations and interactions between sensory modalities previously thought to be distinct. Contrasts between modalities and other crossmodal phenomena, including multisensory integration, synesthesia, and sensory substitution, have also begun to receive more attention in a burgeoning scientific and philosophical literature on multisensory perception and other crossmodal effects. This article focuses on recent empirically informed contributions to the philosophy of perception, as well as key scientific works that provide important background information and insights into the nature of the senses and sensory perception. Indeed, one of the lessons of the multisensory turn, and of contemporary philosophy of mind more generally, is that philosophers ignore this body of empirical research at their peril because many human and animal senses turn out to be richer and more complex than philosophers and scientists had previously imagined, making this a fruitful area for interdisciplinary interaction and research. (shrink)

The paper proposes a novel analysis of quantificational subordination, e.g. Harvey courts a woman at every convention. {She is very pretty. vs. She always comes to the banquet with him.} (Karttunen 1976), in particular of the fact that the indefinite in the initial sentence can have wide or narrow scope, but the first discourse as a whole allows only for the wide scope reading, while the second discourse allows for both readings. The cross-sentential interaction between scope and anaphora is (...) captured in terms of structured anaphora to quantifier domains, formalized in a new dynamic system couched in classical type logic; given the underlying type logic, Montague-style compositionality at sub-clausal level follows automatically. Modal subordination (Roberts 1987) is analyzed in a parallel way, thereby capturing the parallels between the individual and modal domains argued for in Stone (1999). Several other phenomena are analyzed in terms of structured anaphora: exceptional wide scope, weak / strong donkey readings, anaphoric / uniqueness-implying definite descriptions and interactions between same / different and quantifier scope. (shrink)

In the present study, cross-modal (i.e., across sensory modalities such as smell and sound) iconicity (i.e., resemblance) and indexicality (i.e., contiguity) in lexical sensory and emotional signs in Finnish Sign Language will be considered from an articulatory perspective (i.e., the production of signs). Such cross-modal iconicity has not been extensively studied previously, so here, with the help of cognitive semiotics, I aim to carefully describe the cross-modal patterns observed across 118 signs, including 60 sensory (...) signs and 58 emotional signs. The analysis is framed within the theoretical model of Semiotic Hierarchy, which entails a non-reductionist view of meaning. In addition, a pheno-methodological triangulation will be applied: phenomenology (first-person method), literature of phenomenological and semiotic descriptions (second-person perspective) and experimental findings (third-person perspective). The results of this analysis show that (a) 71 of the 118 sensory and emotional signs are cross-modally indexical, (b) only 10 of the 71 signs can be regarded as cross-modally iconic, (c) cross-modal iconicity is highly diagrammatic, (d) iconicity and indexicality are highly integrated, and (e) articulatory feedback matters in the formation of semiotic patterns. This study contributes to our understanding of cross-modal iconicity in signed languages, as well as studies in semiotic systems more generally. (shrink)

The article explores cross-modal iconic relations in nine diverse Western-music songs ranging from 1600 to 2015, all of them thematizing dysphoric weeping. Initial input comes from five recurrent features observed in ancient Greek texts associated with performative events, including the prominence of sound, interjections and strong self-referentiality, repetitions and refrains, the motif of endlessness, and tears associated with streams of water, dew, and libation liquids. The analysis adopts Peirce’s conceptual distinction between image, diagram, and metaphor iconicity, although the (...) continuum reading proposed by several scholars is ultimately favored. The sample turns out to offer plenty of evidence of iconic relations. Interjections, falsetto, verbal repetitions, musical repetitions, musical rendering of sighs and endlessness, ostinato patterns, downward notes, and water images (rain and rivers), all of this is shown to substantiate cross-modal iconicity encompassing weeping, lyrics, music (including musical notation), and images, in different combinations. Quantitative investigations confirming/disconfirming and adding iconicity patterns, and comparative analyses linking Western and non-Western lament traditions constitute desiderata for future work. (shrink)

In this chapter we suggest that the origin of language, specifically the protolinguistic stage, was iconic rather than arbitrary, and fundamentally based on shared cross-modal associations. We provide evidence from natural language in the form of sound symbolism, distinguishing conventional sound symbolism from sensory sound symbolism. Sensory sound symbolism, or the presence of iconicity in natural language, is considered alongside psychological experiments in naming, and other investigations of cross-modal associations specifically involving linguistic sound. This evidence supports (...) the idea that we can directly express a variety of sensory experiences through linguistic sound, and thus that language systems have the capacity to be iconic to a large extent. We outline a theory of sensory protolanguage emergence, considering how and why arbitrariness eventually became the dominant relationship between form and meaning. Finally, we consider the possibility of continuity between cross-modal transfer in other primates and more abstract cross-modal associations among humans, suggesting this may have scaffolded the emergence of symbolic communication. (shrink)

Thepuzzle of cross‐modal shape experienceis the puzzle of reconciling the apparent differences between our visual and haptic experiences of shape with their apparent similarities. This paper proposes that we can resolve the cross‐modal puzzle by reflecting on another puzzle. Thepuzzle of perspectival characterchallenges us to reconcile the variability of shape experience through shifts in perspective with its constancy. An attractive approach to the latter puzzle holds that shape experience is complex, involving bothperspectivalaspects andconstantaspects. I argue here (...) that parallel distinctions between perspectival and constant aspects of shape experience arise in sight and touch, and that perspectival aspects are modality‐specific while at least some constant aspects are constitutively multisensory. I then address a powerful challenge to the idea that aspects of spatial phenomenology are shared cross‐modally. (shrink)

Experiencing a stimulus in one sensory modality is often associated with an experience in another sensory modality. For instance, seeing a lemon might produce a sensation of sourness. This might indicate some kind of cross-modal correspondence between vision and gustation. The aim of the current study was to provide explore whether such cross-modal correspondences influence cross-modal integration during perceptual learning. To that end, we conducted 2 experiments. Using a speeded classification task, Experiment 1 established (...) a cross-modal correspondence between visual lightness and the frequency of an auditory tone. Using a short-term priming procedure, Experiment 2 showed that manipulation of such cross-modal correspondences led to the creation of a crossmodal unit regardless of the nature of the correspondence (i.e., congruent, Experiment 2a or incongruent, Experiment 2b). However, a comparison of priming-effects sizes suggested that cross-modal correspondences modulate cross-modal integration during learning and thus leading to new learned units that have different stability over time. We discuss the implications of our results for the relation between cross-modal correspondence and perceptual learning in the context of a Bayesian explanation of cross-modal correspondences. (shrink)

Three priming experiments were conducted to determine how information about the self from different sensory modalities/cognitive domains affects self-face recognition. Being exposed to your body odor, seeing your name, and hearing your name all facilitated self-face recognition in a reaction time task. No similar cross-modal facilitation was found among stimuli from familiar or novel individuals. The finding of a left-hand advantage for self-face recognition was replicated when no primes were presented. These data, along with other recent results suggest (...) the brain processes/represents information about the self in highly integrated ways. (shrink)

The importance of learning and categorizing social objects and events has become widely acknowledged over a couple of decades. Although findings from field studies have suggested that non-human animals have sophisticated abilities to recognize social objects, there is relatively little experimental evidence on this issue. Some studies have revealed animals’ excellent skills for discriminating visual and auditory social stimuli. However, because of perceptual resemblances among stimuli, it is still not clear that they recognize these objects with conceptual mechanisms that are (...) independent of the perceptual characteristics of the stimuli. At the same time, whether their concepts have an aspect of transferring information from one modality to another has not received much attention. This paper advocates approaches to a cross-modal aspect of concepts as a new framework to solve these problems, and introduces the latest studies on cross-modal representations of social objects in non-humans. (shrink)

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory (...) task-switching paradigm over four sessions, to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks or not trained allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level. (shrink)

I argue that a class of recently-discovered cross-modal illusions gives reason to posit a dimension of content shared across perceptual modalities and to abandon the traditional view according to which perceptual content is exclusively constituted by discrete modality-specific contents.

Cross-modal perceptual illusions occur when a stimulus to one modality impacts perceptual experience associated with another modality. Unlike synaesthesia, cross-modal illusions are intelligible as results of perceptual strategies for dealing with sensory stimulation to multiple modalities, rather than as mere quirks. I argue that understanding cross-modal illusions reveals an important flaw in a widespread conception of the senses, and of their role in perceptual experience, according to which understanding perception and perceptual experience is a (...) matter of assembling independently viable stories about vision, audition, olfaction, and the rest. (shrink)

Fodor claims that cognitive modules can be thought of as constituting a psychological natural kind in virtue of their possession of most or all of nine specified properties. The challenge to this considered here comes from synaesthesia. Synaesthesia is a type of cross-modal association: input to one sensory modality reliably generates an additional sensory output that is usually generated by the input to a distinct sensory modality. The most common form of synaesthesia manifests Fodor's nine specified properties of (...) modularity, and hence, according to Segal (1997), it should be understood as involving an extra module. Many psychologists believe that synaesthesia involves a breakdown in modularity. After outlining how both theories can explain the manifestation of the nine alleged properties of modularity in synaesthesia, I discuss the two concepts of function which initially motivate the respective theories. I argue that only a teleological concept of function is properly able to adjudicate between the two theories. The upshot is a further application of so-called externalist considerations to mental phenomena. (shrink)

Early modern humans developed mental capabilities that were immeasurably greater than those of non-human primates. We see this in the rapid innovation in tool making, the development of complex language, and the creation of sophisticated art forms, none of which we find in our closest relatives. While we can readily observe the results of this high-order cognitive capacity, it is difficult to see how it could have developed. We take up the topic of cave art and archeoacoustics, particularly the discovery (...) that cave art is often closely connected to the acoustic properties of the cave chambers in which it is found. Apparently, early modern humans were able to detect the way sound reverberated in these chambers, and they painted artwork on surfaces that were acoustic “hot spots,” i.e., suitable for generating echoes. We argue that cave art is a form of cross-modality information transfer, in which acoustic signals are transformed into symbolic visual representations. This form of information transfer across modalities is an instance of how the symbolic mind of early modern humans was taking shape into concrete, externalized language. We also suggest that the earliest rock art found in Africa may constitute one of the first fossilized proxies for the expression of full-fledged human linguistic behavior. (shrink)

It is being increasingly recognized that the Saussurean dictum of “the arbitrariness of the linguistic sign” is in conflict with the pervasiveness of the phenomenon commonly known as “sound symbolism”. After first presenting a historical overview of the debate, however, we conclude that both positions have been exaggerated, and that an adequate explanation of sound symbolism is still lacking. How can there, for example, be (perceived) similarity between expressionsand contents across different sensory modalities? We offer an answer, based on the (...) Peircian notion of iconic ground, and G. Sonesson’s distinction betweenprimary and secondary iconicity. Furthermore, we describe an experimental study, in a paradigm first pioneered by W. Kohler, and recently popularized by V.Ramachandran, in which we varied vowels and consonants in fictive word-forms, and conclude that both types of sounds play a role in perceiving an iconic ground between the word-forms and visual figures. The combination of historical conceptual analysis, semiotic explication and psychological experimentationpresented in this article is characteristic of the emerging paradigm of cognitive semiotics. (shrink)

The thesis of this paper is that the capacity to think of one’s perceptions as cross-modally integrated is incompatible with a reductionist account of the self. In §2 I distinguish three versions of the argument from cross-modality. According to the ‘unification’ version of the argument, what needs to be explained is one’s capacity to identify an object touched as the same as an object simultaneously seen. According to the ‘recognition’ version, what needs to be explained is one’s capacity, (...) having once seen an object, to reidentify that same object by touch alone. According to the ‘objectivity’ version, what needs to be explained is one’s capacity to think of one’s perceptions in different modalities as perceptions of one and the same object. The third version seems to establish that one must conceive of oneself substantially, as the numerically identical owner of one’s experiences, a conclusion in agreement with recent work in developmental psychology claiming to show that an infant’s cross-modal capacities are essentially implicated in their development of a sense of self. There is further work to be done if this is to be tumed into an argument against reductionism: there is no swift route from the epistemology of self-consciousness to the metaphysics of the self. In the §3, I will claim that there is, nevertheless, an argument linking the two. What I propose is an argument derived, not from the token-reflexive rule for the first-person, but resting on its anaphoric behaviour spanning intensional operators. (shrink)

In a variety of synaesthesia, photisms result from affect-laden stimuli as emotional words, or faces of familiar people. For R, who participated in this study, the sight of a familiar person triggers a mental image of "a human silhouette filled with colour". Subjective descriptions of synaesthetic experiences induced by the visual perception of people's figures and faces show similarities with the reports of those who claim to possess the ability to see the aura. It has been proposed that the purported (...) auric perception may be easily explained by the presence of a specific subtype of cross-modal perception. We analyse the subjective reports of four synaesthetes who experience colours in response to human faces and figures. These reports are compared with descriptions of alleged auric phenomena found in the literature and with claims made by experts in esoteric spheres. The discrepancies found suggest that both phenomena are phenomenologically and behaviourally dissimilar. (shrink)

Recent studies have provided evidence of associations between neurochemistry and reading (dis)ability (Pugh et al., 2014). Based on a long history of studies indicating that fluent reading entails the automatic convergence of the written and spoken forms of language and our recently proposed Neural Noise Hypothesis (Hancock et al., 2017), we hypothesized that individual differences in cross-modal integration would mediate, at least partially, the relationship between neurochemical concentrations and reading. Cross-modal integration was measured in 231 children (...) using a two-alternative forced choice cross-modal matching task with three language conditions (letters, words, and pseudowords) and two levels of difficulty within each language condition. Neurometabolite concentrations of Choline (Cho), Glutamate (Glu), gamma-Aminobutyric (GABA), and N- acetyl-aspartate (NAA) were then measured in a subset of this sample (n = 70) with Magnetic Resonance Spectroscopy (MRS). A structural equation mediation model revealed that the effect of cross-modal word matching mediated the relationship between increased Glu (which has been proposed to be an index of neural noise) and poorer reading ability. In addition, the effect of cross-modal word matching fully mediated a relationship between increased Cho and poorer reading ability. Multilevel mixed effects models confirmed that lower Cho predicted faster cross-modal matching reaction time, specifically in the hard word condition. These Cho... (shrink)

Discrepancies between an agent's goals and beliefs play an important, if implicit, role in determining what a rational agent is motivated to do. This is most obvious in cases where an agent achieves a complex goal incrementally and must deliberate anew as each milestone is reached. In such cases the concept of goal/belief discrepancy defines an appropriate space to which a degree-of-achievement yardstick can be applied. This paper presents soundness and completeness results concerning a logic for reasoning about goal/belief discrepancy, (...) and it is suggested that a certain species of goal/belief discrepancy captures the concept of desire. (shrink)