This chapter identifies and analyzes ethical issues in video surveillance, with a focus on applications that involve video analytics and facial recognition technology. The first part of the analysis focuses on ethical issues in video analytics, which are techniques that automatically analyze preexisting or gathered video data. Ethical issues are discussed in relation to processes of identification, classification, detection, and tracking of persons. Next, facial recognition technology is discussed, particularly its use in surveillance, and its ethical (...) challenges are investigated. Finally, we focus on three advanced video analytics techniques: facial analysis, behavior analysis, and emotional analysis, which raise additional serious ethical issues. (shrink)

Facial recognition is an artificial intelligence-based technology that, like many other forms of artificial intelligence, suffers from an accuracy deficit. This paper focuses on one particular use of facial recognition, namely identification, both as authentication and as recognition. Despite technological advances, facial recognition technology can still produce erroneous identifications. This paper addresses algorithmic identification failures from an upstream perspective by identifying the main causes of misidentifications (in particular, the probabilistic character of this technology, (...) its ‘black box’ nature and its algorithmic bias) and from a downstream perspective, highlighting the possible legal consequences of such failures in various scenarios (namely liability lawsuits). In addition to presenting the causes and effects of such errors, the paper also presents measures that can be deployed to reduce errors and avoid liabilities. (shrink)

The use of facial recognition technology (FRT) as a form of intelligence has recently made a prominent public appearance in the theater of war. During the early months of Russia's invasion of Ukraine, Ukrainian authorities relied on FRT as part of the country's defensive activities, harnessing the technology for a variety of purposes, such as unveiling covert Russian agents operating amid the Ukrainian population; revealing the identity of Russian soldiers who committed war crimes; and even identifying dead Russian (...) soldiers. This constellation of uses of FRT—in a war increasingly waged on the digital and information front—warrants ethical examination. The essay discusses some of the most serious concerns with FRT in the context of war, including the infringement of informational privacy; the indiscriminate and disproportionate harms it may inflict, particularly when the technology is coupled with social media intelligence; and the potential abuse of the technology once the fog of war dissipates. Some of these concerns parallel those to be found in nations that are not engulfed in war, but others are unique to war-torn settings. (shrink)

Although the prevalence of facial recognition-based COVID-19 surveillance tools and techniques, China does not have a facial recognition law to protect its residents’ facial data. Oftentimes, neither the public nor the government knows where people’s facial images are stored, how they have been used, who might use or misuse them, and to what extent. This reality is alarming, particularly factoring in the wide range of unintended consequences already caused by good-intentioned measures and mandates amid (...) the pandemic. Biometric data are matters of personal rights and national security. In light of worrisome technologies such as deep-fake pornography, the protection of biometric data is also central to the protection of the dignity of the citizens and the government, if not the industry as well. This paper discusses the urgent need for the Chinese government to establish rigorous and timely facial recognition laws to protect the public’s privacy, security, and dignity amid COVID-19 and beyond. (shrink)

Facial recognition is a promising emerging technology, but it sometimes fails to recognize people adequately. Facial recognition applications have been found to regularly misidentify certain demographics, misinterpret traits like gender, age, beliefs, or emotions, and categorize individuals in ways that do not resonate with their own sense of identity. In this paper, I argue that in each of these cases, the person who has their face analyzed is not merely misidentified or misunderstood, but misrecognized in an (...) ethically relevant sense. Following Charles Taylor’s The Politics of Recognition (1992) and Axel Honneth’s The Struggle for Recognition (1996), I describe how those subjected to facial recognition systems on the one hand struggle to obtain adequate recognition on a universal level, as being equally important to others, and lack recognition for their individual uniqueness, on the other hand. These forms of misrecognition are of ethical concern, because they can harm a person’s identity formation. So, ironically, facial recognition technology can give rise to a struggle for recognition. (shrink)

Automated facial recognition methods have become widely used as a way to ascertain the identity of individuals. Yet the methods by which facial recognition technologies (FRT) operate – the machinic performance of the perception of the human face – are often invisible to those under their gaze. This article investigates the machinic perception of the face through an FRT method known as eigenface, in order to both reveal and problematize the ways of seeing that underlie it. (...) As part of its algorithmic processes, eigenface produces an image. This image can be understood as a portrait of machine recognition, making visible the processes through which the algorithm performs recognition and ‘sees’ the human face. The eigenface portrait reveals a way of seeing that is based on statistical processes of pattern recognition. An analogue antecedent of this application of statistics to the recognition of facial images can be found in the composite portrait. Through a dialectical discussion of composite portraiture in multiple disciplinary fields ranging from sociology to philosophy and the visual arts, this article experiments with providing a cultural and social translation of machine processes of visual perception. The discussion shifts the focus of enquiry towards the aesthetics of the algorithmic process in order to provide an entry point for critique and a possible reimagination on algorithmic knowledge production. (shrink)

I would like to introduce the notion of machine hermeneutics in this paper. The notion refers to hermeneutical activity performed by machines. Machines are now capable of making the very interpretive tasks, using artificial intelligence algorithms based on the technology of machine learning that used to be the exclusive domain of human beings. In making this claim, I am not talking about possible conscious machines of the future, but those existing here and now. With facial recognition algorithms, for (...) example, machines are now performing routinely what must be regarded as hermeneutical analyses with astounding accuracy and power. Thus, machine hermeneutics supplements Don Ihde’s notion of material hermeneutics. In the latter, it is still human beings who do the interpretation, through the lenses provided the natural sciences; in this case, the natural sciences, or the technology afforded by the sciences, intervene between the human being and the world. In machine hermeneutics, on the contrary, the intervening comes in two layers. On the one hand, there is the usual intervention that Ihde talks about, but on the other, the artificial intelligence algorithm performs its own kind of intervention and interpretation, presenting an already interpreted result to the human beings, who then perceive it through the aid of the usual intervention such as the normal eyeglasses. Then the paper discusses the problem of how to justify the kind of perception that undergoes this process. In what sense can it be said that the algorithm is performing the right action, i.e., one such that the process comes up with a right picture of the world? I contend that this does not merely consist of technical excellence for the technology involved, but also ethical excellence. The two cannot be considered one apart from the other. (shrink)

Advances in technology have transformed and expanded the ways in which policing is run. One new manifestation is the mass acquisition and processing of private facial images via automatic facial recognition by the police: what we conceptualise as AFR-based policing. However, there is still a lack of clarity on the manner and extent to which this largely-unregulated technology is used by law enforcement agencies and on its impact on fundamental rights. Social understanding and involvement are still insufficient (...) in the context of AFR technologies, which in turn affects social trust in and legitimacy and effectiveness of intelligent governance. This article delineates the function creep of this new concept, identifying the individual and collective harms it engenders. A technological, contextual perspective of the function creep of AFR in policing will evidence the comprehensive creep of training datasets and learning algorithms, which have by-passed an ignorant public. We thus argue individual harms to dignity, privacy and autonomy, combine to constitute a form of cultural harm, impacting directly on individuals and society as a whole. While recognising the limitations of what the law can achieve, we conclude by considering options for redress and the creation of an enhanced regulatory and oversight framework model, or Code of Conduct, as a means of encouraging cultural change from prevailing police indifference to enforcing respect for the human rights violations potentially engaged. The imperative will be to strengthen the top-level design and technical support of AFR policing, imbuing it with the values implicit in the rule of law, democratisation and scientisation-to enhance public confidence and trust in AFR social governance, and to promote civilised social governance in AFR policing. (shrink)

Biometric facial recognition is an artificial intelligence technology involving the automated comparison of facial features, used by law enforcement to identify unknown suspects from photographs and closed circuit television. Its capability is expanding rapidly in association with artificial intelligence and has great potential to solve crime. However, it also carries significant privacy and other ethical implications that require law and regulation. This article examines the rise of biometric facial recognition, current applications and legal developments, and (...) conducts an ethical analysis of the issues that arise. Ethical principles are applied to mediate the potential conflicts in relation to this information technology that arise between security, on the one hand, and individual privacy and autonomy, and democratic accountability, on the other. These can be used to support appropriate law and regulation for the technology as it continues to develop. (shrink)

This paper is an attempt to present disclosive ethics as a framework for computer and information ethics – in line with the suggestions by Brey, but also in quite a different manner. The potential of such an approach is demonstrated through a disclosive analysis of facial recognition systems. The paper argues that the politics of information technology is a particularly powerful politics since information technology is an opaque technology – i.e. relatively closed to scrutiny. It presents the design (...) of technology as a process of closure in which design and use decisions become black-boxed and progressively enclosed in increasingly complex socio-technical networks. It further argues for a disclosive ethics that aims to disclose the nondisclosure of politics by claiming a place for ethics in every actual operation of power – as manifested in actual design and use decisions and practices. It also proposes that disclosive ethics would aim to trace and disclose the intentional and emerging enclosure of politics from the very minute technical detail through to social practices and complex social-technical networks. The paper then proceeds to do a disclosive analysis of facial recognition systems. This analysis discloses that seemingly trivial biases in recognition rates of FRSs can emerge as very significant political acts when these systems become used in practice. (shrink)

This essay examines ethical aspects of the use of facial recognition technology for surveillance purposes in public and semipublic areas, focusing particularly on the balance between security and privacy and civil liberties. As a case study, the FaceIt facial recognition engine of Identix Corporation will be analyzed, as well as its use in “Smart” video surveillance systems in city centers and airports. The ethical analysis will be based on a careful analysis of current facial (...) class='Hi'>recognition technology, of its use in Smart CCTV systems, and of the arguments used by proponents and opponents of such systems. It will be argued that Smart CCTV, which integrates video surveillance technology and biometric technology, faces ethical problems of error, function creep and privacy. In a concluding section on policy, it will be discussed whether such problems outweigh the security value of Smart CCTV in public places. (shrink)

This article examines how facial recognition technology reshapes the philosophical debate over the ethics of video surveillance. When video surveillance is augmented with facial recognition, the data collected is no longer anonymous, and the data can be aggregated to produce detailed psychological profiles. I argue that – as this non-anonymous data of people’s mundane activities is collected – unjust risks of harm are imposed upon individuals. In addition, this technology can be used to catalogue all who (...) publicly participate in political, religious, and socially stigmatised activities, and I argue that this would undermine central interests of liberal democracies. I examine the degree to which the interests of individuals and the societal interests of liberal democracies to maintain people’s obscurity while in public coincide with privacy interests, as popularly understood, and conclude that there is a practical need to articulate a novel right to obscurity to protect the interests of liberal democratic societies. (shrink)

One novel example and/or perspective in support of “Why the learning account fails” is the impressive ability of humans to recognize and memorize facial features and accurately and reliably connect those to related identities. Furthermore, neuroimaging analysis presents an example in support of the crucial role of standardization in the lack of adoption of ideography.

On February 13, 2020, the Toronto Police Services (TPS) issued a statement admitting that its members had used Clearview AI’s controversial facial recognition technology (FRT). The controversy sparked widespread outcry by the media, civil society, and community groups, and put pressure on policy-makers to address FRTs. Public consultations presented a key tool to contain the scandal in Toronto and across Canada. Drawing on media reports, policy documents, and expert interviews, we investigate four consultations held by the Toronto Police (...) Services Board (TPSB), the Office of the Privacy Commissioner (OPC), and the parliamentary Standing Committee on Access to Information, Privacy and Ethics (ETHI) to understand how public opinion and outrage translate into policy. We find that public consultations became a powerful closure mechanism in the policy-making toolbox, inhibiting rather than furthering democratic debate. Our findings show that consultations do not advance public literacy; that opportunities for public input are narrow; that timeframes are short; and that mechanisms for inclusion are limited. Even in the best-case circumstances, consultations are merely one of many factors in AI governance and seldom impact concrete policy outcomes in the cases studied here. (shrink)

The use of facial recognition systems in concerts provides a perfect pretext to semiotically discuss the role of the face in contemporary culture, identifying different strategies and axiologies (systems of values). In his visionary essay Bruits (“noises”) from 1977, the French thinker Jacques Attali establishes a close connection between music and power and locates it in the site of the collective unfolding of music: the concert hall. Following this hint, the article reconstructs the current debate on facial (...) recognition systems in the music business, with a particular focus on concerts and festivals. The different positions (of musicians, authorities, and professionals) are outlined (some are openly in favor, others openly against), key moments are identified (Beyoncé’s concert in Cardiff, 2017; Jacky Cheung’s in Nanchang, 2018; Taylor Swift’s at the Rose Bowl in Pasadena, 2018; the “banfacialrecognition.com” campaign launched in 2019; the case of Barbara Hart at Brandi Carlile’s show at the Madison Square Garden, 2022) and, finally, the issue is framed within the broader debate about the face as an indexical marker of presence, identity, and humanity (the contrast between Italian philosopher Giorgio Agamben and American sociologist and design theorist Benjamin Bratton is addressed). (shrink)

Explanations of Capgras delusion and prosopagnosia typically incorporate a dual-route approach to facial recognition in which a deficit in overt or covert processing in one condition is mirror-reversed in the other. Despite this double dissociation, experiences of either patient-group are often reported in the same way – as lacking a sense of familiarity toward familiar faces. In this paper, deficits in the facial processing of these patients are compared to other facial recognition pathologies, and their (...) experiential characteristics mapped onto the dual-route model in order to provide a less ambiguous link between facial processing and experiential content. The paper concludes that the experiential states of Capgras delusion, prosopagnosia, and related facial pathologies are quite distinct, and that this descriptive distinctiveness finds explanatory equivalence at the level of anatomical and functional disruption within the face recognition system. The role of skin conductance response as a measure of ‘familiarity’ is also clarified. (shrink)

Resumen El reconocimiento facial es una tecnología que se ha utilizado para el control de la población a fin de combatir los efectos de la pandemia por COVID-19. Sin embargo, esta tecnología se utiliza hace bastante tiempo en muchos ámbitos del quehacer económico, social y político. Este trabajo analiza el reconocimiento fa cial entendida como la captura de micro expresiones faciales mediante la cual se puede medir y evaluar emociones con el fin de vigilar movimiento, predecir com portamientos y (...) anticipar intenciones de compra y de voto. Nuestro objetivo es revisar los enfoques teóricos que sustentan esta tecnología que ha devenido en una nueva oportunidad de negocio, para proponer después algunas reflexiones éticas derivadas.Facial recognition is a technology that is used to control people in order to fight the impacts of COVID-19. However, this technology has been used formany years in many economical, social and political areas. This work provides an analysis of the new business based on technologies of facial micro-expressions recognition. By means of this procedure, the involved companies measure and evaluate emotions with the aim of monitoring the movement, predicting behaviors, intentions of purchase and vote, between other applications. Our purpose is to check the theoretical suppositions that sustain this business, to propose later some ethical derivative reflections. (shrink)

This paper discusses prominent examples of what we call “algorithmic anxiety” in artworks engaging with algorithms. In particular, we consider the ways in which artists such as Zach Blas, Adam Harvey and Sterling Crispin design artworks to consider and critique the algorithmic normativities that materialize in facial recognition technologies. Many of the artworks we consider center on the face, and use either camouflage technology or forms of masking to counter the surveillance effects of recognition technologies. Analyzing their (...) works, we argue they on the one hand reiterate and reify a modernist conception of the self when they conjure and imagination of Big Brother surveillance. Yet on the other hand, their emphasis on masks and on camouflage also moves beyond such more conventional critiques of algorithmic normativities, and invites reflection on ways of relating to technology beyond the affirmation of the liberal, privacy-obsessed self. In this way, and in particular by foregrounding the relational modalities of the mask and of camouflage, we argue academic observers of algorithmic recognition technologies can find inspiration in artistic algorithmic imaginaries. (shrink)

As more and more companies adopt artificial intelligence to increase the efficiency and effectiveness of their products and services, they expose themselves to ethical crises and potentially damaging public controversy associated with its use. Despite the prevalence of AI ethical problems, most companies are strategically unprepared to respond effectively to the public. This paper aims to advance our empirical understanding of company responses to AI ethical crises by focusing on the rise and fall of facial recognition technology. Specifically, (...) through a comparative case study of how four big technology companies responded to public outcry over their facial recognition programs, we not only demonstrated the unfolding and consequences of public controversies over this new technology, but also identified and described four major types of company responses—Deflection, Improvement, Validation, and Pre-emption. These findings pave the way for future research on the management of controversial technology and the ethics of AI. (shrink)

The police’s use of facial recognition technologies allows them to verify identification in real-time by mapping facial features into indicators that can be compared with other data stored in its database or in online social networks. Advances in facial recognition technologies have changed law enforcement agencies’ operations, improving their ability to identify suspects, investigate crimes, and deter criminal behavior. Most applications are used in tracking and identifying potential terrorists, searching for abducted and missing persons, and (...) security surveillance at airports, national borders, and large public gatherings. However, for facial recognition technologies to fulfill their potential, they must not only be adopted by the police, but the public must also support their routine use. Using a secondary data analysis of a public opinion survey conducted by the Pew Research Center among a representative sample of US residents (N = 5307), we investigated the factors associated with the public`s expectations about the positive and negative outcomes of the police’s adoption of facial recognition technologies. Our results show that public attitudes to FRT are balanced, indicating awareness of the public for potential advantages, but also disadvantages of police adoption of FRT. Privacy considerations and familiarity with the technology were found to be critical for the explanation of public attitudes expressing both positive and negative expectations from the police adoption of facial recognition technology. Our study contributes to the understanding of the factors associated with public attitudes police’s use of facial recognition technologies. (shrink)

This study examines how different forms of media use predict attitudes toward the development of facial recognition technology (FRT) and applications of it by law enforcement to identify criminal suspects, identify potential terrorists, and monitor public protests. The theoretical framework builds on theories of cultivation and genre-specific viewing to develop hypotheses and research questions. The analyses draw on original data from two nationally representative surveys of the U.S. public conducted in 2020, amid a series of controversies and protests (...) about policing and racial justice. The results demonstrate that overall television viewing and crime media viewing predicted support for multiple uses of FRT, while Fox News viewing predicted support for using FRT to monitor protests. The findings advance our understanding of public opinion toward the technology and its implications for policing, protests, and social justice. (shrink)

In this article I analyse the hidden genealogical link between portrait photography, used for criminological and psychiatric purposes, and contemporary systems of biometric identification of the human face. The aim is to highlight the shift between the emphasis on the importance of the human ‘expert eye’ in recognizing the face when talking about nineteenth-century photography and the use of computer technology that produces and reads digital facial images. In both cases, however, these are modes and variants of reducing and (...) flattening the human face; the face itself disappears under the onslaught of technologies of vision and mediation, becoming a mere data set. Special attention is devoted to the pose, the frontal view, which is the technological a priori of the empirical possibility of recognition, articulated through the history of visualization of the face from early portrait photography to facial recognition systems. Consequently, what we call the ‘face’ is a simulacra of individual the face presented to the apparatus: the ‘real’ face is transposed upon multiple technological layers (camera – plates – photographic surface), losing its characteristic features to be re-written according to specific techniques of measurement. (shrink)

Prior studies provide consistent evidence of deficits for psychopaths in processing verbal emotional material but are inconsistent regarding nonverbal emotional material. To examine whether psychopaths exhibit general versus specific deficits in nonverbal emotional processing, 34 psychopaths and 33 nonpsychopaths identified with Hare's (R. D. Hare, 1991) Psychopathy Checklist-Revised were asked to complete a facial affect recognition test. Slides of prototypic facial expressions were presented. Three hypotheses regarding hemispheric lateralization anomalies in psychopaths were also tested (right-hemisphere dysfunction, reduced (...) lateralization, and reversed lateralization). Psychopaths were less accurate than nonpsychopaths at classifying facial affect under conditions promoting reliance on right-hemisphere resources and displayed a specific deficit in classifying disgust. These findings demonstrate that psychopaths exhibit specific deficits in nonverbal emotional processing. (shrink)

Background: Clinical and experimental findings have suggested that insomnia is associated with altered emotion processing, such as facial emotion recognition and impairments in executive functions. However, the results still appear non-consensual and have recently been presented by a few number of studies. Accordingly, the aim of the present study was to investigate whether patients with Insomnia disorder will present alterations in recognition of facial emotions and that such alterations will be related to Executive Functions and that (...) Insomnia Disorder patients will exhibit a lower Executive Functions performance when compared to normal controls individuals. Method: A total of 11 individuals diagnosed with insomnia disorder and 15 control participants took part in the present cross-sectional study. They completed questionnaires covering socio-demographic data, a sleep and mood evaluation, and a cognitive assessment. Moreover, they underwent a computerized Facial Emotional Recognition test, consisting of facial emotion labeling and facial emotion matching. Results: Individuals with insomnia presented a lower recognition of facial expressions of fear and sadness in comparison to control volunteers. Perceptual Organization Index was found to be statistically significant lower in the insomnia group compared to the control group. There was a statistically significant association between facial recognition emotional and cognitive processes of in executive functions performance, especially in processes related to the inhibitory control, planning capacity, problem solving and cognitive flexibility in insomnia group. Conclusions: Insomnia disorder impairs the facial emotion recognition of the fear and sadness. Insomniacs with impairment in the facial emotion recognition had impaired performance of executive functions. There is a negative impact on processes related to cognitive-emotional capacity than on processes related to social-emotional recognition. Keywords: insomnia, facial emotional recognition, sleep deprivation, cognition, executive functions. (shrink)

A correction to this paper has been published: https://doi.org/10.1007/s00146-021-01236-7.

Patients with behavioral variant of frontotemporal dementia (FTD) have difficulties recognizing facial emotions, a deficit that may contribute to their impaired social skills. In three experiments, we investigated the FTD deficit in recognition of facial emotions, by comparing six patients with impaired social conduct, nine Alzheimer’s patients, and 10 age-matched healthy adults. Experiment 1 revealed that FTD patients were impaired in the recognition of negative facial emotions. Experiment 2 replicated these findings when participants had to (...) determine whether two faces were expressing the same or different emotions. Experiment 3 was a control study in which participants had to discriminate whether two faces were of the same sex. In this non-emotional processing task, both patient groups performed worse than normal participants, but FTD patients performed as well as Alzheimer’s patients. We conclude that FTD patients are impaired in the recognition of negative facial emotions. © 2005 Elsevier Ltd. All rights reserved. (shrink)

Face recognition is a computationally challenging classification task. Deep convolutional neural networks (DCNNs) are brain‐inspired algorithms that have recently reached human‐level performance in face and object recognition. However, it is not clear to what extent DCNNs generate a human‐like representation of face identity. We have recently revealed a subset of facial features that are used by humans for face recognition. This enables us now to ask whether DCNNs rely on the same facial information and whether (...) this human‐like representation depends on a system that is optimized for face identification. In the current study, we examined the representation of DCNNs of faces that differ in features that are critical or non‐critical for human face recognition. Our findings show that DCNNs optimized for face identification are tuned to the same facial features used by humans for face recognition. Sensitivity to these features was highly correlated with performance of the DCNN on a benchmark face recognition task. Moreover, sensitivity to these features and a view‐invariant face representation emerged at higher layers of a DCNN optimized for face recognition but not for object recognition. This finding parallels the division to a face and an object system in high‐level visual cortex. Taken together, these findings validate human perceptual models of face recognition, enable us to use DCNNs to test predictions about human face and object recognition as well as contribute to the interpretability of DCNNs. (shrink)

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition (FER). Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (...) (Azure Face API, Microsoft; Face++, Megvii Technology; FaceReader, Noldus Information Technology) with human emotion recognition (A) for standardized posed facial expressions (from prototypical inventories) and (B) for non-standardized acted facial expressions (extracted from emotional movie scenes). For the standardized images, all three systems classify basic emotions accurately (FaceReader is most accurate) and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions. (shrink)

This article discusses how research on the image-based analysis of facial images has informed this debate by demonstrating that a single representational system for facial identity and facial expression is not only computationally viable, but can simulate existing cognitive data demonstrating apparent dissociable processing of these two facial properties. It discusses the increasing number of cognitive studies that provide support for this view. Neuropsychological case studies of brain-injured patients and provide limited evidence for separate visual routes (...) processing facial identity and facial expression. It deals with neuroimaging studies addressing both facial identity and facial expression processing which shows that the fusiform gyrus contributes to the recognition of both facial properties. Research continues to reinforce the proposal that a common visual framework supports the representation of at least some aspects of facial identity and facial expression. (shrink)

Facial expression recognition plays a significant part in artificial intelligence and computer vision. However, most of facial expression recognition methods have not obtained satisfactory results based on low-level features. The existed methods used in facial expression recognition encountered the major issues of linear inseparability, large computational burden, and data redundancy. To obtain satisfactory results, we propose an innovative deep learning model using the kernel entropy component analysis network and directed acyclic graph support vector machine. (...) We use the KECANet in the feature extraction stage. In the stage of output, binary hashing and blockwise histograms are adopted. We sent the final output features to the DAGSVM classifier for expression recognition. We test the performance of our proposed method on three databases of CK+, JAFFE, and CMU Multi-PIE. According to the experiment results, the proposed method can learn high-level features and provide more recognition information in the stage of training, obtaining a higher recognition rate. (shrink)

Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are (...) more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition. (shrink)

To define the determinants of impaired facial emotion recognition (FER) in patients with mesial temporal lobe epilepsy associated with hippocampal sclerosis (MTLE-HS), we examined 76 patients with unilateral MTLE-HS, 36 prior to antero-mesial temporal lobectomy (AMTL) and 40 after AMTL, and 28 healthy control subjects with a FER test consisting of 60 items (20 each for anger, fear, and happiness). Mean percentages of the accurate responses were calculated for different subgroups: right vs. left MTLE-HS, early (age at onset (...) <6 years) vs. late-onset, and before vs. after AMTL. After controlling for years of education, duration of epilepsy and number of antiepileptic drugs (AEDs) taken, on multivariate analysis, fear recognition was profoundly impaired in early-onset right MTLE-HS patients compared to other MTLE patients and control subjects. Happiness recognition was significantly better in post-AMTL MTLE-HS patients compared to pre-AMTL patients while anger and fear recognition did not differ. We conclude that patients with right MTLE-HS with age at seizure onset <6 years are maximally predisposed to impaired fear recognition. In them, right AMTL does not further worsen FER abilities. Longitudinal studies comparing FER in the same patients before and after AMTL will be required to refine and confirm our cross-sectional observations. (shrink)

BackgroundWe review original papers on ovarian-hormone status in two areas of emotional processing: facial emotion recognition and emotional memory. Ovarian-hormone status is operationalized by the levels of the steroid sex hormones 17β-estradiol and progesterone, fluctuating over the natural menstrual cycle and suppressed under oral contraceptive use. We extend previous reviews addressing single areas of emotional processing. Moreover, we systematically examine the role of stimulus features such as emotion type or stimulus valence and aim at elucidating factors that reconcile (...) the inconsistent results.MethodsWe followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and included papers published until September 2020 indexed in PubMed and Web of Science databases. Search terms were MeSH terms AND AND with representing our separately searched areas, resulting in, and. To be included, articles had to be written and published in English, examine healthy, non-pregnant adult women in their reproductive age, and measure or at least estimate levels of E2 and P4. In PubMed, the search was limited to humans and to the search term present in the title or abstract.ResultsFeatures of the provided stimulus material constitute a relevant influence that interacts with E2- and P4-related ovarian-hormone status. For instance, recognition of basic emotions appears to be more related to P4- than E2-levels. Quite consistent, OC intake was accompanied by impaired recognition accuracy of basic and also complex emotions, although not in a recent large-sample study assessing complex emotions. Memory recall of negative content was mainly enhanced by P4, especially after having been stressed.Discussion and ConclusionWe document the methodological diversity in the field, presumably contributing to the heterogeneity of results. More studies explicitly contrasting the early follicular phase, mid-cycle phase, mid-luteal, and OC intake while standardizing tasks are needed. Research would take advantage of using within-subject designs and accounting for the recognition of complex emotions. (shrink)

Deficits in recognition of facial expressions of emotions are considered to be an important factor explaining impairments in social functioning and affective reactions of schizophrenic patients. Many studies confirmed such deficits while controversies remained concerning the emotion valence and modality. The aim of the study was to explore the process of recognizing facial expressions of emotion in the group of schizophrenic patients by analyzing the role of emotion valence, modality and gender of the model. Results of the (...) group of 35 patients and 35 matched controls indicate that while schizophrenic patients show general impairment in recognizing facial expressions of both positive and the majority of negative emotions, there are differences in deficits for particular emotions. Expressions also appeared to be more ambiguous for the patients while variables connected with gender were found less significant. (shrink)

There are many ways to assess face perception skills. In this study, we describe a novel task battery FEAST (Facial Expression Action Stimulus Test) developed to test recognition of identity and expressions of human faces as well as stimulus control categories. The FEAST consists of a neutral and emotional face memory task, a face and object identity matching task, a face and house part-to-whole matching task, and a human and animal facial expression matching task. The identity and (...) part-to-whole matching tasks contain both upright and inverted conditions. The results provide reference data of a healthy sample of controls in two age groups for future users of the FEAST. (shrink)