The field of AI safety seeks to prevent or reduce the harms caused by AI systems. A simple and appealing account of what is distinctive of AI safety as a field holds that this feature is constitutive: a research project falls within the purview of AI safety just in case it aims to prevent or reduce the harms caused by AI systems. Call this appealingly simple account The Safety Conception of AI safety. Despite its simplicity (...) and appeal, we argue that The Safety Conception is in tension with at least two trends in the ways AI safety researchers and organizations think and talk about AI safety: first, a tendency to characterize the goal of AI safety research in terms of catastrophic risks from future systems; second, the increasingly popular idea that AI safety can be thought of as a branch of safety engineering. Adopting the methodology of conceptual engineering, we argue that these trends are unfortunate: when we consider what concept of AI safety it would be best to have, there are compelling reasons to think that The Safety Conception is the answer. Descriptively, The Safety Conception allows us to see how work on topics that have historically been treated as central to the field of AI safety is continuous with work on topics that have historically been treated as more marginal, like bias, misinformation, and privacy. Normatively, taking The Safety Conception seriously means approaching all efforts to prevent or mitigate harms from AI systems based on their merits rather than drawing arbitrary distinctions between them. (shrink)

This paper presents an argument that certain AI safety measures, rather than mitigating existential risk, may instead exacerbate it. Under certain key assumptions - the inevitability of AI failure, the expected correlation between an AI system's power at the point of failure and the severity of the resulting harm, and the tendency of safety measures to enable AI systems to become more powerful before failing - safety efforts have negative expected utility. The paper examines three response strategies: (...) Optimism, Mitigation, and Holism. Each faces challenges stemming from intrinsic features of the AI safety landscape that we term Bottlenecking, the Perfection Barrier, and Equilibrium Fluctuation. The surprising robustness of the argument forces a re-examination of core assumptions around AI safety and points to several avenues for further research. (shrink)

For decades, the field of system safety has designed safe systems by reducing the risk of physical harm to humans, property and the environment to an acceptable level. Recently, this definition of safety has come under scrutiny by governments and researchers who argue that the narrow focus on reducing physical harm, whilst necessary, is not sufficient to secure the safety of AI systems. There is growing pressure to expand the scope of safety in the context of (...) AI to address emerging harms, with particular emphasis being placed on the ways AI systems can reinforce and reproduce systemic harms. In this paper, we advocate for expanding the scope of conventional safety to include non-physical harms in the context of AI. However, we caution against broadening the scope to address systemic harms, as doing so presents intractable practical challenges for current safety methodologies. Instead, we propose that the scope of safety-related harms should be expanded to include psychological harms. Our proposal is partly motivated by the debates and evidence on social media, which fundamentally reshaped how harm is understood and addressed in the digital age, prompting new regulatory frameworks which aimed to protect users from the psychological risks of the technology. We draw on this precedent to motivate the inclusion of psychological harms in AI safety assessments. By expanding the scope of AI safety to include psychological harms, we take a critical step toward evolving the discipline of system safety into one that is better tuned and equipped to protect users against the complex and emerging harms propagated by AI systems. (shrink)

AI companies are racing to create artificial general intelligence, or “AGI.” If they succeed, the result will be human-level AI systems that can independently pursue high-level goals by formulating and executing long-term plans in the real world. Leading AI researchers agree that some of these systems will likely be “misaligned”–pursuing goals that humans do not desire. This goal mismatch will put misaligned AIs and humans into strategic competition with one another. As with present-day strategic competition between nations with incompatible goals, (...) the result could be violent and catastrophic conflict. Existing legal institutions are unprepared for the AGI world. New foundations for AGI governance are needed, and the time to begin laying them is now, before the critical moment arrives. -/- This Article begins to lay those new legal foundations. It is the first to think systematically about the dynamics of strategic competition between humans and misaligned AGI. The Article begins by showing, using formal game-theoretic models, that, by default, humans and AIs will be trapped in a prisoner’s dilemma. Both parties’ dominant strategy will be to permanently disempower or destroy the other, even though the costs of such conflict would be high. -/- The Article then argues that a surprising legal intervention could transform the game theoretic equilibrium and avoid conflict: AI rights. Not just any AI rights would promote human safety. Granting AIs the right not to be needlessly harmed–as humans have granted to certain non-human animals–would, for example, have little effect. Instead, to promote human safety, AIs should be given those basic private law rights–to make contracts, hold property, and bring tort claims–that law already extends to non-human corporations. Granting AIs these economic rights would enable long-run, small-scale, mutually-beneficial transactions between humans and AIs. This would, we show, facilitate a peaceful strategic equilibrium between humans and AIs for the same reasons economic interdependence tends to promote peace in international relations. Namely, the gains from trade far exceed those from war. Throughout, we argue that human safety, rather than AI welfare, provides the right framework for developing AI rights. This Article explores both the promise and the limits of AI rights as a legal tool for promoting human safety in an AGI world. (shrink)

Artificial Intelligence (AI) is taking centre stage in economic growth and business operations alike. Public discourse about the practical and ethical implications of AI has mainly focussed on the societal level. There is an emerging knowledge base on AI risks to human rights around data security and privacy concerns. A separate strand of work has highlighted the stresses of working in the gig economy. This prevailing focus on human rights and gig impacts has been at the expense of a closer (...) look at how AI may be reshaping traditional workplace relations and, more specifically, workplace health and safety. To address this gap, we outline a conceptual model for developing an AI Work Health and Safety (WHS) Scorecard as a tool to assess and manage the potential risks and hazards to workers resulting from AI use in a workplace. A qualitative, practice-led research study of AI adopters was used to generate and test a novel list of potential AI risks to worker health and safety. Risks were identified after cross-referencing Australian AI Ethics Principles and Principles of Good Work Design with AI ideation, design and implementation stages captured by the AI Canvas, a framework otherwise used for assessing the commercial potential of AI to a business. The unique contribution of this research is the development of a novel matrix itemising currently known or anticipated risks to the WHS and ethical aspects at each AI adoption stage. (shrink)

Machine ethics and robot rights are quickly becoming hot topics in artificial intelligence and robotics communities. We will argue that attempts to attribute moral agency and assign rights to all intelligent machines are misguided, whether applied to infrahuman or superhuman AIs, as are proposals to limit the negative effects of AIs by constraining their behavior. As an alternative, we propose a new science of safety engineering for intelligent artificial agents based on maximizing for what humans value. In particular, we (...) challenge the scientific community to develop intelligent systems that have human-friendly values that they provably retain, even under recursive self-improvement. (shrink)

One objection to conventional AI ethics is that it slows innovation. This presentation responds by reconfiguring ethics as an innovation accelerator. The critical elements develop from a contrast between Stability AI’s Diffusion and OpenAI’s Dall-E. By analyzing the divergent values underlying their opposed strategies for development and deployment, five conceptions are identified as common to acceleration ethics. Uncertainty is understood as positive and encouraging, rather than discouraging. Innovation is conceived as intrinsically valuable, instead of worthwhile only as mediated by social (...) effects. AI problems are solved by more AI, not less. Permissions and restrictions governing AI emerge from a decentralized process, instead of a unified authority. The work of ethics is embedded in AI development and application, instead of functioning from outside. Together, these attitudes and practices remake ethics as provoking rather than restraining artificial intelligence. (shrink)

Ensuring public safety is a critical challenge for rapidly growing urban areas. Traditional policing and emergency response systems often struggle to keep pace with the complexity and scale of modern cities. Artificial Intelligence (AI) offers a transformative solution by enabling real-time crime prediction, optimizing emergency resource allocation, and enhancing situational awareness through IoT-enabled systems. This paper explores how AI-driven analytics, combined with data from surveillance cameras, social media, and environmental sensors, can improve public safety in smart cities. By (...) addressing challenges such as data privacy, algorithmic bias, and system scalability, this study demonstrates the potential of AI to create safer, more resilient urban environments. Experimental results highlight improvements in crime detection rates, emergency response times, and resource utilization, paving the way for smarter public safety systems. (shrink)

As AI technologies increase in capability and ubiquity, AI accidents are becoming more common. Based on normal accident theory, high reliability theory, and open systems theory, we create a framework for understanding the risks associated with AI applications. This framework is designed to direct attention to pertinent system properties without requiring unwieldy amounts of accuracy. In addition, we also use AI safety principles to quantify the unique risks of increased intelligence and human-like qualities in AI. Together, these two fields (...) give a more complete picture of the risks of contemporary AI. By focusing on system properties near accidents instead of seeking a root cause of accidents, we identify where attention should be paid to safety for current generation AI systems. (shrink)

To realize the coordinated and sustainable development of hydropower projects and regional society, comprehensively evaluating hydropower projects’ influence is critical. Usually, hydropower project development has an impact on environmental geology and social and regional cultural development. Based on comprehensive consideration of complicated geological conditions, fragile ecological environment, resettlement of reservoir area, and other factors of future hydropower development in each country, we have constructed a comprehensive evaluation index system of hydropower projects, including 4 first-level indicators of social economy, environment, (...) class='Hi'>safety, and fairness, which contain 26 second-level indicators. To solve the problem that existing models cannot evaluate dynamic nonlinear optimization, a projection pursuit model is constructed by using rough set reduction theory to simplify the index. Then, an accelerated genetic algorithm based on real number coding is used to solve the model and empirical study is carried out with the Y hydropower station as a sample. The evaluation results show that the evaluation index system and assessment model constructed in our paper effectively reduce the subjectivity of index weight. Applying our model to the social impact assessment of related international hydropower projects can not only comprehensively analyze the social impact of hydropower projects but also identify important social influencing factors and effectively analyze the social impact level of each dimension. Furthermore, SIA assessment can be conducive to project decision-making, avoiding social risks and social stability. (shrink)

In the last years, artificial intelligence (AI) safety gained international recognition in the light of heterogeneous safety-critical and ethical issues that risk overshadowing the broad beneficial impacts of AI. In this context, the implementation of AI observatory endeavors represents one key research direction. This paper motivates the need for an inherently _transdisciplinary_ AI observatory approach integrating diverse retrospective and counterfactual views. We delineate aims and limitations while providing hands-on-advice utilizing _concrete practical examples_. Distinguishing between unintentionally and intentionally triggered (...) AI risks with diverse socio-psycho-technological impacts, we exemplify a _retrospective descriptive analysis_ followed by a _retrospective counterfactual risk analysis_. Building on these AI observatory tools, we present near-term transdisciplinary guidelines for AI safety. As further contribution, we discuss _differentiated_ and tailored long-term directions through the lens of two disparate modern AI safety paradigms. For simplicity, we refer to these two different paradigms with the terms _artificial stupidity_ (AS) and _eternal creativity_ (EC) respectively. While both AS and EC acknowledge the need for a hybrid cognitive-affective approach to AI safety and overlap with regard to many short-term considerations, they differ fundamentally in the nature of multiple envisaged long-term solution patterns. By compiling relevant underlying _contradistinctions_, we aim to provide _future-oriented_ incentives for constructive dialectics in practical and theoretical AI safety research. (shrink)

We propose developing AIs whose only final goal is being shut down. We argue that this approach to AI safety has three benefits: (i) it could potentially be implemented in reinforcement learning, (ii) it avoids some dangerous instrumental convergence dynamics, and (iii) it creates trip wires for monitoring dangerous capabilities. We also argue that the proposal can overcome a key challenge raised by Soares et al. (2015), that shutdown-seeking AIs will manipulate humans into shutting them down. We conclude by (...) comparing our approach with Soares et al.'s corrigibility framework. (shrink)

The conventional discourse on existential risks (x-risks) from AI typically focuses on abrupt, dire events caused by advanced AI systems, particularly those that might achieve or surpass human-level intelligence. These events have severe consequences that either lead to human extinction or irreversibly cripple human civilization to a point beyond recovery. This discourse, however, often neglects the serious possibility of AI x-risks manifesting incrementally through a series of smaller yet interconnected disruptions, gradually crossing critical thresholds over time. This paper contrasts the (...) conventional "decisive AI x-risk hypothesis" with an "accumulative AI x-risk hypothesis". While the former envisions an overt AI takeover pathway, characterized by scenarios like uncontrollable superintelligence, the latter suggests a different causal pathway to existential catastrophes. This involves a gradual accumulation of critical AI-induced threats such as severe vulnerabilities and systemic erosion of economic and political structures. The accumulative hypothesis suggests a boiling frog scenario where incremental AI risks slowly converge, undermining societal resilience until a triggering event results in irreversible collapse. Through systems analysis, this paper examines the distinct assumptions differentiating these two hypotheses. It is then argued that the accumulative view can reconcile seemingly incompatible perspectives on AI risks. The implications of differentiating between these causal pathways --- the decisive and the accumulative --- for the governance of AI as well as long-term AI safety are discussed. (shrink)

This paper offers the first careful analysis of the possibility that AI and humanity will go to war. The paper focuses on the case of artificial general intelligence, AI with broadly human capabilities. The paper uses a bargaining model of war to apply standard causes of war to the special case of AI/human conflict. The paper argues that information failures and commitment problems are especially likely in AI/human conflict. Information failures would be driven by the difficulty of measuring AI capabilities, (...) by the uninterpretability of AI systems, and by differences in how AIs and humans analyze information. Commitment problems would make it difficult for AIs and humans to strike credible bargains. Commitment problems could arise from power shifts, rapid and discontinuous increases in AI capabilities. Commitment problems could also arise from missing focal points, where AIs and humans fail to effectively coordinate on policies to limit war. In the face of this heightened chance of war, the paper proposes several interventions. War can be made less likely by improving the measurement of AI capabilities, capping improvements in AI capabilities, designing AI systems to be similar to humans, and by allowing AI systems to participate in democratic political institutions. (shrink)

Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, such as helping to commit crimes or producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about "collective" (...) preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023. (shrink)

In artificial intelligence (AI), responses generated by machine-learning models (most often large language models) may be unfactual information presented as a fact. For example, a chatbot might state that the Mona Lisa was painted in 1815. Such phenomenon is called AI hallucinations, seeking inspiration from human psychology, with a great difference of AI ones being connected to unjustified beliefs (that is, AI “beliefs”) rather than perceptual failures). -/- AI hallucinations may have their source in the data itself, that is, the (...) source content, or in the training procedure, i.e. the way the knowledge was encoded in the model’s parameters, so that errors in encoding and decoding textual and non-textual representations can cause hallucinations. In this paper, we will observe how such errors come to life and how they might be mitigated. For this purpose, we will analyze the usability of justification logics, to behave as a proof checker for validating the correctness of large language models’ (LLM) responses. Justification logic was developed by S. Artemov, and later on mostly by Artemov and M. Fitting, deriving its main idea from the logic of proofs (LP): knowledge and belief modalities are seen as justification terms, i.e. t:X stands for t is a (proper) justification for X. Justification logic originated from attempts to create semantics for intuitionistic logic where proofs were the most proper justifications, but in further development, justification logic could be applied to different kinds of justifications). With the recent attempts to mitigate incorrect LLM responses, we will analyze various guardrails that are currently used for LLM responses, and see how the logic of justification may provide its benefits as an AI safety layer against false data. (shrink)

As suicide rates increase worldwide, the mental health industry has reached an impasse in attempts to assess patients, predict risk, and prevent suicide. Traditional assessment tools are no more accurate than chance, prompting the need to explore new avenues in artificial intelligence (AI). Early studies into these tools show potential with higher accuracy rates than previous methods alone. Medical researchers, computer scientists, and social media companies are exploring these avenues. While Facebook leads the pack, its efforts stem from scrutiny following (...) suicides and suicide attempts broadcasted on Facebook Live. The company continues to face scrutiny for ethical, privacy, and safety concerns associated with the quick rollout of proprietary AI technology that seems more focused on the optics of the company than the protection of its billions of users. This paper explores some of these issues, including a lack of transparency, questionable data practices, escalation to law enforcement, little to no regulation, potential for bias, increased viewership of troubling content, and worsening thoughts of suicide among vulnerable Facebook users. Despite AI tools showing promise in improving the accuracy of suicide risk predictions over traditional methods, without regulation, external review, and increased protection of users and data, these tools have the potential to cause more harm than good in the hands of powerful companies like Facebook. (shrink)

The philosophical–ethical literature and the public debate on autonomous vehicles have been obsessed with ethical issues related to crashing. In this article, these discussions, including more empirical investigations, will be critically assessed. It is argued that a related and more pressing issue is questions concerning safety. For example, what should we require from autonomous vehicles when it comes to safety? What do we mean by ‘safety’? How do we measure it? In response to these questions, the article (...) will present a foundation for a continued discussion on these issues and an argument for why discussions about safety should be prioritized over ethical concerns related to crashing. (shrink)

AI research is growing rapidly raising various ethical issues related to safety, risks, and other effects widely discussed in the literature. We believe that in order to adequately address those issues and engage in a productive normative discussion it is necessary to examine key concepts and categories. One such category is anthropomorphism. It is a well-known fact that AI’s functionalities and innovations are often anthropomorphized (i.e., described and conceived as characterized by human traits). The general public’s anthropomorphic attitudes and (...) some of their ethical consequences (particularly in the context of social robots and their interaction with humans) have been widely discussed in the literature. However, how anthropomorphism permeates AI research itself (i.e., in the very language of computer scientists, designers, and programmers), and what the epistemological and ethical consequences of this might be have received less attention. In this paper we explore this issue. We first set the methodological/theoretical stage, making a distinction between a normative and a conceptual approach to the issues. Next, after a brief analysis of anthropomorphism and its manifestations in the public, we explore its presence within AI research with a particular focus on brain-inspired AI. Finally, on the basis of our analysis, we identify some potential epistemological and ethical consequences of the use of anthropomorphic language and discourse within the AI research community, thus reinforcing the need of complementing the practical with a conceptual analysis. (shrink)

If a future AI system can enjoy far more well-being than a human per resource, what would be the best way to allocate resources between these future AI and our future descendants? It is obvious that on total utilitarianism, one should give everything to the AI. However, it turns out that every Welfarist axiology on the market also gives this same recommendation, at least if we assume consequentialism. Without resorting to non-consequentialist normative theories that suggest that we ought not always (...) create the world with the most value, or non-welfarist theories that tell us that the best world may not be the world with the most welfare, I propose a new theory that justifies giving some resources to humanity in the face of overwhelming AI well-being. I call this new theory, “Group Prioritarianism". (shrink)

We argue that the later Wittgenstein’s philosophy of language and mathematics, substantially focused on rule-following, is relevant to understand and improve on the Artificial Intelligence (AI) alignment problem: his discussions on the categories that influence alignment between humans can inform about the categories that should be controlled to improve on the alignment problem when creating large data sets to be used by supervised and unsupervised learning algorithms, as well as when introducing hard coded guardrails for AI models. We cast these (...) considerations in a model of human–human and human–machine alignment and sketch basic alignment strategies based on these categories and further reflections on rule-following like the notion of meaning as use. To sustain the validity of these considerations, we also show that successful techniques employed by AI safety researchers to better align new AI systems with our human goals are congruent with the stipulations that we derive from the later Wittgenstein’s philosophy. However, their application may benefit from the added specificities and stipulations of our framework: it extends on the current efforts and provides further, specific AI alignment techniques. Thus, we argue that the categories of the model and the core alignment strategies presented in this work can inform further AI alignment techniques. (shrink)

A morally acceptable course of AI development should avoid two dangers: creating unaligned AI systems that pose a threat to humanity and mistreating AI systems that merit moral consideration in their own right. This paper argues these two dangers interact and that if we create AI systems that merit moral consideration, simultaneously avoiding both of these dangers would be extremely challenging. While our argument is straightforward and supported by a wide range of pretheoretical moral judgments, it has far-reaching moral implications (...) for AI development. Although the most obvious way to avoid the tension between alignment and ethical treatment would be to avoid creating AI systems that merit moral consideration, this option may be unrealistic and is perhaps fleeting. So, we conclude by offering some suggestions for other ways of mitigating mistreatment risks associated with alignment. (shrink)

This paper explores the development of an ethical guardrail framework for AI systems, emphasizing the importance of customizable guardrails that align with diverse user values and underlying ethics. We address the challenges of AI ethics by proposing a structure that integrates rules, policies, and AI assistants to ensure responsible AI behavior, while comparing the proposed framework to the existing state-of-the-art guardrails. By focusing on practical mechanisms for implementing ethical standards, we aim to enhance transparency, user autonomy, and continuous improvement in (...) AI systems. Our approach accommodates ethical pluralism, offering a flexible and adaptable solution for the evolving landscape of AI governance. The paper concludes with strategies for resolving conflicts between ethical directives, underscoring the present and future need for robust, nuanced and context-aware AI systems. (shrink)

Roughly, empirical AI alignment research (AIA) is an area of AI research which investigates empirically how to design AI systems in line with human goals. This paper examines the role of non-epistemic values in AIA. It argues that: (1) Sciences differ in the degree to which values influence them. (2) AIA is strongly value-laden. (3) This influence of values is managed inappropriately and thus threatens AIA’s epistemic integrity and ethical beneficence. (4) AIA should strive to achieve value transparency, critical scrutiny (...) from inside and outside the discipline – involving the public –, and to empower actors without strong commercial interests. (shrink)

When is it justified to use opaque artificial intelligence (AI) output in medical decision-making? Consideration of this question is of central importance for the responsible use of opaque machine learning (ML) models, which have been shown to produce accurate and reliable diagnoses, prognoses, and treatment suggestions in medicine. In this article, I discuss the merits of two answers to the question. According to the Explanation View, clinicians must have access to an explanation of why an output was produced. According to (...) the Validation View, it is sufficient that the AI system has been validated using established standards for safety and reliability. I defend the Explanation View against two lines of criticism, and I argue that within the framework of evidence-based medicine mere validation seems insufficient for the use of AI output. I end by characterizing the epistemic responsibility of clinicians and point out how a mere AI output cannot in itself ground a practical conclusion about what to do. (shrink)

Since some AI algorithms with high predictive power have impacted human integrity, safety has become a crucial challenge in adopting and deploying AI. Although it is impossible to prevent an algorithm from failing in complex tasks, it is crucial to ensure that it fails safely, especially if it is a critical system. Moreover, due to AI’s unbridled development, it is imperative to minimize the methodological gaps in these systems’ engineering. This paper uses the well-known Box-Jenkins method for statistical modeling (...) as a framework to identify engineering pitfalls in the adjustment and validation of AI models. Step by step, we point out state-of-the-art strategies and good practices to tackle these engineering drawbacks. In the final step, we integrate an internal and external validation scheme that might support an iterative evaluation of the normative, perceived, substantive, social, and environmental safety of all AI systems. (shrink)

The development and deployment of artificial intelligence (AI) is and will profoundly reshape human society, the culture and the composition of civilisations which make up human kind. All technological triggers tend to drive a hype curve which over time is realised by an output which is often unexpected, taking both pessimistic and optimistic perspectives and actions of drivers, contributors and enablers on a journey where the ultimate destination may be unclear. In this paper we hypothesise that this journey is not (...) dissimilar to the personal journey described by the Kubler-Ross change curve and illustrate this by commentary on the potential of AI for drug discovery, development and healthcare and as an enabler for deep space exploration and colonisation. Recent advances in the call for regulation to ensure development of safety measures associated with machine-based learning are presented which, together with regulation of the rapidly emerging digital after-life industry, should provide a platform for realising the full potential benefit of AI for the human species. (shrink)

Mobility robots may soon be among us, triggering a need for safety regulation. Robot safety regulation, however, remains underexplored, with only a few articles analyzing what regulatory approaches could be feasible. This article offers an account of the available regulatory strategies and attempts to theorize the effects of simulation-based safety regulation. The article first discusses the distinctive features of mobility robots as regulatory targets and argues that emergent behavior constitutes the key regulatory concern in designing robot (...) class='Hi'>safety regulation regimes. In contrast to many accounts, the article posits that emergent behavior dynamics do not arise from robot autonomy, learning capability, or code unexplainability. Instead, they emerge from the complexity of robot technological constitutions coupled with near-infinite environmental variability and non-linear performance dynamics of the machine learning components. Second, the article reviews rules-based and performance-based regulation and argues that both will fail adequately constrain emergent robot behaviors. The article claims that controlling mobility robots requires a simulation-based regulatory approach. Simulation-based regulation is a novelty with significant theoretical and practical implications. The article argues that the approach signifies a radical break in regulatory forms of knowledge and temporalities. Simulations enact virtual futures to create a new regulatory knowledge type. Practically, the novel safety knowledge type may destabilize the existing conceptual space of safety politics and liability allocation patterns. (shrink)

With the introduction of artificial intelligence (AI) to healthcare, there is also a need for professional guidance to support its use. New (2022) reports from National Health Service AI Lab & Health Education England focus on healthcare workers’ understanding and confidence in AI clinical decision support systems (AI-CDDSs), and are concerned with developing trust in, and the trustworthiness of these systems. While they offer guidance to aid developers and purchasers of such systems, they offer little specific guidance for the clinical (...) users who will be required to use them in patient care. This paper argues that clinical, professional and reputational safety will be risked if this deficit of professional guidance for clinical users of AI-CDDSs is not redressed. We argue it is not enough to develop training for clinical users without first establishing professional guidance regarding the rights and expectations of clinical users. We conclude with a call to action for clinical regulators: to unite to draft guidance for users of AI-CDDS that helps manage clinical, professional and reputational risks. We further suggest that this exercise offers an opportunity to address fundamental issues in the use of AI-CDDSs; regarding, for example, the fair burden of responsibility for outcomes. (shrink)

Recent AI progress led to a boost in beneficial applications from multiple research areas including VR. Simultaneously, in this newly unfolding deepfake era, ethically and security-relevant disagreements arose in the scientific community regarding the epistemic capabilities of present-day AI. However, given what is at stake, one can postulate that for a responsible approach, prior to engaging in a rigorous epistemic assessment of AI, humans may profit from a self-questioning strategy, an examination and calibration of the experience of their own epistemic (...) agency – especially to counteract both intentional misdirection by unethical actors and unintentional epistemic self-sabotage. In this paper, we expound on a new avenue of utilizing AIVR tools to advance an AI-related misdirection awareness of humans in the deepfake era. Firstly, we harness scientific knowledge from the psychology and neuroscience of magic where the study of misdirection techniques is center stage. Secondly, we connect the latter to creativity research linking human creative potential to inspiration from the seemingly impossible. Overall, AIVR could become an empowering experiential testbed for human epistemic agency enabling a better rational evaluation of AI capabilities. However, a misuse of the same type of tools could yield AIVR safety risks if not counteracted preemptively. (shrink)

Will AI take away your job? The answer is probably not. AI systems can be good predictive systems and be very good at pattern recognition. AI systems have a very repetitive approach to sets of data, which can be useful in certain circumstances. However, AI does make obvious mistakes. This is because AI does not have a sense of context. As Humans we have years of experience in the real world. We have vast amounts of contextual data stored in our (...) brains that make it possible to predict and to know boundaries of the real world so that even if we have never been in a particular situation, we are still able to deal with it. The unknown situation is where AI will fall down. In engineering, AI is being developed to monitor and control certain systems, this could extend to Nuclear Power Plants for example. If we examine control systems for Nuclear Power Plants on vessels, the system would likely be programmed using the safety of the reactor as the main priority. Automation is crucial in engineering systems like this, as the likelihood and cost of human error is high and human reaction times are slow in comparison to the system. However, there is always the possibility of unforeseen external factors that may override plant safety which cannot be programmed. If we look at a plant onboard a Submarine, for example, we see such additional factors as ship safety. In one example, the power plant might breach safety parameters, in which case an automatic system may shut down the reactor. However, there might be a greater urgency such as a flood in another compartment or an attack upon the vessel, that would override shutting down the reactor. The AI system can neither observe nor utilise information from the external environment to make this type of judgement, whereas the operator may be completely aware. To programme context into AI systems would be near impossible. We still need operators as we still need to understand the environment around us, which may contain an infinite number of possibilities. In a further example of this; if we examine the ICO (Information Commissioner's Office) guidance, it is clear that in clinical settings a human being will always be required to validate findings and to provide context. Sometimes having a lot of data just doesn't replace social interaction, intuition and experience. (shrink)

The purpose of this paper is to increase the understanding of how different types of experts with influence over the development of AI, in this role, reflect upon distribution of forward-looking responsibility for AI development with regard to safety and democracy. Forward-looking responsibility refers to the obligation to see to it that a particular state of affairs materialise. In the context of AI, actors somehow involved in AI development have the potential to guide AI development in a safe and (...) democratic direction. This study is based on qualitative interviews with such actors in different roles at research institutions, private companies, think tanks, consultancy agencies, parliaments, and non-governmental organisations. While the reflections about distribution of responsibility differ among the respondents, one observation is that influence is seen as an important basis for distribution of responsibility. Another observation is that several respondents think of responsibility in terms of what it would entail in concrete measures. By showing how actors involved in AI development reflect on distribution of responsibility, this study contributes to a dialogue between the field of AI governance and the field of AI ethics. (shrink)

I explain the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems show that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it’s costly to do so. And (...) patience trades off against shutdownability: the more patient an agent, the greater the costs that agent is willing to incur to manipulate the shutdown button. I end by noting that these theorems can guide our search for solutions. (shrink)

With the intensifying shortage of care-providers and mounting financial burden of an aging population in Japan, artificial intelligence (AI) offers a potential solution through AI-driven robots, chatbots, smartphone apps, and other AI medical services. Yet Japanese acceptance of medical AI, especially patient care, largely depends on the degree of ‘humanness’ that can be integrated into intelligent technologies. As empathy is considered a core value in the practice of healthcare workers, artificially intelligent agents must have the ability to perceive human emotions (...) and respond in an affective, human-like manner. This paper presents the results of a nationwide survey in Japan 2023 about the public perception of the use of emotionalized AI (EAI) in healthcare. The sample included 2000 Japanese citizens from 20 to 69 years old with an average age of 45 years. The questionnaire included 65 items in six categories: demographic, perception of the existing healthcare system, attitude towards the use of AI, and EAI in healthcare, trust and privacy concerns and fear of AI, and seeking literacy on new technologies. The analysis of the responses demonstrates that overall, the Japanese population has a limited knowledge of AI, EAI, and smart technologies but trusts the healthcare system, physicians, and staff to use it properly, with little worries over the potential pitfalls related to data security and safety. However, they prefer the use of more human resources and are concerned about privacy and the high cost of new technology, and believe little information is being provided to them through government and media. (shrink)

The field of AI safety aims to prevent increasingly capable artificially intelligent systems from causing humans harm. Research on moral alignment is widely thought to offer a promising safety strategy: if we can equip AI systems with appropriate ethical rules, according to this line of thought, they'll be unlikely to disempower, destroy or otherwise seriously harm us. Deontological morality looks like a particularly attractive candidate for an alignment target, given its popularity, relative technical tractability and commitment to harm-avoidance (...) principles. I argue that the connection between moral alignment and safe behavior is more tenuous than many have hoped. In general, AI systems can possess either of these properties in the absence of the other, and we should favor safety when the two conflict. In particular, advanced AI systems governed by standard versions of deontology need not be especially safe. (shrink)

Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems (...) capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance. (shrink)

The year 2012 has witnessed worldwide celebrations of Alan Turing’s 100th birthday. A great number of conferences and workshops were organized by logicians, computer scientists and researchers in AI, showing the continued flourishing of computer science, and the fruitful interfaces between logic and computer science. Logic is no longer just the concept that Frege had about one hundred years ago, let alone that of Aristotle twenty centuries before. One of the prominent features of contemporary logic is its interdisciplinary character, connecting (...) across mathematics, philosophy, modern computer science, and even the cognitive and social sciences. -/- China is no exception. This special issue explores interfaces of logic with AI and other disciplines, and it attempts to give a bird’s eye view of the developing landscape of logic and AI in China. Our contributors come from computer science, AI, philosophy and linguistics, united by their interest in using formal logical tools and notions to deal with substantial problems in their fields. We are aware that AI is a vast area, and that the limitations of space in a volume like this may provide a somewhat incomplete or biased view. Still we hope that the six papers selected here illustrate the concerns and curiosity driving Chinese researchers, on topics shared with our colleagues around the world. -/- One of the active research areas in AI is planning under uncertainty, with incomplete knowledge, where a well-established logical framework is the action language of Son and Baral (Son and Baral 2001). This language can express the notions of sensing actions and making conditional plans. However, its current semantics is not yet satisfactory, since according to the semantics an agent would not have knowledge of what happens after a plan is executed, which can be a serious problem in a safety-critical environment. In their paper Proof Systems for Planning under Cautious Semantics (Shen and Zhao 2013), Yuping Shen and Xishun Zhao address this problem and propose a cautious and weakly cautious semantics for the action language. This new semantics enables the agent to know exactly what happens after executing an action. Hoare style proof systems are then proposed and proved to be sound and complete. What is more, the proposed new system reduces the reasoning complexity of the action language. -/- In Preferential Semantics for Plausible Subsumption in Possibility Theory (Qi and Zhang 2013), Guilin Qi and Zhizheng Zhang investigate another important issue in AI: how to handle exceptions in a knowledge-based system. This has become a hot topic in recent years, especially in ontological design, where description logics are widely used. The question then arises whether we can get a nonmonotonic extension of description logic. To answer this, the authors first propose three preferential semantics for plausible subsumption in the framework of possibility theory (Dubois and Prade 1985) and study their properties and their relationships. Then they introduce preferential subsumptions to a description logic-based knowledge system, and define an entailment relation of plausible subsumption. Moreover, they study reductions of the new semantics to standard semantics for an expressive description logic. -/- Counterfactual reasoning has been explored extensively by philosophers, cf. Lewis (1973), Stalnaker (1968). But in the last two decades, causal models have been proposed for modeling counterfactual reasoning by researchers in AI. In A Lewisian Logic of Causal Counterfactuals (Zhang 2013), Jiji Zhang studies classes of causal models and provides formal characterizations. In particular, a new logic system is proposed and proved complete for the class of ‘Lewisian models’ satisfying all the principles of David Lewis’s logic of counterfactuals. Moreover, Zhang compares his new framework with those of Lewis and Stalnaker, and also establishes formal connections with more recent approaches by Galles and Pearl, and by Halpern. -/- Consistency is a fundamental notion in logic, but it also plays a key role in many techniques for solving constraint satisfaction problems (Mackworth 1977). In Variable-Centered Consistency in Model RB (Li et al. 2013), Liang Li, Tian Liu and Ke Xu start with a discussion of i-consistency in ‘Model RB’ and ‘Model RD’, two models of random constraint satisfaction problems with growing domain size, then compare it with the notion of t-consistency and strong t-consistency in well-known earlier work by Freuder. After that, the authors define a new kind of variable-centered consistency, and show an upper bound on a suitably chosen parameter in Model RB and Model RD, such that up to this bound, all typical instances of Model RB are variable-centered consistent. -/- Language understanding is a cognitive process that human beings carry out every day. Metaphor is a pervasive phenomenon in natural languages, and its cognitive role calls for interpretation. There is a growing body of literature on cognitive mechanisms underlying metaphor. The paper An Ontology-Based Approach to Metaphor Cognitive Computation (Huang et al. 2013) by Xiaoxi Huang, Huaxin Huang, Beishui Liao, and Cihua Xu proposes a theoretical framework for metaphor understanding based on recent results from cognitive science (Shutova 2010). They then introduce ontology as a knowledge representation method, and present a new ontological model to formalize metaphor-based knowledge. An algorithmic description is provided using a quantitative measure of integrated degree, and its effectiveness is demonstrated. -/- Puzzles and their solution mechanisms are a recurrent theme in many fields. Dynamic epistemic logic (van Ditmarsch et al. 2007); (van Benthem 2011) is a recent framework that can handle information dynamics in knowledge puzzles. In Reasoning About Agent Types and the Hardest Logic Puzzle Ever (Liu and Wang 2013), Fenrong Liu and Yanjing Wang propose a new dynamic-epistemic logic that can specify different types of agents and analyse agents’ knowledge, communication and reasoning. To demonstrate its power, Smullyan’s well-known Knights and Knaves puzzles are formalized, as well as the Hardest Logic Puzzle Ever (HLPE) of Boolos (Boolos 1996). Moreover, a spectrum of new puzzles is found involving knowledge-based agent types, and their solution complexity is shown to be harder than that for existing logic puzzles. It is also shown that a version of HLPE in which the agents do not know the others’ types does not have a solution at all. -/- Finally, we would like to thank the editor of Minds and Machines, Gregory Wheeler, for his efficient assistance and kind support. We also thank Ties Nijssen at Springer for his encouragement to start this project at the first place. And of course, our appreciation goes to our authors, as well as all the colleagues who have helped us review the papers in this collection. (shrink)

The idea of artificial intelligence for social good is gaining traction within information societies in general and the AI community in particular. It has the potential to tackle social problems through the development of AI-based solutions. Yet, to date, there is only limited understanding of what makes AI socially good in theory, what counts as AI4SG in practice, and how to reproduce its initial successes in terms of policies. This article addresses this gap by identifying seven ethical factors that are (...) essential for future AI4SG initiatives. The analysis is supported by 27 case examples of AI4SG projects. Some of these factors are almost entirely novel to AI, while the significance of other factors is heightened by the use of AI. From each of these factors, corresponding best practices are formulated which, subject to context and balance, may serve as preliminary guidelines to ensure that well-designed AI is more likely to serve the social good. (shrink)

The European Union Commission’s whitepaper on Artificial Intelligence proposes shaping the emerging AI market so that it better reflects common European values. It is a master plan that builds upon the EU AI High-Level Expert Group guidelines. This article reviews the masterplan, from a culture cycle perspective, to reflect on its potential clashes with current societal, technical, and methodological constraints. We identify two main obstacles in the implementation of this plan: the lack of a coherent EU vision to drive future (...) decision-making processes at state and local levels and the lack of methods to support a sustainable diffusion of AI in our society. The lack of a coherent vision stems from not considering societal differences across the EU member states. We suggest that these differences may lead to a fractured market and an AI crisis in which different members of the EU will adopt nation-centric strategies to exploit AI, thus preventing the development of a frictionless market as envisaged by the EU. Moreover, the Commission aims at changing the AI development culture proposing a human-centred and safety-first perspective that is not supported by methodological advancements, thus taking the risks of unforeseen social and societal impacts of AI. We discuss potential societal, technical, and methodological gaps that should be filled to avoid the risks of developing AI systems at the expense of society. Our analysis results in the recommendation that the EU regulators and policymakers consider how to complement the EC programme with rules and compensatory mechanisms to avoid market fragmentation due to local and global ambitions. Moreover, regulators should go beyond the human-centred approach establishing a research agenda seeking answers to the technical and methodological open questions regarding the development and assessment of human-AI co-action aiming for a sustainable AI diffusion in the society. (shrink)

Service and personal care robots are starting to cross the threshold into the wilderness of everyday life, where they are supposed to interact with inexperienced lay users in a changing environment. In order to function as intended, robots must become independent entities that monitor themselves and improve their own behaviours based on learning outcomes in practice. This poses a great challenge to robotics, which we are calling the “autonomy-safety-paradox” (ASP). The integration of robot applications into society requires the reconciliation (...) of two conflicting aspects: increasing machine autonomy and ensuring safety in end-use. As the level of robot autonomy grows, the risk of accidents will increase, and it will become more and more difficult to identify who is responsible for any damage incurred. However, emphasizing safety impairs the autonomous functioning of the robot. This problem implies the need for a broadened concept of product safety. Our comparative study shows that the institutional framing of the ASP as well as concrete solutions to this problem differs between Europe and Japan in two respects: (1) the understanding of robot agency and (2) the concept of “appropriate” user–robot interaction. (shrink)

There is no strong reason to believe that human-level intelligence represents an upper limit of the capacity of artificial intelligence, should it be realized. This poses serious safety issues, since a superintelligent system would have great power to direct the future according to its possibly flawed motivation system. Solving this issue in general has proven to be considerably harder than expected. This paper looks at one particular approach, Oracle AI. An Oracle AI is an AI that does not act (...) in the world except by answering questions. Even this narrow approach presents considerable challenges. In this paper, we analyse and critique various methods of controlling the AI. In general an Oracle AI might be safer than unrestricted AI, but still remains potentially dangerous. (shrink)

Urban lighting systems are essential for safety, security, and quality of life, but they often consume significant energy and lack adaptability to changing conditions. Traditional lighting systems rely on fixed schedules and manual adjustments, leading to inefficiencies such as over-illumination and energy waste. This paper explores how Artificial Intelligence (AI) and IoT technologies can optimize urban lighting by enabling real-time adjustments, energy savings, and adaptive illumination based on environmental conditions and human activity. By integrating data from motion sensors, weather (...) forecasts, and traffic systems, cities can reduce energy consumption, enhance safety, and improve the quality of life for residents. Experimental results demonstrate significant improvements in energy efficiency, lighting quality, and operational costs, offering a sustainable blueprint for smart urban lighting systems. (shrink)

In this study, we analyse the public perception of military AI in Estonia, a techno-optimistic country with high support for science and technology. This study involved quantitative survey data from 2021 on the public’s attitudes towards AI-based technology in general, and AI in developing and using weaponised unmanned ground systems (UGS) in particular. UGS are a technology that has been tested in militaries in recent years with the expectation of increasing effectiveness and saving manpower in dangerous military tasks. However, developing (...) weaponised systems often raises questions about their safety and accountability. While there have been public debates about the possible outcomes of AI-based civilian technologies like self-driving cars, equivalent discussions have not been held about military technologies. A discourse on potentially sensitive technologies can diminish biased representations and the distribution of worst-case scenarios. The results of this study reveal that an overall positive attitude towards AI-driven technologies is related to a positive perception of military AI systems. However, trustworthiness and human control are the main aspects that are expected from military technologies. (shrink)

In the rapidly evolving landscape of multimodal communication research, this paper explores the transformative role of machine learning (ML), particularly using multimodal large language models, in tracking, augmenting, annotating, and analyzing multimodal data. Building upon the foundations laid in our previous work, we explore the capabilities that have emerged over the past years. The integration of ML allows researchers to gain richer insights from multimodal data, enabling a deeper understanding of human (and non-human) communication across modalities. In particular, augmentation methods (...) have become indispensable because they facilitate the synthesis of multimodal data and further increase the diversity and richness of training datasets. In addition, ML-based tools have accelerated annotation processes, reducing human effort while improving accuracy. Continued advances in ML and the proliferation of more powerful models suggest even more sophisticated analyses of multimodal communication, e.g., through models like ChatGPT, which can now “understand” images. This makes it all the more important to assess what these models can achieve now or in the near future, and what will remain unattainable beyond that. We also acknowledge the ethical and practical challenges associated with these advancements, emphasizing the importance of responsible AI and data privacy. We must be careful to ensure that benefits are shared equitably and that technology respects individual rights. In this paper, we highlight advances in ML-based multimodal research and discuss what the near future holds. Our goal is to provide insights into this research stream for both the multimodal research community, especially in linguistics, and the broader ML community. In this way, we hope to foster collaboration in an area that is likely to shape the future of technologically mediated human communication. (shrink)

This paper offers an overview of the prospects and ethics of using AI to achieve human enhancement, and more broadly what we call intellectual augmentation (IA). After explaining the central notions of human enhancement, IA, and AI, we discuss the state of the art in terms of the main technologies for IA, with or without brain-computer interfaces. Given this picture, we discuss potential ethical problems, namely inadequate performance, safety, coercion and manipulation, privacy, cognitive liberty, authenticity, and fairness in more (...) detail. We conclude that while there are very significant technical hurdles to real human enhancement through AI, and significant ethical problems, there are also significant benefits that may realistically be achieved in ways that are consonant with a rights-based ethics as well. We also highlight the specific concerns that apply particularly to applications of AI for "sheer" IA (more realistic in the near term), and to enhancement applications, respectively. (shrink)

Developers are often the engine behind the creation and implementation of new technologies, including in the artificial intelligence surge that is currently underway. In many cases these new technologies introduce significant risk to affected stakeholders; risks that can be reduced and mitigated by such a dominant party. This is fully recognized by texts that analyze risks in the current AI transformation, which suggest voluntary adoption of ethical standards and imposing ethical standards via regulation and oversight as tools to compel developers (...) to reduce such risks. However, what these texts usually sidestep is the question of how aware developers are to the risks they are creating with these new AI technologies, and what their attitudes are towards such risks. This paper asks to rectify this gap in research, by analyzing an ongoing case study. Focusing on six Israeli AI startups in the field of radiology, I carry out a content analysis of their online material in order to examine these companies’ stances towards the potential threat their automated tools pose to patient safety and to the work-standing of healthcare professionals. Results show that these developers are aware of the risks their AI products pose, but tend to deny their own role in the technological transformation and dismiss or downplay the risks to stakeholders. I conclude by tying these findings back to current risk-reduction recommendations with regards to advanced AI technologies, and suggest which of them hold more promise in light of developers’ attitudes. (shrink)

Large language models now possess human-level linguistic abilities in many contexts. This raises the concern that they can be used to deceive and manipulate on unprecedented scales, for instance spreading political misinformation on social media. In future, agentic AI systems might also deceive and manipulate humans for their own purposes. In this paper, first, I argue that AI-generated content should be subject to stricter standards against deception and manipulation than we ordinarily apply to humans. Second, I offer new characterizations of (...) AI deception and manipulation meant to support such standards, according to which a statement is deceptive (resp. manipulative) if it leads human addressees away from the beliefs (resp. choices) they would endorse under “semi-ideal” conditions. Third, I propose two measures to guard against AI deception and manipulation, inspired by this characterization: “extreme transparency” requirements for AI-generated content and “defensive systems” that, among other things, annotate AI-generated statements with contextualizing information. Finally, I consider to what extent these measures can protect against deceptive behavior in future, agentic AI systems. (shrink)

Recent work in explainable artificial intelligence (XAI) attempts to render opaque AI systems understandable through a divide-and-conquer strategy. However, this fails to illuminate how trained AI systems work as a whole. Precisely this kind of functional understanding is needed, though, to satisfy important societal desiderata such as safety. To remedy this situation, we argue, AI researchers should seek mechanistic interpretability, viz. apply coordinated discovery strategies familiar from the life sciences to uncover the functional organisation of complex AI systems. Additionally, (...) theorists should accommodate for the unique costs and benefits of such strategies in their portrayals of XAI research. (shrink)

Meaningful human control over AI is exalted as a key tool for assuring safety, dignity, and responsibility for AI and automated decision-systems. It is a central topic especially in fields that deal with the use of AI for decisions that could cause significant harm, like AI-enabled weapons systems. This paper argues that discussions regarding meaningful human control commonly fail to identify the purpose behind the call for meaningful human control and that stating that purpose is a necessary step in (...) deciding how best to institutionalize meaningful human control over AI. The paper identifies 5 common purposes for human control and sketches how different purpose translate into different institutional design. (shrink)