Title: Identifying Ethical Hazards in Safety-Critical Systems: The Role of Creativity

Author(s): Austen Rainer, Catherine Menon

Publication Event: Publication of Proceedings of the Thirty third Safety-Critical Systems Symposium

Publication Date: 2025-02-01

Resource URL: https://scsc.uk/r3095.pdf

Abstract:

Safety-critical systems can present a varied range of ethical hazards to users, operators and other stakeholders. Some of these hazards, such as a lack of fairness or transparency, have been discussed extensively in existing literature and appear in guidance documents and international standards. Others, such as cultural flattening, anthropomorphism, automation bias and systemic racial discrimination, are typically harder to identify and consequently harder to mitigate against. This paper presents an argument that creativity and collaboration play an essential role in ethical hazard analysis, and introduces a category of HAZOP-based techniques which can be used for a structured and creative discussion of such ethical hazards, although the two disciplines have not been well integrated. Both now face challenges from the growing use artificial intelligence and machine learning – both severally and jointly – but there are also opportunities. The aim of this paper is to provide a structure for thinking about these challenges and opportunities, especially those where there are shared or overlapping concerns. It is hoped that this will help inform a research agenda for safety and security of systems employing artificial intelligence, and especially machine learning.