Safety Assurance for AI Systems

This event is aimed at safety engineers, managers and regulators involved in the assurance of Artificial Intelligence Systems (AIS). AIS are machine-based systems that, for explicit or implicit objectives, infer, from the input they receive, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. Examples include autonomous vehicles and patient triaging systems.

This 1-day event has a series of talks by researchers seeking to provide practical guidance for assurance of AIS characteristics, such as uncertainty due to non-determinism, and practitioners attempting to employ and assure AIS in practice.

Presenters include:

🔶 Chris Burr, Alan Turing Institute – "Mapping the current shape of the UK's AI assurance ecosystem" 

🔶 Simon Burton, CfAA, UoY - "Managing the paradigm shift from deterministic safety to assurance under uncertainty”

🔶 Ollie James-Knight, SQEP – "Use of LLMs in AS Safety Assurance"

🔶 Mark Nicholson, University of York - "AI Systems Safety Assurance Delta"

🔶 Andrew White, Office for Nuclear Regulation

🔶 Ranj Zuhdi, SDLC Compliance - "Cleared for use, but trusted? The challenges of regulating medical AI"

Talk Abstracts

Chris Burr, Alan Turing Institute – "Mapping the current shape of the UK's AI assurance ecosystem" addresses the UK's wider AI assurance ecosystem addressing recent discussions concerning Sovereign AI

Abstract: The demands we place on AI systems have widened. Explainability, security, accountability, and fairness now sit alongside safety as claims that need evidence and assurance. The UK government's approach to AI assurance and responsible innovation makes this widening explicit. No single organisation or group of stakeholders can deliver assurance on their own. Instead, ensuring a functioning and flourishing assurance ecosystem for the UK requires complementary standards, tools, skilled people, and services that connect claims about AI systems to the evidence they require. This talk maps the current shape of the UK's AI assurance ecosystem, asks where its capabilities fall short, and connects this question to recent discussions concerning Sovereign AI. For instance, if the infrastructure Britain depends on is typically built elsewhere, how can we best develop the capabilities to assure it, or to characterise and interrogate systems we did not build? As part of this talk, I will showcase some of the work and tools developed at the Alan Turing Institute, which have been designed to help teams embed AI assurance capabilities into their projects (e.g. an open database of assurance techniques and dynamic assurance pipelines for systems that continue to change after deployment).

Simon Burton, CfAA, UoY - "Managing the paradigm shift from deterministic safety to assurance under uncertainty” that addresses the assurance gaps that emerge due to uncertainty.

Abstract: The paradigm shift from deterministic safety engineering to the assurance of AI-based autonomy operating under open-world uncertainty challenges the fundamental assumptions within established safety standards. ISO 26262, IEC 61508, and DO-178C rely on explicit requirements decomposition, structural V-model traceability, and the assumption that deterministic behaviour can be designed into software and validated against explicit specifications. When applied to data-driven, opaque machine learning components operating in complex, dynamic contexts, conventional approaches face severe semantic gaps, unquantified epistemic uncertainty, and a lack of scalability in traditional statistical testing. This presentation examines how safety assurance must evolve to address these gaps across multiple sectors, to provide a rigorous framework for the assurance of AI-based autonomy. It contrasts traditional software safety principles with the realities of assuring learning-based autonomous systems. Building on this four complementary pillars are introduced to close these assurance gaps: 1) the need for ethically informed and context-aware safety claims, demonstrating how frameworks such as SACE and structured requirements decomposition can bridge the semantic gap between high-level safety goals and data-driven component expectations; 2) it addresses the shift toward explicit world models, examining how moving from static operational design domain descriptions to causally grounded representations enables principled reasoning about environmental assumptions and context; 3) it discusses strategies for structuring evidence to explicitly account for doubt and bias, highlighting dialectic argumentation methods, that actively pressure-test safety claims against potential defeaters; 4) through-life and reflective assurance architectures, illustrating how continuous operational monitoring, dynamic risk indicators, and iterative feedback loops allow system safety cases to adapt safely as operating conditions and technologies evolve over time. It concludes with a discussion on how these principles can be integrated into existing practices in an incremental manner, so that assurance rigour can be increased in line with the complexity and criticality of the systems and their operating context.

Ollie James-Knight, SQEP – "Use of LLMs in AS Safety Assurance" investigates the case for capabilities such as LLMs in products and the assurance of products from a safety practitioners’ point of view.

Abstract: AI adoption has increased dramatically, but hesitant uptake in safety engineering may be limiting the ability to keep pace. Assurance cases present just one avenue where this gap can be closed. Current practice is slow, expensive, and a persistent bottleneck in delivering safety-critical systems. A problem that will only be exacerbated by autonomous systems, with large evidence volumes, emergent behaviour, and established guidance requiring considerable analytical effort.
Drawing on the speaker’s experience developing assurance arguments for the aerospace industry and master’s research into Large Language Models for safety cases, this talk discusses the merits and drawbacks of using LLMs to support safety assurance and outlines the initial opportunities for their introduction. It will appraise the benefits of LLMs; their speed, capacity to process volumes of evidence, and ability to surface missed argumentation, against the fabrications, omissions, and quiet failures that can produce assurance deficits. Also discussed is the current division in adoption between product and process, with a conclusion on the importance of defensibility in adoption.

Mark Nicholson, University of York - AI Systems Safety Assurance Delta

Ranj Zuhdi, SDLC Compliance - "Cleared for use, but trusted? The challenges of regulating medical AI"

Abstract: AI has made real progress in healthcare, with hundreds of AI/ML enabled medical devices reaching the market despite a regulatory system still adapting to the technology. But approval is not the same as trust. Pre-market clinical evidence is rarely accessible to clinicians, post-market surveillance data is thin, and with most tools positioned as clinical decision support, accountability lines are blurred.  For medical decision support tools, how much of the risk from AI failure modes, and the accountability for them, can be offloaded to users in the name of human oversight? Drawing on experience in the medical device industry, this talk looks at why regulating AI in high-risk domains remains so difficult, and why generative AI raises the stakes further. It asks whether today’s AI meets the safety sectors’ expectations of mature, reliable technology and harmonised standards, and what the EU AI Act’s addition of fundamental rights means for how we assess risk.