Safety Assurance for AI Systems

Speakers include:

Simon Burton, University of York - Safety assurance of AI-based autonomy: Managing the paradigm shift from deterministic safety to assurance under uncertainty

Abstract: The paradigm shift from deterministic safety engineering to the assurance of AI-based autonomy operating under open-world uncertainty challenges the fundamental assumptions within established safety standards. ISO 26262, IEC 61508, and DO-178C rely on explicit requirements decomposition, structural V-model traceability, and the assumption that deterministic behaviour can be designed into software and validated against explicit specifications. When applied to data-driven, opaque machine learning components operating in complex, dynamic contexts, conventional approaches face severe semantic gaps, unquantified epistemic uncertainty, and a lack of scalability in traditional statistical testing. This presentation examines how safety assurance must evolve to address these gaps across multiple sectors, to provide a rigorous framework for the assurance of AI-based autonomy. It contrasts traditional software safety principles with the realities of assuring learning-based autonomous systems. Building on this four complementary pillars are introduced to close these assurance gaps: 1) the need for ethically informed and context-aware safety claims, demonstrating how frameworks such as SACE and structured requirements decomposition can bridge the semantic gap between high-level safety goals and data-driven component expectations; 2) it addresses the shift toward explicit world models, examining how moving from static operational design domain descriptions to causally grounded representations enables principled reasoning about environmental assumptions and context; 3) it discusses strategies for structuring evidence to explicitly account for doubt and bias, highlighting dialectic argumentation methods, that actively pressure-test safety claims against potential defeaters; 4) through-life and reflective assurance architectures, illustrating how continuous operational monitoring, dynamic risk indicators, and iterative feedback loops allow system safety cases to adapt safely as operating conditions and technologies evolve over time. It concludes with a discussion on how these principles can be integrated into existing practices in an incremental manner, so that assurance rigour can be increased in line with the complexity and criticality of the systems and their operating context.