Functional Safety and Oracular Subsystems

An Observation on ISO/IEC TR 5469: Artificial Intelligence — Functional safety and AI systems

Abstract

“AI” subsystems are finding their way into safety-critical systems.  Use of this term has come to mean contemporarily software systems based on machine-learning (ML) techniques.  It is an open question how such subsystems may be verified and validated in safety-critical applications.  A guidance document for functional safety in the presence of AI subsystems has recently been published by ISO and IEC, ISO/IEC TR 5469:2024.  This paper considers critically the conception expounded in TR 5469 of how such subsystems are to be construed architecturally and behaviourally, through considering the example of adaptive control.  Some concepts which may be useful in characterising AI subsystems are proposed.