Title: An Updatable Process to Assess Human Interaction with Artificial Intelligence and Advanced Automation Over Time

Author(s): Abigail M. Post, David D. Woods, Martijn IJtsma

Publication Event: Safety-Critical Systems Symposium 2026

Publication Date: 2025-12-09

Resource URL: https://scsc.uk/r3352.pdf

Abstract:

Advanced technologies such as AI/ML pose novel assurance challenges requiring a through-life safety assurance process to guarantee safety. These challenges include issues that arise through human interaction with this technology, as mis-calibrated assumptions around these interactions often lead to system break-downs. During operation, the safety case provided at design time may fall short warranting modification of the system. Assumptions under which the system is expected to operate will be invalidated during operation requiring traceable engineering practices to uncover them. These must be monitored throughout the lifecycle with well-documented evidence providing the basis for the validity of the assumption. Methods are required to provide evidence to support assumptions around human interaction with this technology. We present an updatable process for de-signers and/or evaluators to proactively assess this interaction and document the context under which these assumptions hold true. Two tools, the Design Characteristics Diagram and the What’s Next Diagram, are used to uncover claims around human interaction by assessing how characteristics of the design support human anticipation and control in a variety of scenarios. This process can be repeated throughout the lifecycle synchronously to update evidence in the safety case as well as asynchronously to detect signals that prompt revision.