Statistical Testing: Qualitative Arguments for a Quantitative Outcome?

Abstract

Statistical testing is a technique accepted by functional safety standard BS EN 61508-2:2010 for higher integrity software-based systems, either as an alternative to black-box testing and field experience or as a technique to support code verification.  It has historically been seen as relevant good practice in the UK nuclear industry as a post-validation activity.  Its primary usage has been to provide evidence to support an upper bound for the numerical reliability claim on a programmable system.  Using past published examples of statistical testing and the authors’ own experience of implementing statistical testing systems, this paper evaluates the practicability of using this approach for complex systems like reactor protection systems, covering each of the core conditions underpinning the execution of the technique (accurate test operating environment, test-case independence, determination of pass/fail and zero test failures).  We examine divergence from these assumptions, and how this may undermine the quantitative claims of the technique.  Additionally, we discuss the validity of the test outcome for a complex system, and in particular whether statistical testing’s implicit model of a software-based system reflects what we know about software-involved faults due to its focus on system reliability, and not on reliability of safety functions.  We conclude that while concepts and approaches used to implement statistical testing can provide useful qualitative evidence, statistical testing itself should not be considered a practicable method for obtaining high-confidence quantitative reliability estimates for complex systems like reactor protection systems, and as a result the benefits of other techniques should be considered.