Computer-based systems may fail catastrophically for a variety of reasons. Assurance processes for safety-related systems have focused primarily on the risks from random hardware faults, design faults, and operator error. Where failures are triggered by unpredictable events such as an unexpected combination of inputs, or one or more operator errors, safety analysts may assume such events are independent and/or stochastic. That assumption cannot be sustained if there is a credible threat of any form of cyber attack, because an attacker might be able to create any desired pattern of unlikely events. Consequently, no safety-related system can be considered adequately safe unless it is also adequately secure against cyber-attack, and this raises issues that need to be considered throughout the safety life-cycle.

Systems are collections of objects exhibiting joint behaviour. Sometimes this behaviour is anticipated, sometimes not. We have studied a number of types of complex systems and their failures, including electricity supply grids, motorways, the financial system, and air traffic control. We argue that the resilience properties of such systems are largely emergent. We illustrate the thesis through analysis of three electricity blackout events. We consider one event in detail and two others summarily.

The commonly practiced approach to the safety assessment of complex technology, including those in the High Speed Railways1 lacks a supporting theoretical foundation as a guiding and supporting backbone. In practice, this results in a confused, poorly conceived and often inadequate application of a mixed bag of methodologies, rules and standards that, due to the effort-intensive nature give a semblance of adequacy and completeness. In this uncharted and poorly structured landscape, demonstration of compliance with a given rule or standard is broadly regarded as adequate input to the safety assessment, potentially missing other analysis, effort and evidence. The key aim of the research outcomes presented in this paper is to give an overview of the principal requirements and qualities for robust, credible and systematic assessment to be supported by a host of relevant processes, rules, tools, codes of practice and standards.

The role of data in influencing the safe operation of systems is just as important but has not attracted the same level of attention; there is no standardisation and little guidance on how the risks associated with data should be managed. There has been a marginalisation of data (inadvertent or otherwise) as a contributor to accidents and incidents, and it is clear there is an “elephant in the room” (Hampton and Parsons 2015). The problem is becoming more acute as many types of data are now used to specify, deploy, configure, operate, test and justify safety systems, moreover the volume of data in systems is also growing at an unprecedented rate. This paper is a retrospective reappraisal of selected historical accidents from the aviation and marine sectors, but viewed afresh from a data perspective. The paper shows that we do have a data problem; in fact we’ve always had a data problem.

Modern smart devices such as pressure transmitters, controllers and valve actuators provide many key advantages but there are well-known difficulties in providing evidence to support the associated safety justifications. This paper reminds the reader of these difficulties but focuses on the need for an “As Low as Reasonably Practicable” approach and hence the requirement to use expert engineering judgement to weigh up the advantages/disadvantages of using a smart device against other possible options. The paper discusses the use of engineering judgement in safety justifications for installing smart devices and highlights that there are more than just software faults to consider. Issues covered include: allocation of ‘best estimate’ reliability data for use in Probabilistic Safety Assessments, ‘proven in use’ arguments and the importance of understanding the wider safety case picture.

The European aviation industry is experiencing wide-ranging change including introduction of new technologies and operational concepts, while also facing demands for higher levels of safety performance. Existing approaches to gaining approval are often perceived as a barrier to adopting innovation and change; they can also miss significant interactions between parts of the system. The EC-funded ASCOS Project has developed a method and supporting tools to address these challenges. The ASCOS Method uses modular safety arguments to provide a framework to integrate existing approval approaches while also providing the flexibility to adapt the approaches where necessary to enable the smooth approval of advances in aviation technology.

This paper reflects on the issues and opportunities raised by the use of Agile practices in the development of high-integrity software, based on the scientific literature, projects, and our own understanding of the relevant regulatory regimes, standards and markets. In particular, this paper considers the assumptions that underpin Agile practices and where these seem to conflict with the disciplines of high-integrity development. Conversely, we'll consider some opportunities where an Agile approach could be significantly improved by the adoption of high-integrity practices.

In 2010, Rolf Spiker approached one of us with a query from a client concerning advisory material in IEC 61508 on the statistical evaluation of software. We realised that there is a dearth of practical guidance for those who wish to evaluate critical software statistically. We believe statistical evaluation of software is an increasingly important assurance technique. We commence with a brief introduction to some of the simpler statistics and then consider discursively the issues which arise during evaluation.

There is a growing threat to the cyber-security of safety-critical systems. The introduction of Commercial Off The Shelf (COTS) software, including Linux, specialist VOIP applications and Satellite Based Augmentation Systems across the aviation, maritime, rail and power-generation infrastructures has created common, vulnerabilities. In consequence, more people now possess the technical skills required to identify and exploit vulnerabilities in safety-critical systems. Arguably for the first time there is the potential for cross-modal attacks leading to future ‘cyber storms’. This situation is compounded by the failure of public-private partnerships to establish the cyber-security of safety critical applications. The fiscal crisis has prevented governments from attracting and retaining competent regulators at the intersection of safety and cyber-security. In particular, we argue that superficial similarities between safety and security have led to security policies that cannot be implemented in safety-critical systems. Existing office-based security standards, such as the ISO27k series, cannot easily be integrated with standards such as IEC61508 or ISO26262. Hybrid standards such as IEC 62443 lack credible validation. There is an urgent need to move beyond high-level policies and address the more detailed engineering challenges that threaten the cyber-security of safety-critical systems. In particular, we consider the ways in which cyber-security concerns undermine traditional forms of safety engineering, for example by invalidating conventional forms of risk assessment. We also summarise the ways in which safety concerns frustrate the deployment of conventional mechanisms for cyber-security, including intrusion detection systems.

It is self-evident that we need an effective safety culture to avoid human error (and its consequences) in programming, yet many of us program as if safety is trivial, and if we just use the right tools it should be even easier. Although it is an unwelcome message, we are deceiving ourselves about how easy safety is, and this deception is self-serving, achieving nothing other than entrenching ignorance of error and its influence over us. The solution is called “resilience” and of the various techniques of resilience, mathematics and Formal Methods are basic tools for safety-critical programming — but they are not sufficient without a proactive commitment to be resilient. We provide a worked example that helps show that error is not “out there” as an abstract concept but deep inside us. Error is an unavoidable companion to our programming that we urgently need to master.

People often use the word 'competence' without understanding what it means even when it is vital for safety. This paper examines common definitions of competence to identify the individual components and understand the principles underlying the specification and assessment process. Safety management must facilitate the achievement and maintenance of competence for those developing and operating safety-related systems. The paper examines the theory and principles underlying the attainment and maintenance of competence providing a framework for a discussion on competence assurance. The specification of competence criteria is an important safety management activity and these are unique for different systems. This paper describes how competence criteria can be specified and assessed for safety-related systems. Safety assurance is ultimately based upon the competence of the people involved hence competence evidence is essential for the validity of any safety claim. The paper examines common safety assurance issues associated with competence and some suggestions are made on how to improve the validity of safety claims based upon competence.

The preparation of a Safety Assurance Case has been an integral part of the development of railway systems for many years, being one of the requirements of EN 50129. For automotive systems, ISO 26262 also mandates the creation of a Safety Case. Increasingly, Safety Cases are also being required for the certification of medical devices. Researchers have demonstrated that producing a Safety Case untainted by confirmation bias is extremely difficult, or even impossible. This seriously affects the level of confidence that can be placed in the Safety Case argument. This paper describes the results of an experiment to determine whether the notation used to represent the Safety Case argument influences the structure of that argument.

High Reliability Organisations (HRO) rightly invest significant effort to ensure their Safety Management System (SMS) is effective – for example ensuring appropriate and reliable engineering along with robust and resilient organisational arrangements. The benefits of these arrangements go far beyond mere compliance and contribute to high performance. As well as the organisations themselves, regulators, such as the Office for Nuclear Regulation (ONR), also recognise the importance of these arrangements, actively seeking evidence of their effectiveness within such themes as Leadership, Capable Organisation, Decision-Making and Learning from Experience. Whilst approaches to describing and assessing the visible elements of the SMS can be readily identified, there needs also to be a focus on the cultural and behavioural elements of the organisation’s commitment to safety and high performance. People are part of the system, rather than an unpredictable add-on. An effective SMS will guide and control behaviours with respect to safety. But how do you go beyond the arrangements in order to understand and be confident in the social and interpersonal influences – leadership and commitment, supervision, prioritisation of safety, and so forth? This paper addresses this need to understand and manage the link between the management system and the organisational safety culture, whilst recognising that neither can be considered independently, due to the interconnected nature of complex systems. Considerations are presented to prompt reflections of SMS and organisational culture with respect to compliance and performance.

This paper describes the invention of an Adaptive Safety Monitoring Function at Jaguar Land Rover within the context of the development of a vehicle propulsion system according to the principles of ISO 26262. It outlines the way that conventional safety monitoring software is currently used to detect and mitigate faults in a propulsion control system. It then describes the typical challenges and drawbacks in developing such software. The paper then presents the theoretical principles behind how the proposed software algorithm could address these problems by adapting over time to the control software that it is monitoring. It explains the key challenge of ensuring safety by only adapting in a manner commensurate with the adaption of a driver’s mental model to a change in relationship between vehicle acceleration and accelerator pedal input. It describes some of the practical problems encountered and solutions found during algorithm development before concluding with an outlook towards potential commercial applications.

In the field of automobile development, the recent automotive functional safety standard (ISO 26262) is now applicable. However, it is difficult to develop a system so as to comply with the standard. In this paper, I will focus on the concept phase of this standard because it includes the new ideas to keep a system safe. For example, the idea of "item" is one of them, and it means an abstraction of a system. It is important to think about safety in the very early phase because we can change the behaviour more readily than do after designing. On the other hand, it requires us to devise the new approach to calculating risk (a.k.a. ASIL) because we don’t have detailed information on an item in the early phase development. In this paper, we introduce our approach CARDION, and we show how we deal with those characteristics of the standard.

Increasing vehicle autonomy has been predicted by many commentators to offer the potential for a huge step change in road safety performance. Such systems will be constantly vigilant, will always respect the rules of the road and will never be tired or drunk. However, the evidence available remains very limited and most studies use only simplistic assumptions to quantify the expected effects. It is also likely that the technology will not be perfect and interactions with humans during a transition phase of mixed vehicle types and various stages of partial autonomy might create new risks. Thatcham has been working with the British motor insurers to investigate the potential effects of autonomous vehicles on motor claims. Changes in regulations and vehicle performance are under discussion, and Thatcham is contributing to the debate from a technical and safety standpoint.

This paper presents an overview of the challenges faced by engineers and companies in the automotive industry with regards to driver assist systems. Starting with an overview of different assist features made possible by the use of electronics in the last 30 years, the paper explores the most innovative systems that also account for characteristics of the environment surrounding the vehicle, such as objects in the vicinity, road characteristics, traffic signs, etc. and how connectivity is both an enabler and a source of concern for these features. In particular the relationship between safety and cyber security in the context of vehicle systems is discussed. New considerations on both subjects are required given that connectivity inside and outside the vehicle is becoming the norm and will have a great impact on future mobility services. Remotely controlled functions will be used as a case study to present a number of design drivers for these systems.

This paper explores some of the potential challenges facing the introduction of autonomous cars, especially in the absence of a clear definition of what an autonomous system is. The automotive industry is currently transitioning to position itself to introduce autonomous cars to the global market. This paper discusses autonomy from the standpoints of the aerospace and automotive industries, focusing on Unmanned Aerial Systems and autonomous cars.

An ontological model can be used to define a rich semantic landscape of terminology for a technical domain. The advantages of doing so are a high degree of self-consistency between the domain’s technical language terms, which increases the quality of their natural language usage. This paper explores an initial attempt to construct an ontological model of data safety terminology for the Data Safety Initiative project. In doing so it introduces the systematic approach ontological modelling brings to not only defining but also understanding a domain’s terminology.

It is not new applying a socio-technical approach to analyse the safety of complex systems. Early works from Reason (Reason et al. 1998), Rasmussen (Rasmussen 1997), and Leveson (Leveson 2004) already provided frameworks of socio-technical approach by identifying layers in a system actually involved in the control of safety. However, as systems are more and more complex, the challenge in these socio-technical approaches to system safety is now a problem of modelling. It is widely accepted that architecture is the foundation of good system engineering. Thus the model in a systems theoretic approach of system safety should be embodied in all components (both social and technical) in the system and their relationships to each other and the environment. The key objective is to explore whether safety analysis on a socio-technical system can benefit from model-based approach in which system engineers and safety engineers share a common model. To evaluate and demonstrate our approach, we developed a software tool to help the application of our approach. The case study analyses a tram accident: the derailment at East Croydon in February 2012. The analysis is purely based on the information from the official investigation report (RAIB 2012) so the architecture of entire organisation may not be represented completely; but it is adequate enough for the discussion of a general architecture-based approach to the safety of social-technical systems.

Safety-critical systems and software require particular care when their parameters have to be verified and validated, as any mistake may lead to a catastrophic scenario during their operating use. A recent technique, called formal data validation, enables an improvement in the level of confidence of the verification/validation process by associating a formal data model to the parameters, and by formally checking that these parameters fit within the model. This paper reports on the development and use of such tools for industrial railway applications.

All major trends in MedTech have one thing in common: medical devices are going to be more complex than ever - and so are the networks they belong to. A necessary measure to cope with this and other emerging challenges is a satisfactory risk management process. Today, manufacturers address safety hazards with a multitude of techniques, all of which are document-based approaches. This paper presents research on how applying model-based risk management could eliminate disadvantages that are endemic to existing methods, like uncertainty of coverage, incompatibility of professional mind-sets or typical bias-by-design flaws. Risk management, based on a structured, computerised model of both the physical product and its lifecycle, has the potential to improve processing in all stages. We explain how our concepts allow for comprehensive risk identification, interconnected expert judgements and standardisable classification for better risk evaluation; they also help enforcing risk treatment by reducing process cost.

Software-induced memory corruptions can be caused by stack overflows, run-time errors such as invalid pointer accesses or buffer overflows, and data races. They can trigger software crashes, invalidate separation mechanisms in mixed-criticality software, and are a frequent cause of errors in multi-core applications. In contrast to hardware faults, software-induced memory corruptions are always systematic errors, and hence it is possible to formally prove their absence. Abstract interpretation is a formal method for static program analysis which supports formal soundness proofs (it can be proven that no error is missed) and which scales. This article gives an overview of abstract interpretation and its application to prove the absence of stack overflow, run-time errors, and data races, and reports on practical experience with the tools StackAnalyzer and Astrée.

A sub-sea glider is a type of autonomous underwater vehicle propelled by changes in buoyancy and is typically used for oceanographic research and monitoring. The BRIDGES (Bringing together Research and Industry for the Development of Glider Environmental Services) project is funded under the European Commission Horizon 2020 programme and aims to develop tools to further understanding, monitoring and sustainable exploitation of the marine environment. BRIDGES has the objective to improve sub-sea glider technology, system integration, operational management and standardisation. This paper summarises the research undertaken by BMT Isis as part of the BRIDGES consortium into understanding the safety benefits and applicability of standardisation to the design, manufacture and operation of gliders. The paper explores to what extent standardisation and guidance should be introduced into what is a relatively new and evolving domain. It also investigates what opportunities there are to transfer approaches from other domains; maximising the potential safety benefits whilst encouraging innovation.

FPGA usage within high integrity systems is becoming both more popular and more complex. One of the challenges of putting an FPGA in a high integrity system is the cost of verifying its correct operation, and this is made significantly more difficult by the increasing complexity of FPGA applications. For a typical DO-254 Level A aerospace FPGA application, at least 50% of the overall effort and engineering budget is spent on verification activities. As design decisions are set in stone early in the development process, it is common to discover unexpected verification problems when it is too late to do anything about it. This paper seeks to explore and quantify the effect of various architectural and design structures on their ‘testability’ for FPGA systems in high integrity applications, as well as identifying test based mitigations for common problems. Using anonymised data from Resource Group’s high integrity customers, a study of varied design structures has been analysed into a set of testability rules and a summary of their respective impact on the effort of verification.

The investigation of aircraft accidents is an important source of learning for the aviation industry and beyond. Whilst protected by international standards and European and national legislation, and emulated across other sectors, aircraft accident investigation faces a number of challenges. These include: technical challenges surrounding complexity, concerns about the protection of sensitive evidence, maintaining investigator currency and the maintenance of investigator competency. Faced with such challenges, is it time that aircraft accident investigation benefited from a change in approach or has it evolved to meet the challenges that have surfaced along the way?