This 1-day seminar examined how safety-critical systems are tested, some types of new testing techniques and the resulting issues. It covered tools, techniques and limitations of testing methodologies. It also covered some historical failures of testing and the lessons learnt.

Meet your friends and share in the safety community!

It is recommended that delegates attend in London: for developing new contacts and networking with colleagues, for the chat over lunch, and especially at the end of the day where many fruitful discussions take place. At recent seminars delegates have stayed on for an hour to catch up with colleagues and discuss future developments, before moving on for a drink!

This event was held at the Guoman Hotel, Great Cumberland Place, 20 Great Cumberland Place, London, W1H 7DL
Venue details: https://www.guoman.com/great-cumberland-place/location-contact

Speakers include:

Paul Albertella, Codethink - "Bad Behaviour: Automating system-level safety testing"

Paul Butcher, Adacore - "Fuzzing for High Integrity Software"

John Dixon, Platinum QA Systems and Ian McDonald, Island Systems Ltd - "Testing in the real world: hard lessons learnt"

Phil Koopman, Carnegie Mellon University - "Bootstrapping Safety Assurance"

Bob Oates, Blackberry - "Cyber Security Testing Techniques"

Talk Abstracts:

Paul Albertella, Codethink - "Bad Behaviour: Automating system-level safety testing"

Testing is a key aspect of any safety project and test automation is a highly recommended practice. But how can we derive and automate tests to verify system-level safety requirements as well as component-level functional requirements, in a way that lets both safety and developer teams understand the relationships between them? And run tests on target hardware platforms? And trace from safety analysis to requirement to test to results? And integrate all this into a continuous integration workflow? I describe my work using the STPA methodology and open source tools such as Gitlab and Lava to solve these and other hard problems for safety critical systems testing.

Paul Butcher, Adacore - "Fuzzing for High Integrity Software"

Fuzzing? What is it, and can it be added to the inventory of verification approaches for future safety critical systems development? This talk will provide a comprehensive overview of fuzz testing from its early beginnings to the state-of-the-art.

John Dixon, Platinum QA Systems and Ian McDonald, Island Systems Ltd - "Testing in the real world: hard lessons learnt"

Testing In the Real World is a set of observations regarding the issues found in Software development and testing from the viewpoint of the presenters Ian McDonald and John Dixon who between them have some 60+ years of experience. A number of examples are given showing what can (and has) gone wrong, hence what to avoid in the future.

Phil Koopman, Carnegie Mellon University - "Bootstrapping Safety Assurance"

The expense and general impracticability of doing enough real-world testing to demonstrate safety for autonomous systems motivates finding some sort of shortcut. A bootstrapped testing approach is often proposed, using evidence from initial mishap-free testing to argue that continued testing is safe enough. In this talk I'll explain why pure bootstrapping based on testing exposure as well as arguments involving "probably perfect" bootstrapping expose public road users to undue risk. Moreover, phased deployments often used to argue safe update release have the same problem. An approach that bootstraps on the safety case rather than on vehicle testing is proposed as a potentially better alternative. While the examples given involve autonomous ground vehicles, the principles involved apply to any argument that safety will be demonstrated via a bootstrap testing process.

Bob Oates, Blackberry - "Cyber Security Testing Techniques"

In a highly interconnected world, cyber security is becoming a pre-requisite for safety systems. In this talk we will explore the testing pressures that cyber security adds to a system, and the impacts to safety critical systems in particular.

Speaker Bios:

Paul Albertella, Codethink

Paul Albertella is a consultant at Codethink, with more than 30 years of experience in the automotive, semiconductor and mobile device sectors. He's passionate about software engineering processes and the role that open source software and communities are playing in their evolution. His current focus is on safety and the use of Linux and open source tools in the Automotive industry. He is a certified functional safety practitioner and a member of the technical steering committee for the ELISA project (Enabling Linux in Safety Applications).

Paul Butcher, Adacore

Paul is a Senior Software Engineer and AdaCore's Lead Engineer in the UK for the HICLASS initiatives. His role is to facilitate the delivery of research, design and implementation for the UK aerospace sector. Before joining AdaCore, Paul was a consultant software engineer working for UK aerospace companies such as Leonardo Helicopters, BAE Systems, Thales UK and QinetiQ. Before becoming a consultant, Paul worked on the Typhoon platform and safety-critical software developments in the rail sector for BAE Systems and military UAVs for Thales UK. Paul graduated from the University of Portsmouth with a Bachelor's Degree with Honours in Computing and a Higher National Diploma in Software Engineering.

John Dixon, Platinum QA Systems

John is an Executive Level Test Manager and Consultant with 30 plus years of experience and extensive specialist experience across diverse sectors.  He has a proven track record in test programme management and guiding organisations and teams through the numerous stages of testing. He is well acquainted with advanced test management practises and frequently co-ordinates multiple programmes/projects within stringent deadlines and high quality constraints.

Phil Koopman, Carnegie Mellon University

Prof. Philip Koopman is an internationally recognized expert on Autonomous Vehicle (AV) safety whose work in that area spans over 25 years. He is also actively involved with AV policy and standards as well as more general embedded system design and software quality. His pioneering research work includes software robustness testing and run-time monitoring of autonomous systems to identify how they break and how to fix them. He has extensive experience in software safety and software quality across numerous transportation, industrial, and defense application domains including conventional automotive software and hardware systems. He was the principal technical contributor to the UL 4600 standard for autonomous system safety issued in 2020. He is a faculty member of the Carnegie Mellon University ECE department where he teaches software skills for mission-critical systems. In 2018 he was awarded the highly selective IEEE-SSIT Carl Barus Award for outstanding service in the public interest for his work in promoting automotive computer-based system safety. In 2022 he was named to the National Safety Council's Mobility Safety Advisory Group. He is the author of the book How Safe is Safe Enough: measuring and predicting autonomous vehicle safety (2022).

Ian McDonald, Island Systems Ltd

Ian is a test consultant having worked in hardware, system and software design. Ian‘s experience includes Radar, law enforcement, aviation communications, mobile phone and military communications.  Focusing upon Quality Assurance and Testing, Ian has over 30 years of experience helping companies to improve their QA, cutting delivery times, while getting products right and protecting companies’ reputation.

Bob Oates, Blackberry

Dr Robert Oates is a Senior IoT Cybersecurity Consultant for BlackBerry QNX. He holds an honorary professorship in Safety and Security with De Montfort University and has over a decade of experience embedding security principles into engineering processes. He was the Head of Cybersecurity for the world’s first commercial remotely operated ship (Project Sisu) and has acted as a consultant in the energy, nuclear, defence, aerospace, maritime, and automotive sectors.