2. Safety validation of an ATM automated system

2.1 The problem to be solved

2.2 Safety Validation Criteria

2.3 Building a safety case for an automated system by a manufacturer

2.1 The problem to be solved

In this document, we are concerned with the question: How do we meet the requirements stemming from general principles and operational concept, regarding the automated system (i.e. the automated part of the advanced ATM system, but keeping in mind that it is to be used by human beings). More precisely, the problem addressed is the following:

An automated ATM system, or an update to an automated system, has to be developed. A list of requirements is available, including some requirements related to safety or impacting on safety. What methodology should be used to validate safety of this system?

Needs raised by this question may be classified into:

· need of assessable safety validation criteria for the automated system

· need of safety validation methods to assess performance against these criteria

2.2 Safety Validation Criteria

Safety criteria must be translated into metrics before safety can be assessed. However it is impossible to prove rigorously that an ATM system will have some level of safety in the future. So direct safety metrics, such as the number of fatalities by passenger-kilometre, have to be ruled out, except possibly for already operational systems.

The only way to assess safety is to find factors that have a (more or less direct) impact on safety, and then, when possible, to define metrics for each of these factors. Such safety factors are, for example, reliability or availability of the automated system.

But even for such metrics, it is often difficult to get figures. Therefore practical metrics often has to be still more indirect, and this analysis must be iterated until measurable indicators are found. For example, such measurable indicators may be test coverage, code complexity (measured through standard metrics), methods used for development and for ensuring safety and to what degree they were used, etc. Feedback from operational systems is required for this analysis (see Section 4).

Then, different methods have to be used for assessing safety of the system, depending on chosen criteria and metrics. The methodological framework proposes a number of methods to be used for ensuring safety.

Some characteristics which make safety validation of ATM automated systems very difficult are as follows:

· ATM automated systems are always complex, because the problems they deal with are complex, as are nuclear power plants, oil production platforms, etc.

· this complexity makes the development of complete systems from the beginning very costly; therefore, most systems are based on previous systems, and also use already developed COTS products, which has a strong impact on safety validation

· the system is used by human controllers, and the "man-machine coupling" is difficult to validate for safety, although it has a major impact on safety

· interfacing of the automated system with the external world is complex, due to the variety of current systems and people involved.

· proving rigorously that an ATM system has some safety characteristics is impossible (e.g. it is impossible to prove that there will be no accidents, or less than 1 accident every 15 years, in traffic controlled by this system). However it is possible to express the system safety requirements and assess the system against them. Therefore, the way of expressing safety requirements is of the utmost importance, and it is currently very heterogeneous.

Throughout the whole report, priority has been given to practicality, and to methods experimented in the ATM domain, in order to ensure that what is proposed is actually applicable, based on experience and feedback available.

2.3 Building a safety case for an automated system by a manufacturer

2.3.1 Introduction

A safety case is a consistent and coherent set of arguments and evidence that the system meets or exceeds the system safety standard or target, used to justify the safety of a system.

The safety case proposed in this part is of the classical type, as opposed to the "modern" safety case, to be built by the ATM service provider, and which is more based on the operational use and operational conditions of use of the system

WP5 recognises three approaches for obtaining evidence for justifying this safety:

· Approach 1 : Use of development standards

· Approach 2 : Independent assessment

· Approach 3 : Reverse engineering

The methodology proposed in this report is based on all of those three approaches: use of development standards, assessment, and reverse engineering (as this document does not deal with responsibility issues, it is not concerned with the independence of assessment).

The major output in the methodological framework is the safety case. Inputs to the safety case are outputs of activities recommended in the methodology, which make it possible to get the required set of arguments. Inputs of these activities are very variable. Some important inputs and outputs are summarised in the figure of section 3.4.1.

This section briefly discusses how to organise this set of arguments into a coherent safety case.

2.3.2 Organisation of the safety case

The safety case proposed, after WP5, is based on a safety argument that describes:

· the safety requirements,

· the evidence available and

· the justification why the evidence should be accepted as meeting the requirements.

The first point (safety requirements) is provided by the higher level safety management of the system, and complemented by new safety requirements found necessary during analysis of these requirements for development of the automated system.

The second point (evidence) includes:

· the safety management plan and other safety-related documents and hazard and safety log files.

· log documents showing that activities required by the safety plan have been performed.

· design documents and other documents produced during development, and which can impact on the safety of the system; this especially includes test reports and review reports

It is important to define the structure of the safety case at the beginning of the programme, so that the process is organised in such a way that the required evidence emerges from activities. The general structure should be standard.

The third point (justification from evidence that safety requirements are satisfied) often proves very difficult or even impossible to really achieve (e.g. it cannot be proven that the accident rate will be inferior to a threshold, or that the average time between failures is above some value). However, evidence should be available to determine whether the controls and mitigation measures required to meet the safety requirements are in place and operating correctly. Therefore, the recommended contents includes:

· justification using standard documents providing equivalence between the kind and level of supporting evidence and resulting safety characteristics. These standard documents should be based on experience and describe what evidence is deemed necessary to justify each kind and each level of safety requirements.
Of course, this kind of justification is possible only if safety requirements are always expressed in a standard way. This method should be the main way to justify safety, as it is the most objective and most standard one.

· specific arguments in the cases:

¨ when the above method is not applicable (e.g. when no standard is available for this case (standard not generic enough, or too generic, or not available at all),

¨ or when methods stated in the standard have not been applied for whatever reason, but good other means were possible in this specific case, etc.