
2
we speak of fully automated vehicles when the driver no longer acts as a fallback [20]. In this case, the automated
systems should take over vehicle control permanently and should act safely in any environment.
The international organisation for standardisation (ISO) provides the ISO/PAS(Publicly Available Specification)
21448 to ensure the safety of the functionalities [21] and the ISO 26262 to ensure, that no hazards are caused by
technical failure [22]. ISO/PAS 21448, safety of the intended funcionality (SOTIF), focuses on predictable misuse by
the driver, as well as accidents that are explicitly not caused by component failure, but situations that were not planned
for during development. Nevertheless, this standard does not provide a detailed strategy for identifying functional
deficiencies. In contrast, ISO 26262 focuses on safety in terms of intrinsic safety (protection of the environment from
the product), therefore functional safety. It is an ISO standard for safety-related electrical/electronic systems in
motor vehicles. It is not enough that the function has been executed correctly, but it must also be ensured that the
function has been executed in the correct context. For example, an airbag must not be triggered when driving too fast
over speed bumps. However, the minimum requirements for safeguarding an autonomous vehicle are not adequately
described by this standard. The goal is to minimize risks to a level that is “acceptable to society”.
Consequently validation of autonomous vehicles cannot be based on existing ISO standards and need extension.
Various approaches have been developed starting with Advanced Driver Assistant Systems (ADAS), where function-
based approaches and real-world testing operate well. In the function-based approach, requirements are defined for the
operating system that are tested by simulation or on the test track. In real-world testing, a mileage-based evaluation
of functionality from field tests with a driver is performed. For both methods, validating full autonomous systems is
economical infeasible in its current state. Furthermore there is the shadow mode, presented by Wang and Winner [23],
in which the automated driving function is excecuted passively in series production vehicles. The driving function
receives the (real) information from the sensors, but will not act. Its actions are evaluated afterwards by simulation.
Considering the real world as an open parameter space, with an infinite number of traffic events, the scenario-based
approach tries to identify these traffic events and describe them in scenarios. It also attempts to exclude non-relevant
traffic events, where neither actions nor events are observed, and to cluster similar traffic events into a representative
scenario [24]. However, this leads to the question of how to find the set of representative scenarios. A good overview
of the problem of identifying critical scenarios is provided by Neurohr et al, Riedmaier et al and Zhang et al. [25–27].
Menzel et al. [28] distinguish three categories of scenarios: functional, logical and concrete scenarios. In the case
of functional scenarios, the scenario space, in the same way as traffic events, is described on a sematic level by ”a
linguistic scenario annotation” [28]. For the logical scenarios, this is described at the state space level. Entities and
their relationships are described using parameter ranges in the state space and optionally specified using correlations
and numerical relationships. A traffic event is finally mapped explicitly to the state space given a concrete scenario.
Entities and their relationships are described using concrete values for each parameter. According to Zhang, the
identification of critical scenarios can be done on all three levels of abstraction. This requires a clear definition of the
operational design domain (ODD), a definition of the operating conditions under which an AD system is attempted
to operate, and ”the formulation of a functional scenario to a logical scenario”.
The German research project Pegasus1followed the approach in [29] of modeling the environment in layers and
extended it to 6 layers describing the environment of a highway. In the follow-up project VVM2, the 6-layer model
was refined, extended to the urban environment, provided with guidelines [30].
To describe the operating conditions of an AD system, the parameters of the six layers are used, such as weather,
number of road users, number of lanes or speed of a cyclist. The assessment of whether a scenario is critical or not
is based on the concrete values for these stochastic parameters. Some studies already consider realistic parameter
distributions [27] obtained from real-life driving databases.
In scenario-based testing, parameter distributions can be used to support the search for critical scenarios within
the scenario space. In doing so, they serve to model the mutuality of the scenarios. In [31], a risk index based
on a Gaussian Mixture Model is used to efficiently select critical traffic conditions. Thereafter, the scenarios are
replicated via simulation and used, for example, to evaluate the criticality of the AD system. For the simulation
of the scenarios, real trajectories, extracted from real data are used. In the same way, characteristic criteria of
scenarios are taken from the real data, e.g. parameter distributions or parameter dependencies, and used to reproduce
trajectories or parameters. Thus, parameter distributions are applied in estimating the failure rate of a scenario. For
example, Wagner et al [32] predict the behavior of road users based on conditional distributions obtained by analyzing
naturalistic driving study called euroFOT (large-scale European Field Operational Test on Active Safety Systems)
and determine the criticality for each predicted event.
1Pegasus - Project for the Establishment of Generally Accepted quality criteria, tools and methods as well as Scenarios and Situations
for the release of highly-automated driving functions, see https://www.pegasusprojekt.de/en/home
2VVM - Verification and Validation Methods for Level 4 and 5 Automated Vehicles, see https://www.vvm-projekt.de/