Intramorphic Testing

2025-05-05 0 0 1.73MB 9 页 10玖币

侵权投诉

A New Approach to the Test Oracle Problem

Manuel Rigger

rigger@nus.edu.sg

National University of Singapore

School of Computing

Singapore

Zhendong Su

zhendong.su@inf.ethz.ch

ETH Zurich

Department of Computer Science

Switzerland

Abstract

Atest oracle determines whether a system behaves correctly

for a given input. Automatic testing techniques rely on an

automated test oracle to test the system without user inter-

action. Important families of automated test oracles include

Dierential Testing and Metamorphic Testing, which are both

black-box approaches; that is, they provide a test oracle that

is oblivious to the system’s internals. In this work, we pro-

pose Intramorphic Testing as a white-box methodology to

tackle the test oracle problem. To realize an Intramorphic

Testing approach, a modied version of the system is created,

for which, given a single input, a test oracle can be provided

that relates the output of the original and modied systems.

As a concrete example, by replacing a greater-equals operator

in the implementation of a sorting algorithm with smaller-

equals, it would be expected that the output of the modied

implementation is the reverse output of the original imple-

mentation. In this paper, we introduce the methodology and

illustrate it via a set of use cases.

CCS Concepts: •Software and its engineering →Soft-

ware verication and validation.

Keywords:

test oracle problem, white-box testing, automated

testing

ACM Reference Format:

Manuel Rigger and Zhendong Su. 2022. Intramorphic Testing: A

New Approach to the Test Oracle Problem. In Proceedings of the

2022 ACM SIGPLAN International Symposium on New Ideas, New

Paradigms, and Reections on Programming and Software (Onward!

’22), December 8–10, 2022, Auckland, New Zealand. ACM, New York,

NY, USA, 9pages. hps://doi.org/10.1145/3563835.3567662

Permission to make digital or hard copies of part or all of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for prot or commercial advantage and that copies

bear this notice and the full citation on the rst page. Copyrights for third-

party components of this work must be honored. For all other uses, contact

the owner/author(s).

Onward! ’22, December 8–10, 2022, Auckland, New Zealand

ACM ISBN 978-1-4503-9909-8/22/12.

hps://doi.org/10.1145/3563835.3567662

1 Introduction

The test oracle problem is one of the greatest challenges for

software testing [

]. A test oracle is a mechanism to check the

correctness of a system’s output for a set of inputs [

]. Given

that software constantly evolves and typically lacks a formal

specication of the expected behavior, general test oracles

are dicult to obtain. However, partial test oracles are still

useful, as they can validate the output for some inputs [

In the most straightforward case, partial test oracles are

specied in the form of regression tests, where developers

specify the expected output of a test case.

A number of approaches have been proposed to allevi-

ate the test oracle problem through partial test oracles that

can be applied in an automated setting (e.g., for automat-

ically generated tests) [

]. The most inuential

ones are dierential testing [

] and metamorphic testing [

which, similar to regression testing, approach the problem

in a black-box manner; that is, they do not require access to

the program’s source code or internals. Dierential testing

compares the output of various systems that implement the

same semantics; a mismatch between outputs for the same

test case indicates that at least one system is aected by a

bug. Metamorphic testing refers to a technique where, based

on an existing input to a system and its output, a new input

can be created for which the expected output is known.

This work presents Intramorphic Testing as a general

methodology toward the test oracle problem to complement

dierential testing and metamorphic testing. The core idea

of Intramorphic Testing is to modify one or multiple com-

ponents of the system under test (SUT) in a way so that the

relationships between the outputs of the modied and orig-

inal systems for a set of inputs are known. Thus, dierent

from dierential testing and metamorphic testing, Intramor-

phic Testing is a white-box approach that assumes access to

and knowledge of the system’s internals. Accordingly, we

expect concrete techniques to be realized by developers—or

automatically derived—rather than implemented by testers.

In this paper, we present the general idea of Intramorphic

Testing and illustrate it with several concrete examples.

We believe that Intramorphic Testing techniques are al-

ready being realized and used by developers as part of an

eort to create testable code. However, they might have

been viewed as an undocumented implementation detail of

arXiv:2210.11228v1 [cs.SE] 20 Oct 2022

Onward! ’22, December 8–10, 2022, Auckland, New Zealand Manuel Rigger and Zhendong Su

a test suite, rather than an instance of a more broad testing

methodology. This paper aims to address this by unifying

such existing and future techniques under a common name

and abstract framework, thus fueling exchange and develop-

ment of Intramorphic Testing techniques.

In summary, this paper contributes the following:

•

Intramorphic Testing, a general conceptual white-box

approach to tackling the test oracle problem;

•

a conceptual comparison with regression testing, dif-

ferential testing, and metamorphic testing;

•examples that illustrate the idea.

2 Background and Motivation

Test oracles.

To the best of our knowledge, the term test

oracle was coined by Howden in 1978 [

]. Since then, a

number of approaches to tackle the problem have been pro-

posed, which were summarized in surveys by, for example,

Barr et al. [

] or Pezzè et al. [

]. Metamorphic testing was

proposed by Chen et al. in a technical report in 1998 [

]. Vari-

ous concrete metamorphic testing techniques were proposed

that were subsequently surveyed by, for example, Segura et

al. [24] or Chen et al. [5].

Terminology.

Originally, a test oracle was dened to vali-

date a system’s output for a set of inputs [

]. This view is

restrictive, given that an input to the program might include

changes to the device or environment. Similarly, rather than

a directly-observable program output, non-functional obser-

vations include the program’s performance or changes to

the device’s state. Thus, Barr et al. [

] used stimuli for inputs

and observations for outputs to account for various testing

scenarios. We continue to use the original terminology of

inputs (denoted as

𝐼

) and outputs (denoted as

𝑂

), but refer

to them in the general sense of stimuli and observations. We

will denote the program under test as 𝑃.

Motivating Example.

To outline the existing techniques

and our idea, let us assume a specic use case, namely that

we want to test the implementation of one or multiple sort-

ing algorithms.

Let us assume that we implemented multi-

ple sorting algorithms such as

bubble_sort()

insertion_sort()

and

merge_sort()

. Let us also assume that we made a mistake

when implementing the in-place

bubble_sort()

algorithm, as

illustrated in Listing 1; the last array index in the code listing

should be

, rather than

. In the subsequent paragraphs,

we discuss how both instantiations of existing techniques as

well as an instantiation of the proposed Intramorphic Testing

technique could nd the bug. In practice, we expect that In-

tramorphic Testing will be realized that can nd bugs that are

overlooked, or dicult to nd, by other testing approaches.

A Jupyter Notebook with the code examples presented in this paper is

available at hps://doi.org/10.5281/zenodo.7229326.

Regression Testing.

Regression testing aims to ensure that

changes do not introduce bugs into the program through

manually written tests in which the developer species the

expected output. One common way of implementing regres-

sion tests is by implementing unit tests, where a specic unit

is tested in isolation.

To test

bubble_sort()

and the other sorting algorithms, we

could introduce unit tests with both typical inputs as well as

boundary values. Listing 2shows a test case that triggers the

bug; sorting an array

[3, 1, 2]

incorrectly results in

[1,←↪

2, 1]

, which does not match the expected array

[1, 2, 3]

thus revealing the bug. Note that the test does not assume

access to the system’s internals; unit testing is a black-box

approach that could also be applied without access to the

source code. While unit testing is eective and widely used,

tests are typically implemented manually, and the developer

needs to specify the expected outcome of the test case.

Dierential Testing.

Dierential testing validates a set of

systems that implement the same semantics, by comparing

their output for a given input. As illustrated in Figure 1, given

input

𝐼

and equivalent systems

𝑃1

𝑃2

, . . . ,

𝑃𝑛

, dierential test-

ing validates that

∀𝑖, 𝑗

𝑃𝑖(𝐼)=𝑃𝑗(𝐼)

. Dierential testing

has been applied to a variety of domains, such as, C/C++

compilers [

], Java Virtual Machines (JVMs) [

], database

engines [

], debuggers [

], code coverage tools [

], sym-

bolic execution engines [

], SMT solvers [

], and Object-

Relational Mapping Systems (ORMs) [26].

As illustrated by Listing 3, we can apply dierential testing

by comparing the sorted arrays for multiple sorting algo-

rithms for the same input array. Given that the test oracle

requires no human in the loop, it can be eectively paired

with automated test generation; for example, in the listing,

we generate random arrays as test input. Note that the loop

does not terminate; in practice, it would be reasonable to

set a timeout or run the tests for a xed number of itera-

tions. For an input array like

[3, 1, 2]

, dierential testing

reveals a discrepancy between the output of the sorting algo-

rithms, demonstrating the bug. Similar to regression testing,

dierential testing is a black-box approach; for example, we

could have also compared sorting algorithms implemented

in various languages based on checking their output alone.

Metamorphic Testing.

Metamorphic testing uses an input

to a system and its output to derive a new input for which

a test oracle can be provided via so-called Metamorphic Re-

lations (MRs). This is illustrated in Figure 1. Given an input

𝐼

and

𝑃(𝐼)=𝑂

, a follow-up input

𝐼′

is derived, so that a

known relationship between

𝑂

and

𝑃(𝐼′)=𝑂′

is validated.

Metamorphic testing is a high-level concept and nding ef-

fective MRs is often challenging; MRs for testing various

systems such as compilers [

], database engines [

SMT solvers [

], Android apps [

], as well as object detec-

tion systems [28] have been proposed in the literature.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

IntramorphicTestingANewApproachtotheTestOracleProblemManuelRiggerrigger@nus.edu.sgNationalUniversityofSingaporeSchoolofComputingSingaporeZhendongSuzhendong.su@inf.ethz.chETHZurichDepartmentofComputerScienceSwitzerlandAbstractAtestoracledetermineswhetherasystembehavescorrectlyforagiveninput.Automatic...

收起<<

Intramorphic Testing.pdf

共9页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Intramorphic Testing

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: