RoseMatcher Identifying the Impact of User Reviews on App Updates

2025-05-02 0 0 1.19MB 20 页 10玖币

RoseMatcher: Identifying the Impact of User Reviews on App Updates

Tianyang Liua,b, Chong Wanga,c,∗, Kun Huanga, Peng Lianga,∗, Beiqi Zhanga, Maya Danevac, Marten van Sinderenc

aSchool of Computer Science, Wuhan University, 430072 Wuhan, Hubei, China

bDepartment of Computer Science and Engineering, University of California San Diego, La Jolla, California 92093, United States

cFaculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, 7500 AE Enschede, The Netherlands

Abstract

Context: The release planning of mobile apps has become an area of active research, with most studies centering on app analysis

through release notes in the Apple App Store and tracking user reviews via issue trackers. However, the correlation between these

release notes and user reviews in App Store remains understudied.

Objective: In this paper, we introduce RoseMatcher, a novel automatic approach to match relevant user reviews with app release

notes and identify matched pairs with high conﬁdence.

Methods: We collected 944 release notes and 1,046,862 user reviews from 5 mobile apps in the Apple App Store as research data

to evaluate the eﬀectiveness and accuracy of RoseMatcher, and conducted deep content analysis on matched pairs.

Results: Our evaluation shows that RoseMatcher can reach a hit ratio of 0.718 for identifying relevant matched pairs, and with the

manual labeling and content analysis of 984 relevant pairs, we identify 8 roles that user reviews play in app updates according to

the relationship between release notes and user reviews in the relevant matched pairs.

Conclusions: Our ﬁndings indicate that both app development teams and users pay close attention to release notes and user reviews,

with release notes typically addressing feature requests, bug reports, and complaints, and user reviews oﬀering positive, negative,

and constructive feedback. Overall, the study highlights the importance of the communication between app development teams and

users in the release planning of mobile apps, with relevant reviews tending to be posed within a short period before and after the

release of release notes, with the average time interval between the post time of release notes and user reviews being approximately

one year.

Keywords: User Reviews, Release Notes, App Store, Natural Language Processing

1. Introduction

With the rapid progress in mobile techniques and smart-

phones, the number of mobile applications (apps for short) has

risen dramatically in recent years. As of the second quarter

of 2022, Android users were able to choose between 3.5 mil-

lion apps, making Google Play the app store with the biggest

number of available apps. The Apple App Store is the second-

largest app store with roughly 2.2 million available apps for

iOS [1]. App stores now become the primary data source to

construct app datasets for the research and practice on app de-

velopment, evolution, and maintenance [2].

In these app stores, app vendors regularly deliver oﬃcial re-

lease notes with the new releases of their apps to highlight the

key or essential updates of its current version, as shown in Fig-

ure 1 (a). Meanwhile, users freely post their reviews on the apps

they use in the app store, including praises, complaints, fea-

ture requests, and recommendations [3], as shown in Figure 1

∗Corresponding author

Email addresses: til040@ucsd.edu (Tianyang Liu),

cwang@whu.edu.cn (Chong Wang), hunk_fe@whu.edu.cn (Kun Huang),

liangp@whu.edu.cn (Peng Liang), zhangbeiqi@whu.edu.cn (Beiqi

Zhang), m.daneva@utwente.nl (Maya Daneva),

m.j.vansinderen@utwente.nl (Marten van Sinderen)

(b). Several existing studies have reported that it is essential for

app vendors to regularly release new versions to ﬁx bugs or in-

troduce new features [4], since user dissatisfaction can quickly

lead to the fall of even popular apps [5]. This implies that con-

stantly monitoring and capturing, and proactively meeting user

needs could lead to a successful app. For example, as the under-

lined text in Figure 1 shows, a user complained about the crash

when he/she added a new preset or created a tempo change (pre-

set and tempo are terms used in the music ﬁeld), and this bug

was ﬁxed in the subsequent release of this app.

This circumstance shows that user reviews could be the

potential evolutionary requirements for for app development

teams to update apps, and write the updating details in release

notes for users.

Although lots of users feedback is generated in diﬀerent plat-

forms, such as Apple App Store [3] or social media [6], only

35.1% user reviews were reported to contain valuable informa-

tion for app development teams [7]. Manual extraction of key

valuable information from user feedback would inevitably be

time-consuming, laborious, and clumsy. In recent years, many

researchers studied the ﬁltering of informative user reviews [7],

the identiﬁcation of relevant user feedback [8], and clustering

and prioritization of user requirements in user reviews [7, 9].

These studies proposed various methods to extract critical in-

Elsevier May 16, 2023

arXiv:2210.10223v4 [cs.SE] 14 May 2023

Figure 1: Exemplary release note (a) and user review (b) from app TonalEnergy Tuner & Metronome in Apple App Store.

formation for app evolution from large amounts of data, but no

analysis was conducted on those user reviews from the perspec-

tive of app release notes, i.e., to explore the echoing relationship

between user reviews and release notes.

To this end, we propose RoseMatcher (Release nOte and

uSer rEview Matcher), which is a novel approach that matches

colloquially written user review sentences with formally written

release note sentences. With RoseMatcher, we can obtain high-

conﬁdence matched pairs of relevant user reviews and release

notes, thus signiﬁcantly reducing the manual annotation eﬀort.

Based on these matched pairs, we detect an echo relationship

between app release notes and user reviews, and we explore the

roles of user reviews in app updates from this mutual response

and conduct analysis at the level of their posting time.

To the best of our knowledge, seldom research explores the

relevance between user reviews and app release notes. In this

paper, we focus on those user reviews that are deemed to be

relevant to speciﬁc release notes. First, we deﬁned eight roles

user reviews take in app updates based on the content associa-

tion between release notes and user reviews. Subsequently, we

explored the attention app vendors pay to user reviews and the

attention users pay to release notes in terms of the temporal di-

mension, based on the diﬀerence between the release time of

release notes and the post time of user reviews.

The main contributions of this study are listed below:

•To detect the relevance between oﬃcial release notes and

informal user reviews, we propose RoseMatcher to mine

the most relevant reviews for app release notes by combin-

ing semantic-sensitive and keyword-based matching algo-

rithms. RoseMatcher can detect semantically similar and

keyword-identical texts and solve the problem of the low

relevance rate of high-similarity matched pairs suggested

by the single model, which dramatically reduces the over-

head and cost of manual annotation.

•To explore the roles user reviews played in app updates, we

manually analyze and compare the content of the relevant

user reviews matched by RoseMatcher and their relevance

with release notes, and ﬁnally, we deﬁne eight diﬀerent

roles user reviews play in app updates.

•To explore the attentiveness of the app development teams

to user reviews and users’ attentiveness to app release

notes posted in the app market, we analyze and visualize

the time interval between the release time of the release

notes and the post time of their relevant user reviews. To

our best knowledge, no prior study analyzed release notes

and their relevant user reviews in the temporal dimension,

and our study ﬁlls the gap in this research area.

The rest of the paper is organized as follows. Section 2

elaborates our proposed methodology for identifying relevant

matched pairs between user reviews and app release notes, in-

cluding overview in Section 2.1, data pre-processing and selec-

tion in Section 2.2, algorithms for matched pair identiﬁcation in

Section 2.3, and speciﬁc methodology for matched pair analysis

in Section 2.4.

Section 3 presents the data collection and experimental re-

sults. Section 4 elaborates the corresponding interpretation de-

rived from the experimental results, and Section 5 explicitly

states the implications from diﬀerent perspectives. Limitations

are explicitly given in Section 6. In Section 7, we brieﬂy review

the related work on user reviews and app release notes. Finally,

Section 8 concludes the paper and describes the possible future

work of our study.

2. Research Methodology

2.1. Overview of our Approach

In our approach, researchers can get high-conﬁdence

matched pairs of release note sentences and user review sen-

tences for a more in-depth study, this process mainly contains

the following three steps as shown in Figure 2.

(1) Data Selection and Processing. focuses on data selec-

tion, pre-processing, and ﬁltering of the acquired dataset to en-

sure a high quality of data input for RoseMatcher and thus pre-

vent invalid calculation, which is elaborated in Section 2.2.

(2) Matched Pairs Identiﬁcation (RoseMatcher). The sec-

ond step is the core of our entire approach – the RoseMatcher.

As described in Section 2.3, we use deep learning models to en-

code sentence-level release notes and user reviews, in order to

get matched pairs by sentence similarity calculation and rank-

ing.

(3) Matched Pairs Analysis. The third step is the subse-

quent analysis of the suggested high-conﬁdence matched pairs.

In Section 2.4, we elaborate on our speciﬁc methodologies for

analyzing matched pairs.

2.2. Data Selection and Processing

Due to the huge amount of computation, we need to make

sure we have high-quality input data, which can heavily reduce

unnecessary computational overhead. The data selection, ﬁlter-

ing, and processing is to ensure the quality of our data as much

as possible before they are input to RoseMatcher, thus making

our algorithm more eﬃcient.

2.2.1. Selection Criteria of Mobile Apps

As reported in [10], nearly 60% of the mobile apps in app

stores are updated in a non-updating pattern, i.e., the same re-

lease notes are pushed repeatedly. Meanwhile, we observed that

some apps usually release notes in a perfunctory manner with

broad and repetitive statements, rather than speciﬁc updating

details. As shown in Figure 3, for example, the release notes of

Facebook and Mcdonald’s are deemed to be low-qualiﬁed be-

cause they seldom convey speciﬁc or valuable information for

RE researchers. Therefore, we introduced the following four in-

clusion criteria (IC) to select mobile apps with highly qualiﬁed

release notes. Note that in this paper, it works as the ﬁrst-round

selection for the construction of our research dataset.

IC1.1 The app should be released for at least three years.

IC1.2 The release notes of this app should have less than 80%

(suggested) repetition rate after splitting into sentence lev-

els.

IC1.3 The app should have a suﬃcient amount of release notes

and user reviews.

2.2.2. Data Processing

Our research dataset consists of app release notes and re-

views of the same apps. Generally, the number of user reviews

is much greater than that of the release notes. Meanwhile, app

release notes are often written in a more regular and structured

way. Therefore, our data processing is divided into two parts:

user review processing and release note processing.

Task 1: Data Processing for User Reviews is conducted by

the following three steps.

Step 1.1: Sentence Splitting. One user review usually de-

scribes user feedback from multiple aspects with several sen-

tences. Taking the user review in Figure 4 as an example, it

describes two types of feedback: the ﬁrst sentence praises the

app, while the second sentence requests a new feature. In or-

der to simplify the analysis of user reviews, NLTK [11] is used

to split the collected user reviews into sentences to ensure each

unit of user reviews only contains one piece of information.

Step 1.2: Sentence pre-processing. In this step, multiple

NLP (Natural Language Processing) techniques were applied to

the textual content of app review sentences. Speciﬁcally, NLTK

[11] was adopted to perform stopword removal, punctuation re-

moval, and lemmatization.

Step 1.3: Non-informative review sentences ﬁltering. First,

this paper follows the deﬁnition of informative reviews and

non-informative reviews in [7]: ‘informative’ implies that the

review is constructive/helpful to app developers, and “non-

informative” means that the review contains no information that

is useful for improving apps. To remove non-informative re-

views, this paper reused EMNB [12], i.e., the semi-supervised

machine learning algorithm adopted in ARMiner [7], as the

classiﬁer of user reviews.

Compared with supervised algorithms, EMNB ﬁts the case of

a small amount of manually-labeled data along with plenty of

unlabeled data to achieve better performance. Speciﬁcally, we

leverage the dataset provided by Chen et al. [7] as the training

data to build the classiﬁer for data ﬁltering.

Task 2: Data Processing for Release Notes, which is con-

ducted by the following two steps.

Step 2.1: Sentence splitting. Similar to user reviews, a re-

lease note usually contains of several release sentences in a list

format. As shown in Figure 1 (a), each piece of information

basically starts a new line with a special symbol (e.g., “−”, “∗”,

or “·”). We observed that some release notes do not end with

a period at the end of each line, which may leads to failure in

sentence-level splitting using the natural language processing

toolkits, like NLTK or SpaCy [13], so we divide the release

notes by paragraphs.

Step 2.2: Duplication removal. This step can be used

to clean duplicated release note sentences from the following

three aspects. First, as mentioned earlier, many apps tend to

make ‘perfunctory’ updates in a pattern that does not deliver

the content of the updating, for example, “Fixed stability is-

sues (Sporify, 2021-11-11)”, “Bug ﬁxes and performance im-

provements (Google Drive, 2020-10-27)”, and “Some bug ﬁxes.

(SHEIN, 2017-11-03)”. These release note sentences occur fre-

quently but seldom bring any details. Therefore, the removal

of this type of release note sentences can greatly increase the

quality of release notes to be included in the research dataset.

Second, some release notes start or end with a welcome or ap-

preciation statement at the beginning or the end of each update,

or for example, “Take collaboration to the next level by connect-

ing over video with Google Meet, part of Google Workspace.

(Google Meet, 2021-08-31)”, and some release notes also tend

to have a general statement before the list of new features, such

as “Some recent additions include: (Pandora, 2019-01-31)”

and “This release brings enhancements that help you get more

out of your inbox: (Gmail, 2018-08-13)”. These release notes

are also often duplicated, and the process can signiﬁcantly re-

duce their weight in the update log sentences.. Third, release

notes often contain major updates and subsequent minor up-

dates, and a small number of apps repeat the content of major

updates in minor updates for the purpose of reminding users of

their new major features. The de-duplication of keeping only

the initial occurrence of the release notes can ensure the re-

moval of such duplicate information.

Apple App Store

Apps

2nd Round Selection

App Release NotesApp User Reviews

RN Processing

1. Sentence Splitting

2. Duplication Removal

UR Processing

1. Sentence Splitting

2. Text Preprocessing

3. Non-informative Reviews

Filtering

Set of user review

sentences

Input

Word2Vec POS-Tagger

Retain Nouns, Verbs

and Adjectives

Release Note

Embedding

Review

Embeddings

Release note

sentece

Contextualized

Embeddings

Release Note

Embedding

Review

Embeddings

Sentence Embedding Sentence Embedding

Cosine Similarity Cosine Similarity

High-confidence matching pairs

RankRank

1st Round Selection

Top N Pairs

by Word2Vec

Top N Pairs

by SBERT

Embedding

Identify Matching Pairs (RoseMatcher) (Sec. 5)

SentenceBERT

Select and Process Research Data (Sec. 4)

Analyze Matching Pairs (Sec. 6)

Further Analysis

Manual Labelling

User Review Roles Identification

Time Difference Calculation

Figure 2: Overview of the data processing and RoseMatcher approach.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

RoseMatcher:IdentifyingtheImpactofUserReviewsonAppUpdatesTianyangLiua,b,ChongWanga,c,,KunHuanga,PengLianga,,BeiqiZhanga,MayaDanevac,MartenvanSinderencaSchoolofComputerScience,WuhanUniversity,430072Wuhan,Hubei,ChinabDepartmentofComputerScienceandEngineering,UniversityofCaliforniaSanDiego,LaJolla,Ca...

展开>> 收起<<

RoseMatcher Identifying the Impact of User Reviews on App Updates.pdf

共20页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

RoseMatcher Identifying the Impact of User Reviews on App Updates

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: