How many submissions does it take to discover friendly suggested reviewers Pedro Pessoa12 Steve Press e123

2025-04-29 0 0 9.65MB 30 页 10玖币

侵权投诉

How many submissions does it take to discover friendly suggested

reviewers?

Pedro Pessoa1,2, Steve Press´e1,2,3

1Center for Biological Physics, Arizona State University, Tempe, AZ, USA

2Department of Physics, Arizona State University, Tempe, AZ, USA

3School of Molecular Sciences, Arizona State University, Tempe, AZ, USA

Abstract

It is ever more common in scientiﬁc publishing to ask authors to suggest some reviewers for their own

manuscripts. The question then arises: How many submissions does it take to discover friendly suggested

reviewers? To answer this question, we present an agent-based simulation of (single-blinded) peer review, followed

by a Bayesian classiﬁcation of suggested reviewers. To set a lower bound on the number of submissions possible,

we create a optimistically simple model that should allow us to more readily deduce the degree of friendliness of

the reviewer. Despite this model’s optimistic conditions, we ﬁnd that one would need hundreds of submissions

to classify even a small reviewer subset. Thus, it is virtually unfeasible under realistic conditions. This ensures

that the peer review system is suﬃciently robust to allow authors to suggest their own reviewers.

Keywords: Peer review, Simulation, Agent-based model, Bayesian statistics

arXiv:2210.00905v3 [stat.AP] 16 Jan 2023

1 Introduction

Peer review is the cornerstone of quality control of academic publishing. However, the daunting task of

selecting appropriate reviewers [1, 2] relies in identifying at least two scholars, free of conﬂict of interest, who

have: 1) the necessary expertise to judge the quality and perceived impact; and 2) the willingness to perform

the work pro bono. On account of this, it is ever more common that journals request, and often require, authors

to suggest candidate reviewers. That is, provide names and contact information of scholars the authors deem

qualiﬁed to review.

It is natural to imagine, at ﬁrst glance, that this incentivizes authors to submit “friendly” names, implying

suggesting reviewers that they have reason to believe would be favorably inclined toward them. The fear of such

peer review manipulation is potentiated by reports that author-suggested reviewers are more likely to recommend

acceptance [3–10]. However, some of these same studies mention that the quality of reports of author-suggested

reviewers does not diﬀer from the ones of editor-suggested reviewers [3–5, 8, 9]. It is also reported that the

diﬀerence in suggesting acceptance by author-suggested and editor-suggested reviewers is not signiﬁcant when

comparing reports of the same submission [7] nor it is observed to have an eﬀect in the article’s acceptance [3,7]

and this discrepancy can even vanish entirely in some ﬁelds [11].

The question then naturally arises: can a scientist infer from their personal history of submissions which

reviewers are likely to bias the decision in their favor? In what follows, we present an optimistic agent-based

model that surely underestimates the number of submissions required to ascertain the friendliness of the reviewer

with high conﬁdence. What we ﬁnd is that, due to multiple sources of uncertainty (e.g., lack of knowledge as

to which reviewer the editor selects), such an eﬀort would require a number of submissions vastly exceeding

the research output of all but the most productive scientists. That is, hundreds and sometimes thousands of

submissions.

As neither a manuscript’s submission history, reviewers selected by the editor, nor suggested reviewers by

the authors are publicly available, we adapt agent-based simulation models [12–14], already used in generating

simulated peer review data [14], and develop an inference strategy on this model’s output to ask whether we

can uncover favorably biased reviewers. This ﬁts into a larger eﬀort to quantitatively study the dynamics of

scientiﬁc interactions [15–18].

As we initially simulate the data, we intentionally make assumptions using agent-based models that would

result in easy classiﬁcation in order to obtain a lower bound on the number of submissions required to conﬁdently

classify reviewers. These assumptions read as follows:

i) For each submission, the author will always suggest a small number of reviewers (three, in our simulation)

from a ﬁxed and small (ten elements, in our simulation) pool of names.

ii) The editor will always select one of the reviewers suggested by the authors.

iii) The “friendliness” of any given reviewer remains the same for all subsequent submissions.

iv) Submissions from the same author all have the same overall quality.

Shortly we will lift the assumptions of this “cynical model” and introduce a “quality factor model” or simply,

quality model. In particular, we will lift assumption iv). As we will see, lifting assumptions will only raise, often

precipitously, the already unfeasibly high lower bound on the number of submissions required to conﬁdently

classify reviewers and leverage this information to bias reports in their favor.

2 Methods

In order to set a lower bound on the number of submissions required to conﬁdently classify reviewers, the

present study focuses on a simpliﬁed peer review process characterized by three types of agents: the author(s),

the editor, and the reviewers. Each submission is reviewed according to the following steps:

1) During submission, the author will send to the editor a list of suggested reviewers, S. The suggested

reviewers are chosen from a larger set of possible reviewers R— such that Sis a subset of R.

2) The editor will select one reviewer, namely r1, from Srandomly with uniform probability.

3) The editor will also select a second reviewer, r2, from a pool of reviewers considerably larger than Rand

representative of the scientiﬁc community.

4) The reviewers will write single blind reports, either overall positive or negative, and the author will have

access to the number of positive reviews a.

A diagram of this idealized process is presented in Fig. 1.

In the spirit of identifying a lower bound on submissions, we make the dramatic assumption that r1either

belongs to friend or rival class while r2is otherwise neutral. Later we will devise a Bayesian inference strategy

to achieve suggested reviewer (r1) classiﬁcation.

Author

Editor r1Editor

apositive reviews

Figure 1: Diagram presenting the simpliﬁed peer review process. See the steps described in section 2 for deﬁnitions.

jxj= [ xj

1xj

1x1= [ rival,rival ]

2x2= [ rival,friend ]

3x3= [ friend,rival ]

4x4= [ friend,friend ]

Table 1: Example of the construction and enumeration of the possible conﬁgurations, xj, for a set of two possible

suggested reviewers (|R| = 2). As described in the ﬁrst paragraph of section 2.1.

The procedure described in the bullet points above refers to a single submission. However, as our end goal is

to determine how many submissions are necessary to classify reviewers, we must consider multiple submissions.

For this reason, we represent a history of M, identical and independent, submissions using the index µ∈

{1,2,...,M}, such that Sµand aµare, respectively, the set of suggested reviewers and positive reviews accrued

for the µ-th submission.

Now that we have qualitatively described our agent-based model, we provide next a detailed mathematical

formulation of the simulation and inference.

2.1 Mathematical formulation

Here each element of Ris a reviewer. We denote xithe state of each reviewer as belonging to one of two

classes: either xi=friend or xi=rival. The method can immediately be generalized to accommodate the

addition of a third (neutral) class. Put diﬀerently, each suggested reviewer is treated as a Categorical random

variable realized to either friend or rival. Collecting all states as a sequence, we write x=x1, x2,...,x|R|with

|R| understood as the cardinality of R. For two classes, we have 2|R| allowed conﬁgurations of x. It is convenient

to index conﬁgurations with a jsuperscript where j∈ {1,2,...,2|R| }for which xj=hxj

1, xj

2,...,xj

|R|i.

For sake of clarity alone, we provide a concrete example enumerating all conﬁgurations for two possible

suggested reviewers in Table 1.

We will now use Bayesian inference to determine the probability we assign to each conﬁguration. That is, to

compute posterior probabilities, P(xj|{aµ},{Sµ}), over each xjgiven the set of positive reports {aµ}received

after suggesting a subset {Sµ}of reviewers. Such inference is only feasible because friend and rival classes exhibit

diﬀerent behaviors when writing reports. In the present article, we will study two models for reviewer behavior.

The ﬁrst is the, simpler, cynical model where the friend writes a positive review with unit probability and,

by contradistinction, the rival writes a positive review with null probability. The reviewer not selected from the

author’s list, r2, will write a positive review with probability 1

/2. In this iteration of the model it should be

easiest (i.e., quickest in terms of number of submissions) to sharpen our posterior and classify reviewers.

The second model is the quality model that introduces a new layer of stochasticity. Here, a submission is

associated a quality factor q∈(0,1) reﬂecting the quality of each submission . In this model an unbiased reviewer

(r2) would write a positive review with probability q. By contrast, rivals and friends will “double guess” their

accept reject

r2q1−q

r1is a rival q21−q2

r1is a friend 1−(1 −q)2=q(2 −q) (1 −q)2

Table 2: Probabilities for reviewers of each class to write a positive report (accept) or a negative report (reject)

according to the quality model when reviewing a paper of quality factor q.

own judgment of the article implying that they will evaluate the submitted article twice independently. A rival

will only suggest acceptance if they deem the submission worthy of publication in both assessments, meaning

a rival will write a positive review with probability q2. Analogously, a friend will reject if they “reject twice”,

hence they write a negative review with probability (1 −q)2or, equivalently, a positive review with probability

1−(1 −q)2=q(2 −q). A summary of these probabilities is presented in Table 2. As done with aµand Sµ, we

index the quality factor of the µ-th submission as qµ.

Not all authors, naturally, have distributions over qcentered at the same value. It is therefore of interest to

compute the eﬀect on the lower bound of submission needed (i.e., how quickly our posterior sharpens around the

ground truth) for diﬀerent distributions over qcentered at the extremes (average high or average low quality) in

addition to middle-of-the-road distributions centered at q=1

/2. As we will see, middle-of-the-road distributions

allow for more rapid posterior sharpening. Notwithstanding this paltry incentive to write middle-of-the-road

papers, we will see that the lower bound on the number of submissions remains unfeasibly high. Even for this

idealized scenario.

2.2 Simulation

Following the steps described at the beginning of Section 2, the ﬁrst step of the simulation involves editorial

selection from the list of suggested reviewers with |R|

|Sµ|possible sets of suggested reviewers possible, or 120

given our simulation parameters (|Sµ|= 3 and |R| = 10 for all µ). Each Sµfor any µis independently sampled

with uniform probability.

We must initialize the ground truth conﬁguration (the identity of x). Initially, we set an equal number

of friends and rivals though we generalize to two other cases (seven and nine friends) in the Supplemental

Information A.

The subsequent steps (steps 2-3) are straightforward. Step 4 for the cynical model is equally straightforward

(and deterministic in r1): a positive review is returned if r1µis a friend, a negative review s returned otherwise,

while r2writes a positive review with probability 1

/2. Further mathematical simulation details are found in

Supplemental Information B.1.

For the quality model, to each submission (µ) is associated a quality factor qµ∈(0,1). As is usual for a

variable bounded by the interval (0,1), we we take qµas a Beta random variable such that

P(qµ) = qα−1

µ(1 −qµ)β−1

B(α, β)(1)

where B(α, β) = Γ(α)Γ(β)

Γ(α+β)where Γ being the Euler’s gamma function. Again, in an eﬀort to compute a lower

bound alone on the number submissions required, we assume that all qµare sampled from the same, stationary,

distribution with constant α= 12 and β= 12 for now (middle-of-the-road quality distribution) for which the

mean hqi=1

/2and the variance is σq=.01.

In reality, it is conceivable that one’s quality factor distribution shifts to the right with experience. It is

also conceivable that a proliﬁc researcher would have its quality factor shift to the left as they start venturing

into new ﬁelds. This eﬀect only makes it harder to assess which reviewer is friendly and further raises the lower

bound required on the number of submissions. In any case, in the Supplemental Information C, we consider

diﬀerent quality distributions (both high and low). Foreshadowing the conclusions, it may be intuitive to see

that very high or very low quality factors result in less information gathered per reviewer report. That is, we

learn best the class to which reviewers belong by sampling quality factors around 1

/2. Not by constant rejection

or acceptance.

Thus, with each sampled qµ, step 4) of the quality model is implemented by observing that reviewers write

positive reviews according to the probabilities in Table 2. Further mathematical details of the quality model are

relegated to Supplemental Information B.2.

Importantly, for the purposes of classifying which reviewers are friendly, it is not necessary to know whether

the article is accepted by the editor, only the count of positive or negative reviews per submission.

2.3 Inference strategy

Inference consists of constructing the posterior P(xj|{aµ},{Sµ}) and drawing samples from it. To construct

this posterior, we update the likelihood, P({aµ}|xj,{Sµ}), over all independent submission

P({aµ}|xj,{Sµ}) = Y

P(aµ|xj,Sµ) (2)

as follows

P(xj|{aµ},{Sµ}) = P(xj|{Sµ})

P({aµ}|{Sµ})P({aµ}|xj,{Sµ}).(3)

Since the number of conﬁgurations is ﬁnite, we may start by taking the prior as uniform over these countable

options (P(xj|{Sµ}) = 2−|R|). Keeping all dependency on xjexplicit, we may write

P(xj|{aµ},{Sµ})∝P({aµ}|xj,{Sµ}) = Y

P(aµ|xj,Sµ).(4)

We end with a note on the likelihood which we compute explicitly by treating r1µas a latent variable over

which we sum. That is,

P(aµ|xj,Sµ) = X

r1µ

P(aµ|r1µ)P(r1µ|xj,Sµ).(5)

In terms of the factors within the summation, P(r1µ|xj,Sµ) follows from step 2). That is, if the editor selects

r1µwith uniform probability from Sµ, the probability of selecting a r1µfrom the class of friends is the ratio of

friends, f, in Sµaccording to the conﬁguration xj. This can be written more rigorously as

P(r1µ=friend |xj,Sµ) = f(xj,Sµ).

|S| X

i∈S

F(xj

i),(6)

where

F(xj

i) = (0 if xj

i=rival

1 if xj

i=friend .(7)

It follows that P(r1µ=rival |xj,Sµ) = 1 −f(xj,Sµ).

We now turn to the term P(aµ|r1µ) within (5) computed diﬀerently within both the cynical and quality

models.

2.3.1 Inference in the cynical model

Calculating P(aµ|r1µ) for the cynical model is straightforward. That is, given that a friendly r1always writes

a positive review and a rival r1always writes a negative one, and r2writes a positive review with probability

/2, values for P(aµ|r1µ) immediately follow as tabulated in Table 3. Equations (3 – 7) and Table 3 summarize

what is needed to perform Bayesian classiﬁcation within the cynical model formulation.

P(aµ|r1µ)aµ= 0 aµ= 1 aµ= 2

r1µ=friend 01

/21

r1µ=rival 1

/21

/20

Table 3: Probabilities for the number of positive reports, aµ, in the cynical model, conditioned on the class of the

suggested reviewer, r1µ.

2.3.2 Inference in the quality model

The major diﬀerence between inference in the quality and cynical models relies on the fact that the author will

not have access to individual qµ’s. However, since we aim for a lower bound, we will proceed with the calculation

under the assumption that while individual qµ’s are unknown the author knows the distribution from which qµ

is sampled. If the author were uncertain of the distribution, this would add yet another layer of stochasticity

and further raise the lower bound. From Table 2, it is straightforward to calculate the probability of each aµ

given r1µand qµin the quality model. The result is found in Table 4.

Without access to qµin (5), we further need to marginalize P(aµ|qµ, r1µ) over qµas follows

P(aµ|r1µ) = ZdqµP(aµ, qµ|r1µ) = ZdqµP(aµ|qµ, r1µ)P(qµ) = P(aµ|qµ, r1µ)qµ

.(8)

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Howmanysubmissionsdoesittaketodiscoverfriendlysuggestedreviewers?PedroPessoa1;2,StevePresse1;2;31CenterforBiologicalPhysics,ArizonaStateUniversity,Tempe,AZ,USA2DepartmentofPhysics,ArizonaStateUniversity,Tempe,AZ,USA3SchoolofMolecularSciences,ArizonaStateUniversity,Tempe,AZ,USAAbstractItisevermoreco...

展开>> 收起<<

How many submissions does it take to discover friendly suggested reviewers Pedro Pessoa12 Steve Press e123.pdf

共30页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

How many submissions does it take to discover friendly suggested reviewers Pedro Pessoa12 Steve Press e123

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: