Identifying Difﬁcult exercises in an eTextbook Using Item Response Theory and Logged Data Analysis

2025-05-08 0 0 225.24KB 6 页 10玖币

侵权投诉

Identifying Difﬁcult exercises in an eTextbook

Using Item Response Theory and Logged Data

Analysis

1st Ahmed Abd Elrahman

Information System Department

Faculty of Computers and Information

Assiut University

Egypt

ahmedabdo@aun.edu.eg

2nd Ahmed Ibrahim Taloba

Information System Department

Faculty of Computers and Information

Assiut University

Egypt

Taloba@aun.edu.eg

3rd Mohammed F Farghally

Information System Department

Faculty of Computers and Information

Assiut University

Egypt

mfseddik@aun.edu.eg

4th Taysir Hassan A Soliman

Information System Department

Faculty of Computers and Information

Assiut University

Egypt

taysirhs@aun.edu.eg

Abstract—The growing dependence on eTextbooks and Massive

Open Online Courses (MOOCs) has led to an increase in the

amount of students’ learning data. By carefully analyzing this

data, educators can identify difﬁcult exercises, and evaluate the

quality of the exercises when teaching a particular topic. In this

study, an analysis of log data from the semester usage of the

OpenDSA eTextbook was offered to identify the most difﬁcult

data structure course exercises and to evaluate the quality of

the course exercises. Our study is based on analyzing students’

responses to the course exercises. We applied item response

theory (IRT) analysis and a latent trait mode (LTM) to identify

the most difﬁcult exercises .To evaluate the quality of the course

exercises we applied IRT theory. Our ﬁndings showed that the

exercises that related to algorithm analysis topics represented

the most difﬁcult exercises, and there existing six exercises were

classiﬁed as poor exercises which could be improved or need

some attention.

Index Terms—interactive learning ,item response theory, eText-

books, item characteristics curves, item Information Curve, data

structures and algorithms.

I. INTRODUCTION

The rising usage of interactive online course materials at

all levels of education, including online eTextbooks, Massive

Open Online Courses (MOOCs), and practice platforms like

Khan Academy and Code Academy has spread especially

with the spread of COVID-19 worldwide [1, 2, 3 and 4].In

order to reduce its spread in educational institutions, most

educational institutions have resorted to teaching their courses

via online platforms. During teaching of any online course, the

interaction between students and educators takes place very

little. It could be challenging for educators to know parts

that students suffer from, as well as to assess the quality

of exercises due to the lack of interaction with students. So

knowing which topics students struggle with and attempting

to improve or develop new methods to present these topics

may be an essential step in enhancing the educational process

and boosting the quality of the educational process. When

students struggle with some issues in a course and no one

strives to treat and simplify these topics. It is possible that they

will drop out of the course and they will not ﬁnish studying

the course resulting in failure in the educational process [2].

Knowing what topics students ﬁnd difﬁcult helps instructors

to better allocate course resources. Based on the interactions

of students with the OpenDSA eTextbook system [3, 5], we

present techniques for automatically determining the most

difﬁcult exercises for students. The exercises students struggle

with the most can be detected by experienced instructors, but

this may frequently takes a long time and effort. The topic

of our study is a data structure and algorithms course (CS2).

Our study has two aims, the ﬁrst one is identifying the most

difﬁcult CS2 exercises, and the second is the evaluation of the

quality of exercises in the CS2 course. To identify the most

difﬁcult exercises, we applied two different approaches, the

ﬁrst one is IRT theory and an LTM technique for analyzing

student responses to exercises. LTM assumes that speciﬁc traits

or characteristics can predict test performance [7]. IRT offers

a model-based association between the test characteristics

and the item responses [8]. The second approach involved

analyzing how students interacted with exercises to see which

ones were more challenging. We looked at how often students

used hints. We found that the topics related to algorithm

analysis have the most difﬁcult exercises. To evaluate the

course exercises, we also applied IRT. To classify each exercise

as poor or good, we computed the item difﬁculty and item

arXiv:2210.05294v2 [cs.DS] 25 Nov 2022

discrimination. Based on ﬁnding obtained, we found that six

o exercises were classiﬁed as poor exercises that could be

improved. In a sizable public institution, a CS2 course was

taught using OpenDSA as the main eTextbook [1]. A module

in OpenDSA represents one topic or portion of a typical lec-

ture, such as one sorting algorithm, and is regarded as the most

elementary functional unit for OpenDSA materials [2]. There

is a range of various exercises in each module. One of these

exercises requires the student to manipulate a data structure

to demonstrate how an algorithm affects it. These are known

as ”Proﬁciency Exercises” (PE). PE exercises were developed

and utilized for the ﬁrst time in the TRAKLA2 system [9]. The

other type of exercise is the Simple questions, which include

various types of system questions such as true/false, multiple-

choice, and short-answer questions. OpenDSA made utilized

the exercise framework from Khan Academy (KA) [10] to

save and present Simple questions.

II. RELATED WORK

In [11], the responses of 372 students who registered in

one ﬁrst-year undergraduate course were utilized to evaluate

the quality of 100 MCQs written by an instructor that was

used in an undergraduate midterm and ﬁnal exam. In order to

compute item difﬁculty, discrimination, and chance properties

they applied Classical test theory and IRT analysis models.

The two-Parameter logistic (2PL) model consistently had the

best ﬁt to the data, they discovered. According to the analyses,

higher education institutions need to guarantee that MCQs are

evaluated before student grading decisions are made. In an

introductory programming course, IRT was applied to assess

students’ coding ability [12]. They developed a 1PL Rasch

model using the coding scores of the students. Their ﬁndings

revealed that students with prior knowledge performed sta-

tistically much better than students with no prior knowledge.

In order to analyze the questions for the midterm exam for

an introductory computer science course, the authors of [13]

utilized IRT. The purpose of this study was to study questions’

item characteristic curves in order to enhance the assessment

for future semesters. The authors applied IRT for problem

selection and recommendation in ITS. To automatically select

problems, the authors created a model using a combination of

collaborative ﬁltering and IRT [14].

III. EXPERIMENTAL ANALYSIS

Students make many interactions during their dealing with

the eTextbook, every student interaction represents a log, and

all student logs are stored in the OpenDSA system. OpenDSA

contains different types of interactions. Interactions are divided

into two types the ﬁrst one is only interactions with the eText-

book itself, such as loading a page, reloading a page, clicking

on a link to go somewhere else, or viewing slideshows. The

second type is the interactions with all types of eTextbook

exercises such that attempts to answer any exercise, submit

an answer, or request a hint. This study focused more on

the second type. In [2] a more description of interactions

and exercise types.The amount of questions in each exercise

varies.In this work, during the fall of 2020, we analyzed data of

students who were enrolled in the CS2 course. There are about

303,800 logs that represent the interactions of students with

the eTextbook. These logs contain the name and description of

the action, the time of the interactions, and which module the

student executed the interactions on it. As for the interactions

of students with the exercises, we analysed about 200,000

logs .every log consists of time in which a student interacted

with the question,total seconds in which a student ﬁnished

interacting with a question,total count of hints that the student

requested when interacting with a question,Total counts of

attempts for student attempts to a question, and The type

of request to a question (attempt or hint).In [15], different

measures were applied in order to determine the difﬁcult

topics in a CS3 course. These measures are correct attempt

ratio (r), difﬁculty level (dl), students’ hint usage (hr), and

incorrect answer ratio (it). We computed these measures for

every exercise. In the next subsections, we will talk about

them.

A. Analysis of the ratios of right answers

We aim to give a value to every exercise in terms of “relative

difﬁculty”. Our aim is to ﬁnd which exercises average-ability

students ﬁnd comparatively difﬁcult. From this, we intend to

learn which themes are the most difﬁcult for students. As a

result, maybe lead us to refocus the instructional efforts. In

the OpenDSA, students can answer an exercise as many times

as they want until they get it correct. This will result in most

students receiving almost full marks for their exercises[15].

Among the vulnerabilities, as is typical with online courseware

that most students exploit is that some exercises can be

”gamed” [16], In OpenDSA means that in order for students

to get a question instance which easy to solve they reload the

current page repeatedly. Due to the previous reasons, we have

not counted the number of students who completed an exercise

correctly. Instead, we employed other deﬁnitions for difﬁculty.

To measure the exercise difﬁculty, we looked at the ratio of

correct to incorrect answers in OpenDSA exercises, such that

the correct attempt ratio for difﬁcult exercises should be lower.

We utilized the fraction r to evaluate student performance.

r=#of correct attempts

#of total attempts (1)

We calculate the difﬁculty level (dl) for each exercise, such

that

dl =1−Pn

ir(i)

N(2)

The number of students is referred to as N, and the ratio of

correct attempts is referred to as r. In [15] ,the same measure

was used to identify the difﬁcult topics in CS3 course.In [17],

similar measures was utilized to rate the difﬁculty of exer-

cises, the authors utilized “the number of attempts it takes a

student to ﬁgure out the right answer once making their initial

mistake” as a metric of how difﬁcult a logic exercises are. To

determine the workout difﬁculty for an ITS, history of attempts

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

IdentifyingDifcultexercisesinaneTextbookUsingItemResponseTheoryandLoggedDataAnalysis1stAhmedAbdElrahmanInformationSystemDepartmentFacultyofComputersandInformationAssiutUniversityEgyptahmedabdo@aun.edu.eg2ndAhmedIbrahimTalobaInformationSystemDepartmentFacultyofComputersandInformationAssiutUniversity...

展开>> 收起<<

Identifying Difﬁcult exercises in an eTextbook Using Item Response Theory and Logged Data Analysis.pdf

共6页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Identifying Difﬁcult exercises in an eTextbook Using Item Response Theory and Logged Data Analysis

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: