Assessing the format of the presentation of text in developing a Reading Strategy Assessment Tool (R-SAT)

Gilliam, Sara; Magliano, Joseph P.; Millis, Keith K.; Levinstein, Irwin B.; Boonthum, Chutima

doi:10.3758/bf03193148

Cited by 23 publications

(31 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each answer to the target sentences was automatically scored by identifying the number of content words in the answer that was also in the text or in an ideal answer (Gilliam et al, 2007;Magliano et al, in press2011). Content words included nouns, adverbs, adjectives and verbs (semantically depleted verbs, such as is, are, were omitted).…”

Section: Participantsmentioning

confidence: 99%

“…To assess the change in self-explanation usage and the alignment between selfexplanation processes and comprehension, we used an automated reading strategy assessment tool called the Reading Strategy Assessment Tool (RSAT; Gilliam, Magliano, Millis, Levinstein, & Boonthum, 2007;Magliano, et al, 2011). RSAT was developed to provide a computer-based approach for collecting and analyzing verbal protocols produced while reading texts.…”

Section: Changing How Students Process and Comprehend Texts With Compmentioning

confidence: 99%

See 1 more Smart Citation

Changing How Students Process and Comprehend Texts with Computer-Based Self-Explanation Training

Kurby

Magliano

Dandotkar

et al. 2012

Journal of Educational Computing Research

Self Cite

View full text Add to dashboard Cite

This study assessed whether and how self-explanation reading training, provided by iSTART (Interactive Strategy Training for Active Reading and Thinking), improves the effectiveness of comprehension processes. iSTART teaches students how to self-explain and which strategies will most effectively aid comprehension from moment-to-moment. We used RSAT (Reading Strategy Assessment Tool) to assess how iSTART changes the relation between important selfexplanation reading strategies-bridging and elaboration-and online comprehension, and how often they are produced. College and high school students received iSTART and were administered RSAT prior to and post-training. Results from three experiments showed that iSTART primarily benefits bridging inferences when self explaining. The frequency of bridging inferences was higher post training than prior to training, but only in the experiments involving college students. Additionally, prior to exposure to iSTART, RSAT bridging scores did not predict comprehension performance, whereas they did after iSTART, suggesting that iSTART may improve comprehension processes by teaching students how to appropriately use selfexplanation to address comprehension difficulties. Finally, the results from this study suggest that RSAT may provide a valuable computer-based assessment of the effectiveness of selfexplanations that could be used in conjunction with iSTART and in future research on selfexplanation.

show abstract

Section: Participantsmentioning

confidence: 99%

Section: Changing How Students Process and Comprehend Texts With Compmentioning

confidence: 99%

Changing How Students Process and Comprehend Texts with Computer-Based Self-Explanation Training

Kurby

Magliano

Dandotkar

et al. 2012

Journal of Educational Computing Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although not a typical use, LSA "in the small" has been utilized for evaluating students' answers against predefined system answers in several ITSs including Autotutor [28], iSTART [64], and R-SAT [22]. We also employed LSA in our system for this purpose, making use of the Gensim library [76] implementation of LSA with stopwords removed and a similarity threshold ranging between 0.5-0.85.…”

Section: Measuring Semantic Similarity Between Argumentsmentioning

confidence: 99%

Supporting Students in the Analysis of Case Studies for Professional Ethics Education

Sharipova

McCalla

2012

Intelligent Tutoring Systems

View full text Add to dashboard Cite

Intelligent tutoring systems and computer-supported collaborative environments have been designed to enhance human learning in various domains. While a number of solid techniques have been developed in the Artificial Intelligence in Education (AIED) field to foster human learning in fundamental science domains, there is still a lack of evidence about how to support learning in so-called ill-defined domains that are characterized by the absence of formal domain theories, uncertainty about best solution strategies and teaching practices, and learners' answers represented through text and argumentation.This dissertation investigates how to support students' learning in the ill-defined domain of professional ethics through a computer-based learning system. More specifically, it examines how to support students in the analysis of case studies, which is a common pedagogical practice in the ethics domain.This dissertation describes our design considerations and a resulting system called Umka. In Umka learners analyze case studies individually and collaboratively that pose some ethical or professional dilemmas. Umka provides various types of support to learners in the analysis task. In the individual analysis it provides various kinds of feedback to arguments of learners based on predefined system knowledge. In the collaborative analysis Umka fosters learners' interactions and self-reflection through system suggestions and a specifically designed visualization. The system suggestions offer learners the chance to consider certain helpful arguments of their peers, or to interact with certain helpful peers. The visualization highlights similarities and differences between the learners' positions, and illustrates the learners' level of acceptance of each other's positions.This dissertation reports on a series of experiments in which we evaluated the effectiveness of Umka's support features, and suggests several research contributions.Through this work, it is shown that despite the ill-definedness of the ethics domain, and the consequent complications of text processing and domain modelling, it is possible to build effective tutoring systems for supporting students' learning in this domain. Moreover, the techniques developed through this research for the ethics domain can be readily expanded to other ill-defined domains, where argument, qualitative analysis, metacognition and interaction over case studies are key pedagogical practices.ii

show abstract

“…RSAT is a computer-administered test that is designed to assess a student's level of comprehension and the processes that support it while the student is reading (Gilliam et al, 2007;Magliano et al, 2011). Students read texts one sentence at a time and are prompted to answer indirect questions ("what are you thinking now") that require responses that are akin to thinking aloud (Trabassso & Magliano, 1996).…”

Section: Two Approaches For Developing Assessment Targetsmentioning

confidence: 99%

“…We define student-constructed responses as those that require a student to produce an answer in natural language that may range from a couple of sentences to several paragraphs. These advances have been in the context of computer-based assessments of explanations and think-aloud protocols during reading comprehension (Gilliam, Magliano, Millis, Levinstein, & Boonthum, 2007;Magliano, Millis, the RSAT Development Team, Levinstein, & Boonthum, 2011), the grading of essays and text summaries (Attali & Burstein, 2006;Burstein, Marcu, & Knight, 2003;Franzke, Kintsch, Caccamise, Johnson, & Dooley, 2005;Landauer, Laham, & Foltz, 2003), the grading of short-answer questions (Leacock & Chodorow, 2003), and intelligent tutoring systems and trainers that require students to produce constructed responses during interactive conversations (Graesser, Jeon, & Dufty, 2008;Litman et al, 2006;McNamara, Levinstein, & Boonthum, 2004;VanLehn et al, 2007). These can take the form of directed responses to specific questions or less directed thinkaloud and self-explanation responses.…”

mentioning

confidence: 99%

Computer-based assessment of student-constructed responses

Magliano

Graesser

2012

Behav Res

Self Cite

View full text Add to dashboard Cite

Student-constructed responses, such as essays, short-answer questions, and think-aloud protocols, provide a valuable opportunity to gauge student learning outcomes and comprehension strategies. However, given the challenges of grading student-constructed responses, instructors may be hesitant to use them. There have been major advances in the application of natural language processing of student-constructed responses. This literature review focuses on two dimensions that need to be considered when developing new systems. The first is type of response provided by the student-namely, meaning-making responses (e.g., think-aloud protocols, tutorial dialogue) and products of comprehension (e.g., essays, open-ended questions). The second corresponds to considerations of the type of natural language processing systems used and how they are applied to analyze the student responses. We argue that the appropriateness of the assessment protocols is, in part, constrained by the type of response and researchers should use hybrid systems that rely on multiple, convergent natural language algorithms.

show abstract

Assessing the format of the presentation of text in developing a Reading Strategy Assessment Tool (R-SAT)

Cited by 23 publications

References 15 publications

Changing How Students Process and Comprehend Texts with Computer-Based Self-Explanation Training

Changing How Students Process and Comprehend Texts with Computer-Based Self-Explanation Training

Supporting Students in the Analysis of Case Studies for Professional Ethics Education

Computer-based assessment of student-constructed responses

Contact Info

Product

Resources

About