Validating process variables of sourcing in an assessment of multiple document comprehension

Nagel

et al. 2020

Front. Educ.

Critical evaluation skills when using online information are considered important in many research and education frameworks; critical thinking and information literacy are cited as key twenty-first century skills for students. Higher education may play a special role in promoting students' skills in critically evaluating (online) sources. Today, higher education students are more likely to use the Internet instead of offline sources such as textbooks when studying for exams. However, far from being a value-neutral, curated learning environment, the Internet poses various challenges, including a large amount of incomplete, contradictory, erroneous, and biased information. With low barriers to online publication, the responsibility to access, select, process, and use suitable relevant and trustworthy information rests with the (self-directed) learner. Despite the central importance of critically evaluating online information, its assessment in higher education is still an emerging field. In this paper, we present a newly developed theoretical-conceptual framework for Critical Online Reasoning (COR), situated in relation to prior approaches (“information problem-solving,” “multiple-source comprehension,” “web credibility,” “informal argumentation,” “critical thinking”), along with an evidence-centered assessment framework and its preliminary validation. In 2016, the Stanford History Education Group developed and validated the assessment of Civic Online Reasoning for the United States. At the college level, this assessment holistically measures students' web searches and evaluation of online information using open Internet searches and real websites. Our initial adaptation and validation indicated a need to further develop the construct and assessment framework for evaluating higher education students in Germany across disciplines over their course of studies. Based on our literature review and prior analyses, we classified COR abilities into three uniquely combined facets: (i) online information acquisition, (ii) critical information evaluation, and (iii) reasoning based on evidence, argumentation, and synthesis. We modeled COR ability from a behavior, content, process, and development perspective, specifying scoring rubrics in an evidence-centered design. Preliminary validation results from expert interviews and content analysis indicated that the assessment covers typical online media and challenges for higher education students in Germany and contains cues to tap modeled COR abilities. We close with a discussion of ongoing research and potentials for future development.

Section: Relevance and Research Backgroundmentioning

confidence: 99%

Assessing University Students' Critical Online Reasoning Ability: A Conceptual and Assessment Framework With Preliminary Evidence

Molerov

Nagel

et al. 2020

Front. Educ.

“…Students’ mental CR strategies may likely be insufficient for what is demanded for understanding heterogeneous information and, what is more, for effective and productive participation in a complex information environment (for a meta-study, see Huber and Kuncel, 2016 , for university students, see McGrew et al, 2018 ; Wineburg et al, 2018 ; Hahnel et al, 2019 ; Münchow et al, 2019 ; Shavelson et al, 2019 ; Zlatkin-Troitschanskaia et al, 2019b ). As a coping strategy, they may choose to reduce complexity by various means, for instance, by using cognitive heuristics, preferring simplified forms of information presentation, or relying on sources without verification, which can be exploited for manipulation ( Walthen and Burkell, 2002 ; Metzger, 2007 ; Horstmann et al, 2009 ; Metzger et al, 2010 ).…”

Section: Research Background and Study Objectivesmentioning

confidence: 99%

“…Insufficient reasoning can be amplified when information on a topic is distorted or counterfactual and when students do not recognize biased or false information and use it to build knowledge. As a result, learners may neglect complex, academically warranted knowledge and rely more on lower-quality information that is consistent with their beliefs and biases and that is easier to comprehend ( Hahnel et al, 2019 ; Schoor et al, 2019 ). The internalization of this biased information may subsequently affect learning by acting to inhibit or distort more advanced information processing and knowledge acquisition ( List and Alexander, 2017 , 2018 ).…”

Section: Research Background and Study Objectivesmentioning

confidence: 99%

The Role of Students’ Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

Beck

Fischer

et al. 2020

Front. Psychol.

Critical reasoning (CR) when confronted with contradictory information from multiple sources is a crucial ability in a knowledge-based society and digital world. Using information without critically reflecting on the content and its quality may lead to the acceptance of information based on unwarranted claims. Previous personal beliefs are assumed to play a decisive role when it comes to critically differentiating between assertions and claims and warranted knowledge and facts. The role of generic epistemic beliefs on critical stance and attitude in reflectively dealing with information is well researched. Relatively few studies however, have been conducted on the influence of domain-specific beliefs, i.e., beliefs in relation to specific content encountered in a piece of information or task, on the reasoning process, and on how these beliefs may affect decision-making processes. This study focuses on students' task-and topicrelated beliefs that may influence their reasoning when dealing with multiple and partly contradictory sources of information. To validly assess CR among university students, we used a newly developed computer-based performance assessment in which the students were confronted with an authentic task which contains theoretically defined psychological stimuli for measuring CR. To investigate the particular role of task-and topic-related beliefs on CR, a purposeful sample of 30 university students took part in a performance assessment and then were interviewed immediately afterward. In the semistructured cognitive interviews, the participants' task-related beliefs were assessed. Based on qualitative analyses of the interview transcripts, three distinct profiles of decision-making among students have been identified. More specifically, the different types of students' beliefs and attitudes derived from the cognitive interview data suggest their influence on information processing, reasoning approaches and decision-making. The results indicated that the students' beliefs had an influence on their selection, critical evaluation and use of information as well as on their reasoning processes and final decisions.

“…In particular, the results on the influence of overload due to media use over the course of higher education studies stress the relevance of carefully filtering out and critically dealing with different kinds of information, which are considered crucial generic skills in the twenty-first century. As many current studies indicate, university students have great deficits when it comes to critically dealing with this multitude of media and online information (Hahnel et al 2019a;Hahnel et al 2019b;McGrew et al 2018;Zlatkin-Troitschanskaia et al 2019b). This is evidently the case for beginning students when entering higher education but also for advanced bachelor's and even master's students (Shavelson et al 2019;Nagel et al 2020).…”

Section: Discussionmentioning

confidence: 99%

Entry Assessment of Student Learning Preconditions in Higher Education: Implications for the Transition from Secondary to Tertiary Education in Germany

Schlax

2020

European Review

The acquisition of domain-specific knowledge and interdisciplinary skills such as critical thinking is increasingly gaining significance as key learning outcomes in higher education that are crucial for all professionals and engaged citizens and that enable lifelong learning. Despite this socio-political consensus, up until the last decade there have only been a few evidence-based insights into the competencies of higher education students. Therefore, the Germany-wide research program Modelling and Measuring Competencies in Higher Education (KoKoHs) was established in 2011 by the Federal Ministry of Education and Research. In the 85 projects, theoretical-conceptual competence models and corresponding assessments were developed for selected large study domains (e.g. economics) to reliably measure the students’ competencies in different phases of higher education (entering, undergraduate, graduate). More than 100 technology-based assessments of both discipline-specific competencies and generic skills were validated across Germany at over 350 universities with over 75,000 students. This article presents findings from the Germany-wide entry diagnostics in the one KoKoHs project (WiWiKom II) with beginning students in business, economic and social sciences that provide evidence-based insights into students’ learning preconditions and their impact on domain-specific knowledge acquisition in bachelor’s degree courses. The results lead to far-reaching practical implications for successful transitions between secondary and tertiary education, including recommendations for the development of mechanisms to support access to tertiary education and to prevent high dropout rates.