A Preliminary Exploration of the Role of Surveys In Student Reflection and Behavior

Levine, Amber Dale; Björklund, Tua; Gilmartin, Shannon Katherine; Sheppard, Sheri

doi:10.18260/1-2--27500

Cited by 2 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: The Ems Study and Entrepreneurial Behavior Predictorsmentioning

confidence: 99%

Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)

2022

View full text Add to dashboard Cite

The bi-encoder design of dense passage retriever (DPR) is a key factor to its success in open-domain question answering (QA), yet it is unclear how DPR's question encoder and passage encoder individually contributes to overall performance, which we refer to as the encoder attribution problem. The problem is important as it helps us identify the factors that affect individual encoders to further improve overall performance. In this paper, we formulate our analysis under a probabilistic framework called encoder marginalization, where we quantify the contribution of a single encoder by marginalizing other variables. First, we find that the passage encoder contributes more than the question encoder to indomain retrieval accuracy. Second, we demonstrate how to find the affecting factors for each encoder, where we train DPR with different amounts of data and use encoder marginalization to analyze the results. We find that positive passage overlap and corpus coverage of training data have big impacts on the passage encoder, while the question encoder is mainly affected by training sample complexity under this setting. Based on this framework, we can devise data-efficient training regimes: for example, we manage to train a passage encoder on SQuAD using 60% less training data without loss of accuracy. * Equal contribution Q-Encoder-NQ* P-encoder-NQ P-encoder-Trivia P-encoder-SQuAD P-encoder-Curated NQ-test Wikipedia Corpus Data Encoders Vector Search Average Retrieval Score Petr Baudiš and Jan Šedivỳ. 2015. Modeling of the question answering task in the YodaQA system. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 222-228.

show abstract

Section: The Ems Study and Entrepreneurial Behavior Predictorsmentioning

confidence: 99%

Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)

2022

View full text Add to dashboard Cite

show abstract

“…In this paper, we work with the Engineering Major Survey (EMS) longitudinal study of students' career goals by . Analysis of the contents of this study was previously conducted mainly by the social sciences with a focus on qualitative approaches to extract the most influential variables on career goals (Grau et al, 2016;Levine et al, 2017). Quantitative correlation between variables was previously explored by Atwood et al (2020) relating Social Cognitive Career Theory (SCCT) (Lent et al, 1994) to different predefined topics for the purpose of survey design, such as students demographics, first-generation status, and family background.…”

Section: The Ems Study and Entrepreneurial Behavior Predictorsmentioning

confidence: 99%

Explaining Neural NLP Models for the Joint Analysis of Open-and-Closed-Ended Survey Answers

Mosca¹,

Harmann²,

Eder³

et al. 2022

Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)

View full text Add to dashboard Cite

Large-scale surveys are a widely used instrument to collect data from a target audience. Beyond the single individual, an appropriate analysis of the answers can reveal trends and patterns and thus generate new insights and knowledge for researchers. Current analysis practices employ shallow machine learning methods or rely on (biased) human judgment. This work investigates the usage of state-of-the-art NLP models such as BERT to automatically extract information from both open-and closed-ended questions. We also leverage explainability methods at different levels of granularity to further derive knowledge from the analysis model. Experiments on EMS-a survey-based study researching influencing factors affecting a student's career goals-show that the proposed approach can identify such factors both at the input-and higher concept-level.

show abstract

A Preliminary Exploration of the Role of Surveys In Student Reflection and Behavior

Cited by 2 publications

References 13 publications

Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)

Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)

Explaining Neural NLP Models for the Joint Analysis of Open-and-Closed-Ended Survey Answers

Contact Info

Product

Resources

About