Evaluating Sentence‐BERT‐powered learning analytics for automated assessment of students' causal diagrams.
Background: When learning causal relations, completing causal diagrams enhances students' comprehension judgements to some extent. To potentially boost this effect, advances in natural language processing (NLP) enable real‐time formative feedback based on the automated assessment of students' diagra...
| Publicado en: | Journal of Computer Assisted Learning Vol. 40; no. 6; pp. 2667 - 2681 |
|---|---|
| Autores principales: | , , , |
| Formato: | research tables/charts Journal Article |
| Publicado: |
Wiley-Blackwell
Dec2024
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=180899661&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 180899661 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 02664909 6M1 jtl: Journal of Computer Assisted Learning issn: 02664909 maglogo: Y pubinfo: dt: Dec2024 vid: 40 iid: 6 pid: 480 pub: Wiley-Blackwell place: Malden, Massachusetts artinfo: ui: 180899661 176779630 180899661 180899661 10.1111/jcal.12992 180899661 ppf: 2667 ppct: 14 formats: tig: atl: Evaluating Sentence‐BERT‐powered learning analytics for automated assessment of students' causal diagrams. aug: au: Pijeira‐Díaz, Héctor J. Subramanya, Shashank van de Pol, Janneke de Bruin, Anique affil: Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences (FHML), Maastricht University, Maastricht, The Netherlands sug: subj: Data Analytics Methods Support Vector Machine Neural Networks (Computer) Students, High School Automation Reliability Funding Source Human Semantic Analysis Educational Technology Machine Learning Feedback Workload Natural Language Processing ab: Background: When learning causal relations, completing causal diagrams enhances students' comprehension judgements to some extent. To potentially boost this effect, advances in natural language processing (NLP) enable real‐time formative feedback based on the automated assessment of students' diagrams, which can involve the correctness of both the responses and their position in the causal chain. However, the responsible adoption and effectiveness of automated diagram assessment depend on its reliability. Objectives: In this study, we compare two Dutch pre‐trained models (i.e., based on RobBERT and BERTje) in combination with two machine‐learning classifiers—Support Vector Machine (SVM) and Neural Networks (NN), in terms of different indicators of automated diagram assessment reliability. We also contrast two techniques (i.e., semantic similarity and machine learning) for estimating the correct position of a student diagram response in the causal chain. Methods: For training and evaluation of the models, we capitalize on a human‐labelled dataset containing 2900+ causal diagrams completed by 700+ secondary school students, accumulated from previous diagramming experiments. Results and Conclusions: In predicting correct responses, 86% accuracy and Cohen's κ of 0.69 were reached, with combinations using SVM being roughly three‐times faster (important for real‐time applications) than their NN counterparts. In terms of predicting the response position in the causal diagrams, 92% accuracy and 0.89 Cohen's κ were reached. Implications: Taken together, these evaluation figures equip educational designers for decision‐making on when these NLP‐powered learning analytics are warranted for automated formative feedback in causal relation learning; thereby potentially enabling real‐time feedback for learners and reducing teachers' workload. Lay Description: What is currently known about this topic?: Students' monitoring accuracy of causal relation learning is on average low.Completing causal diagrams improves monitoring accuracy to some extent.Advances in natural language processing (NLP) enable automated diagram assessment.NLP‐powered learning analytics can be used for automated formative feedback. What does this paper add?: Evaluation of the reliability of the automated diagram assessment.Performance comparison of different language technologies and techniques.The accuracy of the automated diagram assessment ranged from 84% to 86%.Human‐computer Cohen's κ surpassed that of human–human (0.89 vs. 0.84). Implications for practice/or policy: The tested technologies can be embedded into digital learning environments.Diagram assessment can be reliably (semi‐)automated.This can reduce teachers' workload and enable real‐time formative feedback.This evaluation enables testing feedback interventions in future work. pubtype: Academic Journal doctype: research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|