Investigating the reliability of aggregate measurements of learning process data: From theory to practice.
Background: Learning analytics (LA) research often aggregates learning process data to extract measurements indicating constructs of interest. However, the warranty that such aggregation will produce reliable measurements has not been explicitly examined. The reliability evidence of aggregate measur...
| Publicado en: | Journal of Computer Assisted Learning Vol. 40; no. 3; pp. 1295 - 1309 |
|---|---|
| Autores principales: | , , , , |
| Formato: | algorithm research tables/charts Journal Article |
| Publicado: |
Wiley-Blackwell
Jun2024
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=177193481&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 177193481 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 02664909 6M1 jtl: Journal of Computer Assisted Learning issn: 02664909 maglogo: Y pubinfo: dt: Jun2024 vid: 40 iid: 3 pid: 480 pub: Wiley-Blackwell place: Malden, Massachusetts artinfo: ui: 177193481 175187651 177193481 177193481 10.1111/jcal.12951 177193481 ppf: 1295 ppct: 14 formats: tig: atl: Investigating the reliability of aggregate measurements of learning process data: From theory to practice. aug: au: Zhang, Yingbin Ye, Yafei Paquette, Luc Wang, Yibo Hu, Xiaoyong affil: Institute of Artificial Intelligence in Education, South China Normal University, Guangzhou, China sug: subj: Reliability Evaluation Learning Methods Data Analytics Theory-Practice Relationship Human Male Female Psychometrics Academic Performance Conceptual Framework Research Personnel Internal Consistency Test-Retest Reliability Random Assignment Computer Simulation Computers and Computerization Machine Learning Funding Source Male Female ab: Background: Learning analytics (LA) research often aggregates learning process data to extract measurements indicating constructs of interest. However, the warranty that such aggregation will produce reliable measurements has not been explicitly examined. The reliability evidence of aggregate measurements has rarely been reported, leaving an implicit assumption that such measurements are free of errors. Objectives: This study addresses these gaps by investigating the psychometric pros and cons of aggregate measurements. Methods: This study proposes a framework for aggregating process data, which includes the conditions where aggregation is appropriate, and a guideline for selecting the proper reliability evidence and the computing procedure. We support and demonstrate the framework by analysing undergraduates' academic procrastination and programming proficiency in an introductory computer science course. Results and Conclusion: Aggregation over a period is acceptable and may improve measurement reliability only if the construct of interest is stable during the period. Otherwise, aggregation may mask meaningful changes in behaviours and should be avoided. While selecting the type of reliability evidence, a critical question is whether process data can be regarded as repeated measurements. Another question is whether the lengths of processes are unequal and individual events are unreliable. If the answer to the second question is no, segmenting each process into a fixed number of bins assists in computing the reliability coefficient. Major Takeaways: The proposed framework can be a general guideline for aggregating process data in LA research. Researchers should check and report the reliability evidence for aggregate measurements before the ensuing interpretation. Lay Description: What is currently known about this topic: Aggregating learning process data is common in learning analytics.The psychometric pros and cons of aggregating process data are rarely examined. What this paper adds: If the construct of interest is stable during a period, aggregation over this period is desirable.Aggregation over a long period masks meaningful changes in learning behaviours.A guideline for choosing proper reliability coefficients for aggregate measurements is proposed.Methods for computing reliability estimates when processes vary in length are provided. Implications for practice: Report reliability evidence for aggregate measurements for the sake of psychometric rigour.A short period causes unreliable action‐related indicators in learning analytics dashboards.A long period causes inaccurate indicators of learners' current state. pubtype: Academic Journal doctype: algorithm research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|