| Sumario: | Background: Learners' writing skills are critical to their academic and professional development. Previous studies have shown that learners' self‐assessment during writing is essential for assessing their writing products and monitoring their writing processes. However, conducting practical self‐assessments of writing remains challenging for learners without help, such as formative feedback. Objectives: To facilitate learners' self‐assessment in writing, we developed a writing analytics toolkit and used data visualisation and cutting‐edge machine learning technology that provides real‐time and formative feedback to learners. Methods: To investigate whether our newly‐developed tool affects the accuracy and process of learners' self‐assessment, we conducted a lab study. We assigned 59 learners to complete writing (2 h) and revising (1 h) tasks. During the revision stage, we randomly assigned the learners to two groups: one group used the writing analytics toolkit while the second group was not granted access to the toolkit. Learners' self‐assessment accuracy and process of self‐assessment were compared between the two groups. Results: In our study, we found the toolkit helped learners in the experimental group improve the self‐assessment accuracy of their writing products compared to the learners in the control group. In addition, we also found that the affordances of the toolkit affected the learners' self‐assessment process, and poor design affordances may have prevented the learners from reflecting by themselves. Conclusions: Together, our empirical study shed light on the design of future writing analytics tools which aim at improving learners' self‐assessment during formative writing processes. Lay Description: What is already known about this topic: Learners' self‐assessment during writing is essential for assessing their writing products and monitoring their writing processes.Formative feedback is vital for improving learners' self‐assessment capabilities.Writing analytics tools offer feedback to learners, however, how these tools facilitate self‐assessment remains under investigated. What this paper adds: We developed a writing analytics toolkit and used data visualisation and cutting‐edge artificial intelligence technology that provides real‐time and formative feedback to learners.We found the toolkit helped learners improve the self‐assessment accuracy of their writing products.We also found that the affordances of the toolkit affected the learners' self‐assessment process, and poor design affordances may have prevented the learners from reflecting by themselves. Implications for practice and/or policy: It is crucial for learning analytics and educational technology developers to design feedback tools and provide formative assessments.Learning analytics tools should have high affordance so that learners can easily understand the formative feedback, thereby promoting actionable self‐assessment.
|
|---|