Competency model development: The backbone of successful stealth assessments.
Background: Stealth assessment is a learning analytics method, which leverages the collection and analysis of learners' interaction data to make real‐time inferences about their learning. Employed in digital learning environments, stealth assessment helps researchers, educators, and teachers evaluat...
| Publicado en: | Journal of Computer Assisted Learning Vol. 40; no. 6; pp. 2772 - 2790 |
|---|---|
| Autores principales: | , , , , , , , , , , , , |
| Formato: | pictorial review tables/charts Journal Article |
| Publicado: |
Wiley-Blackwell
Dec2024
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=180899666&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 180899666 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 02664909 6M1 jtl: Journal of Computer Assisted Learning issn: 02664909 maglogo: Y pubinfo: dt: Dec2024 vid: 40 iid: 6 pid: 480 pub: Wiley-Blackwell place: Malden, Massachusetts artinfo: ui: 180899666 177793563 180899666 180899666 10.1111/jcal.13025 180899666 ppf: 2772 ppct: 18 formats: tig: atl: Competency model development: The backbone of successful stealth assessments. aug: au: Rahimi, Seyedahmad Almond, Russell Ramírez‐Salgado, Andrea Wusylko, Christine Weisberg, Lauren Song, Yukyeong Lu, Jie Myers, Ted Wang, Bowen Wang, Xiaomaon Francois, Marc Moses, Jennifer Wright, Eric affil: School of Teaching and Learning, University of Florida, Gainesville, USA sug: subj: Clinical Competence Evaluation Video Games Digital Technology Educational Technology Educational Measurement Software Design Communication Latent Structure Analysis Nomenclature Learning Methods Systems Analysis Health Information Thinking Computers and Computerization Creativeness Collaboration ab: Background: Stealth assessment is a learning analytics method, which leverages the collection and analysis of learners' interaction data to make real‐time inferences about their learning. Employed in digital learning environments, stealth assessment helps researchers, educators, and teachers evaluate learners' competencies and customize the learning experience to their specific needs. This adaptability is closely intertwined with theories related to learning, engagement, and motivation. The foundation of stealth assessment rests on evidence‐cantered design (ECD), consisting of four core models: the Competency Model (CM), Evidence Model, Task Model, and Assembly Model. Objective: The first step in designing a stealth assessment entails producing operational definitions of the constructs to be assessed. The CM establishes a framework of latent variables representing the target constructs, as well as their interrelations. When developing the CM, assessment designers must produce clear descriptions of the claims associated with the latent variables and their states, as well as sketch out how the competencies can be measured using assessment tasks. As the designers elaborate on the assessment model, the CM definitions need to be revisited to make sure they work with the scope and constraints of the assessment. Although this is the first step, problems at this stage may result in an assessment that does not meet the intended purpose. The objective of this paper is to elucidate the necessary steps for CM development and to highlight potential challenges in the process, along with strategies for addressing them, particularly for designers without much formal assessment experience. Method: This paper is a methodological exposition, showcasing five examples of CM development. Specifically, we conducted a qualitative retrospective analysis of the CM development procedure, wherein participants unfamiliar with ECD applied the framework and showcased their work. In a stealth assessment course, four groups of students (novice stealth assessment designers) engaged in developing stealth assessments for challenging‐to‐measure constructs across four distinct projects. During their CM development process, we observed various activities to pinpoint areas of difficulty. Results: This paper presents five illustrative examples, including one for assessing physics understanding and four for the development of CMs for four complex competencies: (1) systems thinking, (2) online information credibility evaluation, (3) computational thinking, and (4) collaborative creativity. Each example represents a case in CM development, offering valuable insights. Conclusion: The paper concludes by discussing several guidelines derived from the examples discussed. Emphasizing the importance of dedicating ample time to fine‐tune CMs can significantly enhance the accuracy of assessments related to learners' knowledge and skills. It underscores the significance of qualitative phases in crafting comprehensive stealth assessments, such as CMs, alongside the quantitative statistical modeling and technical aspects of these assessments. Lay Description: What is currently known about this topic?: Stealth assessment represents an unobtrusive, automated formative assessment method.This method uses learning analytics within digital learning environments (e.g., games).The main purpose is to assess and foster the competencies of diverse learners. What does this paper add?: This paper serves as a conceptual and methodological guide.This paper focuses on the critical process of competency model development.Competency model development is a crucial step in the creation of stealth assessments. Implications for practice/or policy: Learning scientists and assessment designers can leverage this paper as a resource.Assessment designers can benefit from this paper and see various examples of the process. pubtype: Academic Journal doctype: pictorial review tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|