A comparative study of AI‐generated and human‐crafted learning objectives in computing education.
Background: In computing education, educators are constantly faced with the challenge of developing new curricula, including learning objectives (LOs), while ensuring that existing courses remain relevant. Large language models (LLMs) were shown to successfully generate a wide spectrum of natural la...
| Publicado en: | Journal of Computer Assisted Learning Vol. 41; no. 1; pp. 1 - 17 |
|---|---|
| Autores principales: | , , , , |
| Formato: | research tables/charts Journal Article |
| Publicado: |
Wiley-Blackwell
Feb2025
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=183981441&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 183981441 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 02664909 6M1 jtl: Journal of Computer Assisted Learning issn: 02664909 maglogo: Y pubinfo: dt: Feb2025 vid: 41 iid: 1 pid: 480 pub: Wiley-Blackwell place: Malden, Massachusetts artinfo: ui: 183981441 183981441 183981441 10.1111/jcal.13092 183981441 ppf: 1 ppct: 16 formats: tig: atl: A comparative study of AI‐generated and human‐crafted learning objectives in computing education. aug: au: Doyle, Aidan Sridhar, Pragnya Agarwal, Arav Savelka, Jaromir Sakr, Majd affil: Language Technologies Institute, Carnegie Mellon University, Pittsburgh Pennsylvania,, USA sug: subj: Artificial Intelligence Curriculum Learning Methods Computer-Assisted Instruction Program Evaluation Human Comparative Studies Natural Language Processing Data Analysis Software Descriptive Statistics Machine Learning Algorithms Linear Regression Random Forest ab: Background: In computing education, educators are constantly faced with the challenge of developing new curricula, including learning objectives (LOs), while ensuring that existing courses remain relevant. Large language models (LLMs) were shown to successfully generate a wide spectrum of natural language artefacts in computing education. Objectives: The objective of this study is to evaluate if it is feasible for a state‐of‐the‐art LLM to support curricular design by proposing lists of high‐quality LOs. Methods: We propose a simple LLM‐powered framework for the automatic generation of LOs. Two human evaluators compare the automatically generated LOs to the human‐crafted ones in terms of their alignment with course goals, meeting the SMART criteria, mutual overlap, and appropriateness of ordering. Results: We found that automatically generated LOs are comparable to LOs authored by instructors in many respects, including being measurable and relevant while exhibiting some limitations (e.g., sometimes not being specific or achievable). LOs were also comparable in their alignment with the high‐level course goals. Finally, auto‐generated LOs were often deemed to be better organised (order, non‐overlap) than the human‐authored ones. Conclusions: Our findings suggest that LLM could support educators in designing their courses by providing reasonable suggestions for LOs. Lay Description: What is already known about this topic: Large language models (LLMs) were shown to successfully generate a wide spectrum of natural language artefacts in computing education.GPT‐4 was used to automatically generate module level learning objectives for a practically oriented university course on artificial intelligence.The generated learning objectives were largely sensible and described key sub‐concepts related to the relevant topics.All the learning objectives were properly started with action verbs, and the distribution of action verbs across the learning objectives for conceptual modules and hands‐on projects was as expected.There is no other work utilising LLMs to generate LOs. What this paper adds: The paper goes beyond action verbs and surface language properties and focuses on evaluating the LOs in terms of the SMART criteria, alignment of the LOs with course goals and module topics, as well as the organization of multiple LOs into meaningful lists, that is, if the LOs were ordered appropriately and did not overlap.This is the first study that performs rigorous evaluation of LOs generated by an LLM (the SMART criteria, alignment, ordering, and non‐overlap). Implications for practice and/or policy: LLMs cannot be relied on to generate high‐quality LOs on their own.LLMs generate reasonable draft LOs that can be utilized by an instructor to author higher‐quality LOs more efficiently. pubtype: Academic Journal doctype: research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|