Computers & Education review

This article proposes to evaluate the effects of adding adaptivity and personalization to an educational game for teaching computational thinking (CT). The article reports on an experiment comparing use of a game called AutoThinking with traditional teaching methods for students’s CT learning, interest in future CT learning, satisfaction with CT learning, and reported flow state.

The article is well-written, well-researched, and careful with statistics. I had a few quibbles with claims made in the literature review (for example, the article makes claims about differences in learning CT knowledge and CT skills, but the review does not address the epistemological difference between these constructs), but it was generally thorough and interesting. The article builds on what appears to be a substantial body of authors’ prior work (as it is anonymous, I am unable to say much here), using validated instruments and procedures.

However, I believe there are major issues in the relationship between the experimental results and what they are purported to mean. At core, the issue is that this article does not theorize the design of AutoThinking. AutoThinking is claimed to be an educational game without addressing the substantial literature on games for learning (some of which would likely take issue with AutoThinking being called a game) and how game mechanics might support learning. We are told that AutoThinking features adaptivity and personalization without explaining what these terms mean or theorizing why they might support learning.

The article claims that AutoThinking “reduces cognitive load” by using “icons rather than syntax.” (p. 9) I do not accept this claim without justification. For example, similar icon-based programming interfaces are used by Kibo, Botley, Cubetto, Mochi Robot, Project Bloks, and others. One common issue this strategy encounters is that children can confuse the spatial arrangement of the icons and the grid of the action space. This leads to misunderstandings like thinking a down-arrow icon will cause the icon below the current icon to execute next. In such cases, an icon-based interface could actually increase cognitive load compared to alternatives. We need to hear more.

The failure to theorize the design of the intervention cuts to the heart of the experimental design. Using AutoThinking is proposed as an experimental condition with a “traditional technology-enhanced learning approach” described as “a PowerPoint presentation with Multimedia” as a control (p. 12). I do not see how this can be justified as a control: technologies with designed interfaces were involved here too. Without an analysis of the deisgn of AutoThinking or the design of the alternative, we are left with statistics showing a difference between two conditions, but no way to know way to understand what was meaningfully different about the conditions, much less which of them mattered. If the authors wanted to explore the effects of adding uncertainty to the programming scenarios (“adaptivity” and “personalization”), it might have been better to contrast students’ performances and experiences in two different versions of the same interface.

One smaller consequence of not addressing affordances or game mechanics is that we cannot know how similar the sample problem from the CT assessment (Fig. 5, p. 33) is to the interfaces used in the “traditional” control condition. It is well-known that transfer across programming interfaces is difficult, so it could be that the experimental (AutoThinking) students’ better performance on the assessment was just a result of the assessment feeling more familiar.

Without knowing more about the issues raised above, I cannot say whether they are fatal to this paper. It could be that providing more detail about the differences between the experimental condition and the control would rescue the experiment, particularly if the case can be made that the experiment represents a specific difference in affordances, and that this difference can be justified as the reason for the difference in performance.

In my own personal taste, the most interesting part of the article was the student work shown in Fig. 3 (p. 31), where it is possible to infer some of the unanswered questions about the design of the interface, how the design shaped student practice with the interface, and how students understood these practices to constitute durable computational thinking knowledge and skills. A deeper analysis here would also address the issue that this article proposes a new medium for learning computational thinking without discussing the substantially-researched relationship between computational thinking and the media used for representation and thinking about it.

USEFUL FOR ME: Review of empirical studies on assessment of CT. Tang, Yin, Lin, Hadad, and Zhai (2020)