Work in Progress

SY Tu

Erhel & Jamet, 2013

1. Identify the clarity with which this article states a specific problem to be explored.

Erhel & Jamet, (2013) identify an important problem, but it does not present it as one clear problem statement throughout the article. The abstract notes this problem: that the advantages of digital-game-based learning (DGBL) have not been systematically shown. In the first paragraph of the introduction, the authors expand the question of whether using games to teach educational content is compatible with deep learning. In section 1.3, where the authors discuss contradictory research findings and explain media comparison and previous research gaps, this leads to a clearer research problem, such as: which specific features inside the DGBL influence learning. Then, the authors use experiments to identify that instructional types and simple feedback in DGBL are crucial to producing deep learning. The driving core research problem for the experiment is clearer and could be: what types of instruction and feedback does DGBL support for deep learning and motivation? At the end of the article, the results answer the question, “Is deep learning compatible with serious games?” (Erhel & Jamet, 2013). It would be better to state the core research problem briefly in the abstract and expand it with background and justification in the introduction.

2. Comment on the need for this study and its educational significance as it relates to this problem.

The study makes a strong case for why this study is needed. In the literature review by (Erhel & Jamet, 2013) mention that the effects of DGBL on learning and motivation are inconsistent. For example, about one-third of the studies reviewed by Randel et al. (as cited in Erhel & Jamet, 2013) found a learning benefit; about half of the studies reviewed by Kebritchi et al. (as cited in Erhel & Jamet, 2013) found that the quality of learning was not improved. Concerning the uncertainty, schools and educational designers must evaluate DGBL’s effectiveness before investing time and money in creating and implementing it. Furthermore, the authors identify an important research gap for motivation and DGBL improvement: the wording of instructions (introducing the game) and feedback. This is a key takeaway for educators and designers looking to design effective educational games. Furthermore, the study can practically apply the findings to teachers, game designers, health educators, etc., and evaluate the learning experience and outcomes. 

3. Comment on whether the problem is researchable. Can it be investigated through the collection and analysis of data?

The problem can be researched, but some parts are harder to measure than others. Instruction types are straightforward because the researchers control the type of instruction by presenting ASTRA as learning or entertainment. Erhel & Jamet, (2013) measured learning through recall, paraphrase, and inference questions. Inference questions indicated deep learning because learners had to connect questions to what they learned and already knew. Motivation is harder to measure because it relies on self-report. The questionnaire addresses mastery goals, performance goals, and intrinsic motivation, allowing these variables to be compared across experimental groups. Self-regulation is even harder to measure, and the authors used it to explain the results of experiment 2. Azevedo et al. (2010) argue that self-regulation develops during learning and should be examined as an event instead of an aptitude, so it is best captured with think-aloud protocols, log files, and eye tracking. These methods provide more direct evidence but require extensive coding and analysis. Therefore, the problem is researchable, but beyond questionnaires and test scores, the measured data are needed and important for better analysis.

4. Critique the author’s conceptual framework?

The authors did not present the conceptual framework as a complete model. The conceptual framework is composed of sections of the literature review. 

• (Section 1.1) DGBL is considered an educational activity that uses interactive entertainment to achieve educational goals. According to the authors’ citations and broad definition, DGBL includes competition and applies to both digital and traditional games. However, ASTRA, an animated pedagogical agent with instructional videos, questions, quizzes, and feedback, has no competitive features, which the authors did not explain. Therefore, the concept of DGBL is not really operationalized.
• (Section 1.2) Motivation is a process that can physiologically influence people’s behavioral performance and direction. The achievement goals and intrinsic motivation are defined and measured with questionnaire items including “What is being assessed” and “Example of items” (Erhel & Jamet, 2013, p.160). This is the most developed portion of the framework. But Flow theory is only mentioned without being measured.
• (Section 1.3) Explanation of choosing a value-added approach other than comparing a game to conventional instruction. There is not much conceptual argument about how learning occurs.
• (Section 1.4) Justification of the core framework of instruction and depth of processing. Instruction can affect learners’ goals and the cognitive process they use. Instruction that promotes deep learning leads to deeper processing, while instruction that promotes entertainment leads to surface processing. Deep learning is operationalized via inference questions, while memorization is operationalized via paraphrase questions.

From these key points, the base of the conceptual framework can presumably be described as a linear direction, as instruction affects learner goals; goals affect deep learning, then deeper processing and learning outcomes. The authors mentioned the relationship between entertainment and instruction, and feedback in the experiments. These concepts don’t seem to be included in the core framework, so it would be better to connect them to the core framework and build a diagram to illustrate it.

5. How effectively does the author tie the study to relevant theory and prior research? Are all cited references relevant to the problem under investigation?

The authors did an excellent job connecting certain aspects of their study to past research; particularly, the relationship between instruction and deep learning. For example, Van den Broek et al. (2001, as cited in Erhel & Jamet, 2013), compared study and entertainment in instructions and found that study instructions led to more inference, and the authors built Experiment 1 based on this theoretical foundation to determine if different instructions have similar effects in DGBL. However, there are some areas of the literature review that seem not to directly relate to the study. Section 1.3 spends a lot of its portion listing positive and negative findings from previous game studies that are not consistent, and this could be briefed instead. Flow theory is not measured afterward, so it might not necessarily be mentioned in Section 1.2. Additionally, the study participants were university students aged 18-26, so the initial reference to leisure time for children and adolescents spending playing time is not directly relevant (Erhel & Jamet, 2013). Also, self-regulation and the research of Moos and Azevedo (2006, as cited in Erhel & Jamet, 2013) support much of the authors’ explanation in General Discussion, but are not mentioned in the literature review.

Most importantly, some concepts used later in the study are not covered by the literature review. First, feedback is the key concept only mentioned in Experiment 2 examining knowledge of correct response. Therefore, readers would not realize that feedback is an important element of the whole study. Second, in Section 1.2, motivation theory, there is no explanation of the dimension of approach-avoidance used in the questionnaire in Experiment 2, so readers may feel unprepared to understand the interpretation of the findings.

Finally, most references are relevant, but some not relevant references can be briefed and should include feedback, approach-avoidance goals, and self-regulation in the literature review.

6. Does the literature review conclude with a brief summary of the literature and its implication for the problem investigated?

The literature review is missing a clearly defined summary at the end in Section1.4. The authors did not formally conclude the literature review and brought the literature gradually into two locations. First, in Section 1.3, the authors conclude that the current research has shown no clear evidence of positive effects of educational games on learning and motivation. Based on the inconsistent study results and limitations of media-comparison studies, the authors use the value-added approach (Erhel & Jamet, 2013, p.157). They compare two versions of the same ASTRA environment, changing only the instructions. The end of Section 1.3 summarizes the problem and explains the study’s methodology. Second, at the beginning of Experiment 1, the authors connect the methodological decision with theories discussed in Section 1.2’s contrasting the ideas of entertainment instructions increasing motivation and learning instructions promoting cognitive processing and in Section 1.4’s research on instruction and deep processing. Though the synthesis links different topics in the literature review, the concept of feedback is not discussed until Experiment 2. To summarize all sections in one place would be much clearer.

7. Evaluate the clarity and appropriateness of the research questions or hypotheses?

The authors have some research questions, but they never state them clearly. There are three statements that seem to serve this function. For Experiment 1, the authors wanted to see if the effects of instructions that were found in text-based learning would also be found in DGBL (Erhel & Jamet, 2013, p.158). For Experiment 2, they wanted to see if KCR feedback would affect the learning strategies that were produced by learning and entertainment instructions (p.162). The most general question, “Is deep learning compatible with serious games?” does not appear until General Discussion (p.165). The first question is relevant but too long. The term “reading phase” is used incorrectly in the paper and easily confuses readers, because many materials in ASTRA are presented in other formats such as audio, video, a pedagogical agent, questions, and quizzes. No metric is defined for measuring outcomes in Experiment 2. The researchers did not directly observe or measure the learning strategies used by participants but instead evaluated memory, paraphrasing, and inference, and measured achievement goals and intrinsic motivation. Therefore, the wording in the paper is too long for the measures that were taken. This design does not allow the study to prove that feedback changed how the instructions worked. The questions and hypotheses are relevant and testable, but they should be grouped together in one clearly labeled section at the end of the introduction.

8. Critique the appropriateness and adequacy of the study’s design in relation to the research questions or hypotheses?

The authors of the study gave a good rationale for using a value-added design and explained that media-comparison studies are often affected by difficult control factors such as format, pace, educational content, and even teachers’ social presence (Erhel & Jamet, 2013). So, it would be difficult to establish whether the differences found between a digital game and conventional classroom instruction were caused by the game itself or by other difficult control factors between the two conditions. Keeping the ASTRA environment and the educational content the same, the authors were able to isolate the design feature of interest to focus on the learning and entertainment instruction comparison.

However, the authors mention the value-added approach has some drawbacks. It can only show which version of the ASTRA learning environment is more effective than another. It cannot provide evidence as to whether ASTRA learning environment is more effective than other environment settings. Each example was a between-subjects/groups design with random assignment to the two levels of the single independent variable, learning vs. entertainment instructions. The instruction content, multimedia, quizzes, and the pedagogical agent remained constant between groups. In Drost, (2011) research, the order of the four diseases was counterbalanced across participants, which reduced possible order effects such as fatigue, declining attention, etc. Experiment 1 applied to this design well.

One main weakness is that in Experiment 1, no feedback was provided, whereas in Experiment 2 both groups of participants received feedback. Erhel & Jamet, (2013) conclusion that the negative effect of entertainment instructions was canceled by the factor of feedback was based on comparing results between two separate experiments. In fact, a statistical factorial design was recommended to assess the significance of the interactive effect of instruction depending on whether feedback is present directly.

9. Critique the appropriateness and adequacy of the study’s design in relation to the research questions or hypotheses?

The sample of students used in the above research was selected using a nonprobability convenience sample. This means that the sample was selected from a pool of people that were available to the researchers. Because the researcher used a convenience sample, the results of the research cannot be generalized to other groups of people (Siegle, 2019). In addition to the sample being selected using a convenience sample, the researchers applied a purposive eligibility criteria by excluding medical and allied-health students, and participants with prior-knowledge scores greater than the cutoff. After the sample was selected, the participants were randomly assigned to the different instructional conditions, but this does not make the convenience sample a random sample. The purpose of selecting learners with limited knowledge of the four diseases was adequate for the experiments. The sample of French undergraduates, ages 18-26, with limited medical knowledge and who completed the short learning session of the experiment, can be generalized to other French undergraduates with similar characteristics. However, the findings can’t be generalized to older adults, medical students, children or other groups. In addition, the numbers in the description of Experiment 1 are inconsistent, including the number of participants and the mean age of the groups. For Experiment 2, the number of participants differs from the number in the analysis report. Although the discrepancies might be due to typos or not affect the analyzed results, it is still better to review the data and correct them.

10. Critique the adequacy of the study’s procedures and materials (e.g., interventions, interview protocols, data collection procedures, instruments).

The procedures were controlled by exposing two groups to the same ASTRA environment, content, questions, quizzes, multimedia, and pedagogical agent. Only difference is the instruction screen, as a learning module for one group, and a game/entertainment module for the other group (Erhel & Jamet, 2013). The procedure of conduct is in line with the study’s focus on intentional versus incidental learning, such as materials and knowledge questionnaires developed prior to data collection, and participants were informed about assessment of learning and motivation after completion of the study.

However, the motivation questionnaire was the most challenging part of this study. The design of the questions was based on the existing measure without trial testing or examining the reliability, and some of them may not be relevant to the learning environment such as performance comparison of a single-person activity between people. So, it can’t be certain the accuracy of measuring motivation. Additionally, there was no manipulation check to ensure that the participants’ perceptions of whether the activity is learning or a game were the same as what the instructor said. So, the learning versus gaming mindsets are not certain. And there was no justification of the chosen cutoff score, the is the concerns whether the participants have excessive knowledge prior to experiments.

11. Critique the appropriateness and quality (e.g., reliability, validity) of the measures used

Validity and reliability are important and could affect the interpretation of the results. Validity is about whether a tool measures what it is supposed to measure. According to Drost, (2011), reliability means the measurements remain consistent within the test and between raters. There is not enough evidence about the reliability of the measures for the achievement goal or intrinsic motivation subscales (Erhel & Jamet, 2013). Also, the authors do not state the number of raters scored the responses, because it affects the reliability of the measures.

The learning measures showed some evidence of validity. The learning materials created in collaboration with a physician increase the accuracy of health content; separating paraphrase and inference questions can distinguish between surface and deep learning, both these actions can improve validity. However, the inference scores, based only on the indirect measure of learning strategies, are somewhat weak.

The motivation questionnaire was the least valid of the measures, as it only referred to the previous measures and was not relevant to the learning environment, such as performance comparison of a single-person activity between people. In addition, the prior-knowledge test had weak validity because it did not measure knowledge of the four diseases taught in ASTRA.

As for the statistical conclusion validity, the Experiment 2 inference results were at p=0.05 without reporting a correction. The Cohen’s d, effect size, and a confidence interval were not provided, as these statistics can support more than significance.

12. Critique the author’s discussion of the methodological and/or conceptual limitations of the results.

The authors mentioned three limitations to their study (Erhel & Jamet, 2013).

• Limited game-like interactivity. Participants to show individual differences even if they have similar motivation and cognitive processes. Also, the study did not mention if the game is fully digital or not, so the results cannot be generalized to every digital game.
• Easy recall quizzes. The authors recognize that quizzes used in this study were not difficult, so the participants’ average scores were high, meaning they rarely received
• the feedback. So the results cannot be used to support the claim that feedback is beneficial for learning.
• Use of offline data. The authors collected post test scores and motivation measures rather than collecting data during the learning process, such as log files, think-aloud protocols, or eye-tracking (Azevedo et al., 2010).

In addition to these three points, small convenience sample, and evidence of reliability as mentioned in previous problems are also limited to the confidence of the analyzed results.

13. How consistent and comprehensive are the author’s conclusions with the reported results?

The authors overstate the results of Experiment 2. In Section 2.3, the measure of recall, paragraph, and motivation did not show a significant difference between learning and entertainment instruction groups. But the learning instruction group topped the entertainment instruction group on the four inference questions. The conclusion was that deep processing can be observed in the learning-instruction group, but there were no measures of cognitive processing and mental effort. Section 3.2, the entertainment instruction group performed better on the inference questions with feedback than the learning instruction group. Also, the learning instruction group has more performance avoidance goals than the entertainment instruction group. Erhel & Jamet, (2013) conclude that entertainment instructions with feedback promote deep processing and can increase “motivational investment”. This conclusion is not fully supported by the data since there were no significant differences on measures of intrinsic motivation and other achievement goal measures. Section 4 also overstates the results. The authors claim that Experiment 1 suggested that learning instructions may support inference performance better than entertainment instruction when feedback is absent. However, since the feedback was tested in two different experiments with different participants, it can’t draw conclusions about its effect.

14. How well did the author relate the results to the study’s theoretical base?

The authors presented a reasonable connection between their findings and the theoretical foundation for the study. The authors adopt Van den Broek et al. Section 1.4 of the literature review to support their findings with respect to the influence of instructions on learners’ information processing. Also, the authors relied on Beentjes et al. (1993) and Van Asselen et al. (2006, as cited in Erhel & Jamet, 2013) to discuss the shallow processing that result from entertainment instructions. Finally, the authors referred to Graesser et al.’s (2009) discussion of digital game-based learning and Moos and Marroquin (2010, as cited in Erhel & Jamet, 2013) to relate learners’ goals to their processing strategies. The Experiment 2 discussion made much stronger theoretical connections than the Experiment 1 discussion. The authors linked feedback to cognitive-load theory, specifically to the work of Sweller et al. (1998) and Kester et al. (2007). In addition, they compared their findings to Leuther’s (1993) research on feedback in educational games. The authors also referred to the different types of feedback identified by Shute (2008), More and Mayer (2005) and Mayer and Johnson (2010, as cited in Erhel & Jamet, 2013). This provided a reasonable explanation for the role of KCR feedback in guiding learners’ attention and supporting learning. The coverage of the theoretical background was not even. The authors used the studies of Moos and Azevedo (2006) and Spinath and Spinath (2005) in the literature review. Also, the authors did not use the concepts of flow as described by Csikszentmihalyi (1988), Liu et al. (2011), and the theory of intrinsic motivation as described by Deci and Ryan (2000) when interpreting the findings regarding motivation.

15. In your view, what is the significance of the study, and what are its primary implications for theory, future research, and practice?

The importance of this study is moderate but meaningful. It moves the study of whether video games can promote learning to focus on what characteristics of games can promote learning. The study provides some insights into how instructional framing and feedback on a game can interact to promote learning (Erhel & Jamet, 2013). For theoretical contributions, this study builds upon previous studies that have examined the effect of reading purposes in digital game-based learning environment. Previous studies have found that learning goals affect both attention and processing (Van den Broek et al., 2001; McCrudden et al., 2010, as cited in Erhel & Jamet, 2013). There is some indication of a link between instructional framing, fear of failure, self-regulation, and cognitive processing; however, these variables were not directly measured in the study. Future research could investigate the effects of learning versus entertainment instructions on learning and feedback versus no feedback in a 2×2 factorial design. Future researchers should include larger sample sizes, reliable measures of motivation, harder quiz questions, and a delayed post-test. Log files, think-aloud protocols or eye tracking could also provide direct evidence of cognitive and self-regulatory processes (Azevedo et al., 2010). A comparison between KCR feedback and explanatory feedback would allow researchers to determine which type of feedback is most effective.

Reference

Azevedo, R., Moos, D. C., Johnson, A. M., & Chauncey, A. D. (2010). Measuring Cognitive and Metacognitive Regulatory Processes During Hypermedia Learning: Issues and Challenges. Educational Psychologist, 45(4), 210–223. https://doi.org/10.1080/00461520.2010.515934

Drost, E. A. (2011). Validity and Reliability in Social Science Research. Education Research and Perspectives, 38(1), 105–123. https://doi.org/10.70953/ERPv38.11005

Erhel, S., & Jamet, É. (2013). Digital Game-Based Learning: Impact of Instructions and Feedback on Motivation and Learning Effectiveness. Computers & Education, 67, 156–167. https://doi.org/10.1016/j.compedu.2013.02.019

Siegle, D. (2019, June 20). Sampling. University of Connecticut. Educational Research Basics. https://researchbasics.education.uconn.edu/sampling/