Intelligent Tutoring in Serious Games

BACK TO AGENDA

Thursday August 24, 2006

Michael van Lent, Institute for Creative Technologies/USC
9:15 - 10:15 

Eight Serious Games at ICT:  Lessons Learned and Challenges Identified

Over the past five years ICT has led the design, development and (to various degrees) deployment of more than 10 serious games.  This talk will briefly describe eight of these games and discuss lessonslearned (both the hard way and the easy way) and challenges identified.

Lewis Johnson, Information Sciences Institute/USC
10:30 - 11:30

Tactical Language and Culture Training System  

University of Southern California and Tactical Language Training, LLC
http://www.tacticallanguage.com

The Tactical Language and Culture Training System (TLCTS) is a PC-based interactive learning system employing serious game design concepts.  TLCTS is a combination of serious game and interactive multimedia learning system.  Two serious games are included: an Arcade Game in which the user learns to give and receive directions in the target language, and a Mission Game in which the user employs his or her communication skills to carry out a mission.  The Skill Builder multimedia learning system includes interactive lessons and exercises that help learners acquire the skills that they need to play the games.  Both make extensive of speech recognition technology and are highly interactive.

From the beginning, the project employed an iterative design approach – create preliminary versions of the game, and its features, get them in the hands of prospective users quickly, note the problems that arise, and try to correct those problems.  This was helpful in refining the design and evaluating and refining the technology.  In terms of design, one of the key questions was how best to combine game elements and multimedia learning elements to maximize learning.  TLCTS employs a number of feedback techniques, including both a virtual tutor character that plays a role within the game, and heads-up displays that provide feedback regarding character attitudes and player options.   A combination of such techniques has proven most effective.

TLCTS is in regular use at military training sites in the US and overseas.  Over 1500 copies have been distributed.  It will soon be available for on-line distribution to members of the US Special Forces and US Marine Corps.

Bob Hausmann, University of Pittsburgh
11:30 - 12:15
An analysis of generative dialogue patterns across interactive learning environments: Explanation, elaboration, and co-construction

How well a student learns from any given experience critically depends on the types of interactions that occur between the student and the learning environment. The interactions can take place between the student and an intelligent tutoring system, a human tutor, a collaborative peer, or any combination thereof. The effectiveness of these interactions can be assessed through the process of categorizing the interaction patterns and correlating them with learning gains. To better understand the design
constraints for interactive learning environments, it may be informative to examine a variety of learning environments in an effort to identify a general class of effective interaction patterns. Preliminary results suggest three patterns that seem to occur across environments: explanation, elaborations, and co-construction. Evidence demonstrating the prevalence of these interaction patterns will be presented, along with a comparison of their effectiveness relative to other interaction patterns that also occur frequently, but have not been shown to facilitate learning.

Rich Mayer, University of California-Santa Barbara
1:00 - 2:00  [word]
Research-Based Principles for the Design of Multimedia Learning Environments

The design of web-based training should be based on scientific research and grounded in a cognitive theory of how people learn.   In this presentation I examine three classic problems in the design of web-based learning environments: (1) the material is presented in a way that is insensitive the learner's cognitive processing system, (2) the content is inherently difficult for the learner, and (3) the material is presented in a way that is unfriendly to the learner.  On the basis of the cognitive theory of multimedia learning (Mayer, 2001; Mayer & Moreno, 2003; Mayer, 2005) and on a body of scientifically rigorous empirical research involving approximately 80 experimental comparisons (Mayer, 2001; Mayer, 2005b, 2005c, 2005dd), I describe solutions to each e-learning problem. When the material is presented in an insensitive way, the solutions include weeding (in which extraneous words, sounds, and graphics are eliminated), decaptioning (in which presentations consist of animation and narration rather than animation, narration, and onscreen text), signaling (in which essential words and graphics are highlighted), aligning (in which corresponding words and graphics are presented near rather than far from each other on the page or screen), and synchronizing (in which corresponding narration and animation are presented simultaneously rather than successively).   When the content is difficult, the solutions include segmenting (in which a lesson is broken into segments that can be paced by the learner rather than given as a continuous presentation), pre-training (in which the learner is given pre-training in the names and characteristics of the key concepts before the lesson), and off-loading (in which material is presented as graphics and spoken text rather than graphics and printed text). When the material is presented in an unfriendly way, the solutions include personalizing (in which the words are presented in conversational style using "I" and "you" rather than formal style) and articulating (in which the words are spoken in a clear human voice rather than a machine voice).  Well-designed web-based training can result in large improvements in learners' performance on tests of transfer in which they are able to use what was taught to solve new problems.

Bill Swartout, Institute for Creative Technologies/USC
3:15 - 4:00     

Negotiating with Virtual Humans

Virtual humans are computer generated characters that are intended
to serve as surrogates for real people in a variety of educational and
training simulations.  Virtual humans are not pre-scripted, but instead
dynamically reason about what is going in their simulated world and respond
appropriately.  Ideally, virtual humans should be able to communicate in a
fluid, natural way using the full repertoire of verbal and non-verbal
communication techniques that humans use, they should model and exhibit
emotions, and reason about their own and other's beliefs, desires and
intentions. While we have not yet achieved this goal, significant progress
has been made.  In this talk, I will describe the ICT virtual human effort
and work we have done in developing virtual humans to support negotiation
training.

Bruce Roberts, BBN
4:00 - 4:45
Real-time coaching in virtual worlds

I will summarize the lessons learned from several projects that have focused on providing coaching feedback during a “real-time” task, one in which the trainee must stay engaged continuously in a dynamically changing virtual environment. Challenges include questions of whether and when to intervene, the duration and style of an intervention, its modality and content. I’ll will also touch on the architecture that has evolved for dealing with coaching in real-time tasks.

Tom Livak, MathWorks
5:00 - 5:45     
Collaborative Warrior Tutoring

The goal of the Warrior Tutoring project is to develop an intelligent tutoring system to train US soldiers. One main difference in this domain is that one of the main skills to be learned is cooperation between teammates, so the tutor must emphasize collaboration as a skill. In addition, to help train this skill the system must be able to run in real-time, and provide intelligent computer generated teammates. This system is the first real-time, multi-user, model tracing tutor with simulated teammates. 

James Lester, North Carolina State University
9:00 - 10:00

Narrative-Centered Learning Environments

Recent years have seen significant growth in work on the role of narrativein interactive learning environments in general and intelligent tutoring systems in particular.  A key challenge posed by these environments is narrative planning, in which a director agent orchestrates all of the events in a storyworld to create an optimal experience for a student, who is herself an active participant in the unfolding story.  To effectively plan
narratives for learning environments, a director agent should coordinate tutorial and narrative planning, recognize students' goals, and reason about students' affective states, all the while coping with the inherent uncertainty in the task and working within the real-time performance constraints of interactive systems.  We have recently launched the Crystal Island project, a narrative-centered learning environment for the domain of microbiology for middle school students.  Leveraging the Half-Life 2 game
engine, Crystal Island will generate interactive science mysteries for the domain of microbiology for middle school students.   We report on preliminary results on narrative-tutorial planning and goal recognition, as well as early work on affective reasoning including empathy modeling for companion agents and the induction of self-efficacy student models.

Mark Riedl, Institute for Creative Technologies/USC
10:00 - 10:45

Emergent and Guided Narrative for Training and Education in Virtual Worlds

Kurt VanLehn, University of Pittsburgh
11:00 - 12:00

When are tutorial dialogues more effective than reading?

It is often assumed that engaging in a one-on-one dialogue with a tutor is more effective than listening to a lecture or reading a text.  Although earlier experiments have not always supported this hypothesis, this may be due in part to allowing the tutors to cover different content than the non-interactive instruction.  In 7 experiments, we tested the interaction hypothesis under the constraint that (1) all students covered the same content during instruction, (2) the task domain was qualitative physics, (3) the instruction was in natural language, as opposed to mathematical or other formal languages, and (4) the instruction conformed with a widely observed pattern in human tutoring, Graesser, Person and Magliano’s five-step frame.  The experiments compared 2 kinds of human tutoring (spoken and computer-mediated) with 2 kinds of natural-language-based computer tutoring (Why2-Atlas and Why2-AutoTutor) and 3 control conditions that involved studying texts.  The results depended on whether the students’ preparation matched the content of the instruction.  When novices (students who had not taken college physics) studied content that was written for intermediates (students who had taken college physics), then tutorial dialogue was reliably more beneficial than less interactive instruction, with large effect sizes.  When novices studied material written for novices, or intermediates studied material written for intermediates, then tutorial dialogue was not reliably more effective than the text-based control conditions. 

Jen Solberg, Army Research Institute
12:45 - 1:30      

Researching the Effectiveness of Serious Games: Asking the Right Questions 

As the use of serious games in teaching environments increases, new issues about the effectiveness of these games emerge. What kinds of questions should researchers be asking in order to develop games as valid instructional tools? I argue that the effectiveness of a serious game depends on a complex interaction between the player and the game, and that understanding the factors involved in this interaction are key to informing serious game design. I will discuss potential research questions relating to
the player, the game, and the delivery process. Also, I will discuss potential measures that could be used to address these questions. Finally, I will comment on future directions for the field of researching serious games. 

Dick Clark, University of Southern California
1:30 - 2:30 [word]
Evaluating the Learning and Motivation Effects of Serious Games

O’Neil, Wainess and Baker (2005) surveyed over 4,000 published accounts of serious games and found only 19 studies in peer-reviewed journals where either qualitative and/or quantitative data about learning or motivation had been assessed. A similar result was reported by Grendler (1996) in an earlier review.  None of the peer reviewed studies reported compelling evidence that games produced significantly more learning or motivation than less expensive, traditional instructional platforms. 

This presentation will suggest three important evaluation strategies for serious games which, if implemented in future assessments, might provide more compelling evidence for the benefits of serious games.  The three strategies are:  1) Use reliable and valid tests of learning and motivation before, during and after games; 2) Build in robust pedagogical and motivational strategies (avoid minimally guided pedagogy); and 3) Offer a viable, robust non-game alternative way to teach the same knowledge that uses the same or similar pedagogical strategy (compare game and non-game instruction and avoid “straw man” comparisons).

Henry Marshall, US Army Research Development and Engineering Command
3:30 - 4:15

H. Chad Lane, Institute for Creative Technologies/USC
4:15 - 5:00
Winning and Learning: Making the link between domain content and game success explicit