Integrated learning for interactive synthetic characters

Abstract

The ability to learn is a potentially compelling and important quality for interactive synthetic characters. To that end, we describe a practical approach to real-time learning for synthetic characters. Our implementation is grounded in the techniques of reinforcement learning and informed by insights from animal training. It simpliÞes the learning task for characters by (a) enabling them to take advantage of predictable regularities in their world, (b) allowing them to make maximal use of any supervisory signals, and (c) making them easy to train by humans.We built an autonomous animated dog that can be trained with a technique used to train real dogs called "clicker training". Capabilities demonstrated include being trained to recognize and use acoustic patterns as cues for actions, as well as to synthesize new actions from novel paths through its motion space.A key contribution of this paper is to demonstrate that by addressing the three problems of state, action, and state-action space discovery at the same time, the solution for each becomes easier. Finally, we articulate heuristics and design principles that make learning practical for synthetic characters.

Experiments in socially guided exploration: lessons learned in building robots that learn with and without human teachers

“…The cognitive and learning system extends the C5M architecture [Blumberg02]. The Perception and Belief Systems are most relevant to the learning abilities described in this paper.…”

“…For example, many prior works have given a human trainer control a reinforcement learner's reward [Blumberg02,Kaplan02,Saksida98], allow a human to provide advice [Clouse92,Maclin05], or have the human tele-operate the agent during training [Smart02]. Exploration approaches have the benefit that learning does not require the human's undivided attention.…”

TAMER: Training an Agent Manually via Evaluative Reinforcement

“…Kaplan et al [13] and Blumberg et al [14] respectively implement clicker training on a robotic and a simulated dog. Blumberg et al's system is especially interesting, allowing the dog to learn multi-action sequences and associate them with verbal cues.…”

Toward gesture-based behavior authoring

“…Through reinforcement learning, the virtual dog in [3] gradually learns to perform new behaviors. Many other works also proposed similar ways to animate virtual characters [12,32,6].…”

“…Unfortunately, behavior modeling has long been a difficult and cumbersome task. To enable nonprogrammers to create such "behavioral animation" [3,27,32] more easily, a large amount of work have been proposed in the research community [14,25], as well as in the commercial domain 1 . These techniques, although attempted to be user-friendly and powerful, still ask for "literate users" who are familiar with finite state machines.…”