Machine Teaching in Hierarchical Genetic Reinforcement Learning: Curriculum Design of Reward Functions for Swarm Shepherding

Abstract

The design of reward functions in reinforcement learning is a human skill that comes with experience. Unfortunately, there is not any methodology in the literature that could guide a human to design the reward function or to allow a human to transfer the skills developed in designing reward functions to another human and in a systematic manner. In this paper, we use Systematic Instructional Design, an approach in human education, to engineer a machine education methodology to design reward functions for reinforcement learning. We demonstrate the methodology in designing a hierarchical genetic reinforcement learner that adopts a neural network representation to evolve a swarm controller for an agent shepherding a boids-based swarm. The results reveal that the methodology is able to guide the design of hierarchical reinforcement learners, with each model in the hierarchy learning incrementally through a multi-part reward function. The hierarchy acts as a decision fusion function that combines the individual behaviours and skills learnt by each instruction to create a smart shepherd to control the swarm.

A model of symbiomemesis: machine education and communication as pillars for human-autonomy symbiosis

“…ML is 'the ability of a machine to improve performance based on experience' [18]. Experience is presented to the machine in many forms including: raw or processed data, demonstrations from human experts [19] or from other machines [20], rewards [21], and/or punishments [22]. Recently, machine teaching (MT) has emerged as a dedicated field focusing on the design of experience for the ML agent.…”

“…While the delivery of the above curriculum could be carried out in a variety of ways, so far the ME literature has focused on systematic instructional design (SID). In particular, the Dick & Carey [36] model has been used by Clayton and Abbass to map the 10 concrete steps to design ‘a systematic approach for the design, development, implementation and evaluation of instruction’ [21] for an AIAS. SID prescribes learning as a cognitive process guided and supported by purposeful instruction within an interdependent instructional system.…”

Machine Education: Designing semantically ordered and ontologically guided modular neural networks

“…However, similar to Elman, the work is mostly driven by experts in machine learning with intimate knowledge of algorithms, simple tasks, and, in most cases, exiguous grounds in teaching methodologies. Moreover, in the field of evolutionary-learning, other than limited recent attempts [5], the research area is in its infancy. To the best of our knowledge, attempts in the evolutionary neural network literature are rare.…”

“…Clayton and Abbass [5] designed a curriculum suitable for reinforcement learning used within an autonomous agent in shepherding tasks. They extended Dick and Carey's Model for Systematic Instructional Design (SID) [8], [9], which offers the following ten steps: identification of an instructional goal, conducting instructional analysis, identification of entry behaviours and characteristics, write-up of performance objectives, development of criterion-referenced assessments, development of instructional strategies, development of and/or select instruction, development of and carrying out of formative evaluations, revising the instruction, and conducting summative evaluation.…”

“…A common aspect of this domain's literature is that it is driven by machine learners alone with a focus on data engineering and model engineering to improve machine learning. As such, except for a couple of recent attempts [4], [5], [6], the literature on machine teaching and education has not taken inspiration from the more mature literature on human teaching and education which houses a wealth of pedagogical frameworks, philosophies and theories that could benefit the machine education field.…”