A key element in the solution of reinforcement learning problems is the value function. The purpose of this function is to measure the long-term utility or value of any given state. The function is important because an agent can use this measure to decide what to do next. A common problem in reinforcement learning when applied to systems having continuous states and action spaces is that the value function must operate with a domain consisting of real-valued variables, which means that it should be able to represent the value of infinitely many state and action pairs. For this reason, function approximators are used to represent the value function when a close-form solution of the optimal policy is not available. In this article, we extend a previously proposed reinforcement learning algorithm so that it can be used with function approximators that generalize the value of individual experiences across both state and action spaces. In particular, we discuss the benefits of using sparse coarse-coded function approximators to represent value functions and describe in detail three implementations: cerebellar model articulation controllers, instancebased, and case-based. Additionally, we discuss how function approximators having different degrees of resolution in different regions of the state and action spaces may influence the performance and learning efficiency of the agent. We propose a simple and modular technique that can be used to implement function approximators with nonuniform degrees of resolution so that the value function can be represented with higher accuracy in important regions of the state and action spaces. We performed extensive experiments in the double-integrator and pendulum swing-up systems to demonstrate the proposed ideas.'

This paper compares direct reinforcement learning (no explicit model) and model-based reinforcement learning on a simple task: pendulum swing up. We find that in this task model-based approaches support reinforcement learning from smaller amounts of training data and efficient handling of changing goals.

No abstract

This paper introduces a new set of software tools that integrate near-real-time visualization with a publish-and-subscribe mechanism to achieve remote monitoring and control of dynamic objects in an underwater scene. The approach proposed in this paper involves the integration of existing technologies to produce a powerful and flexible solution easily adapted to the extensive and diverse set of situations encountered in underwater construction, underwater surveying, and maritime navigation.Users can define elaborate virtual scenes that can accurately represent all the relevant elements associated with an underwater construction job, including complex structures and dynamic objects, such as ROVs, vessels, etc. Real-time data from instruments and positioning sensors is made available by using a publishing mechanism and a remote data server. Users with Internet or intranet access can subscribe to any real-time data field being published and receive updates each time the information changes, allowing them to monitor and log the events of the underwater job at the same time they are taking place. The use of advanced cueing techniques and multi-resolution rendering make it possible to achieve interactive frame rates without sacrificing accuracy and realism. The concept of 3D modules will also be introduced. These powerful modules can be linked to live data and attached to any element in the virtual environment including dynamic ones. This flexibility allows the users to monitor the data, not only as it changes, but also within the spatial context that makes the most sense. This paper focuses on the visualization components of the proposed solution, while providing a general description of the other two components (data acquisition, data distribution), and describes the Ehime Maru recovery mission as an example of an underwater job that benefited by using this technology.

No abstract