Learning robust driving policies without online exploration

Affordance as general value function: a computational model

“…In practice, multiple such concerns may simultaneously matter about an option and multiple options may be simultaneously evaluated under the same concern. These can be easily accommodated by introducing the corresponding cumulants and options and, where appropriate, allow the various V s and Q s to share underlying computational implementation (Graves et al, 2020b; Sherstan & Pilarski, 2014; Ugur et al, 2007).…”

“…We see evidence of this in learning affordances that relate to success/failure of bin picking (Zeng et al, 2018), traversability (Ugur et al, 2007), and success/failure of mobile manipulation (Wu et al, 2020) where thousands of predictions structured spatially in a grid pattern are learned. Second, when combined with DL, multiple predictions can be made with the same neural network by sharing earlier layers across different prediction heads, allowing for potentially sublinear scaling in the number of predictions (Graves et al, 2020b; Sherstan & Pilarski, 2014; Zeng et al, 2018). Third, as noted above, for a single prediction, the direct perception architecture operates in constant time regardless of how distant a future the prediction needs to be concerned about.…”

“…However, in spite of these nice properties of GVFs for making predictions, in spite of the well-established mechanisms for learning them, and in spite of some specific examples of how GVF predictions may be used to control real-world robots (Edwards et al, 2016; Graves et al, 2019, 2020b; Günther et al, 2018), there is as yet no general computational framework for using such predictions, such as using them in downstream learning or optimization of policies.…”

“…Since any meaningful scalar-valued signal could be used as cumulant, all sorts of valence predictions can be made. In practice, the cumulant can be hand-engineered features such as safety (Graves et al, 2019), comfort, centeredness (Graves et al, 2020b), and duration (Sutton et al, 2011). The cumulant can also represent outcomes such as traversability (Ugur et al, 2007), success or failure (Zeng et al, 2018), or human-provided quality labels (Günther et al, 2016).…”

“…While affordance perception as GVF prediction may take geometric measurements and object detection results as input (Graves et al, 2019), dependency on these are strictly optional. Instead, predictions can be made directly from raw sensory inputs (Graves et al, 2020b; Günther et al, 2016; Zeng et al, 2018). In contrast, if object detection is a necessary dependency of affordance prediction (Hassanin et al, 2018), errors may be introduced through the data labeling needed for setting up the required supervised learning.…”