Agent-mediated application emergence through reinforcement learning from user feedback

Abstract

Cyber-physical and ambient systems surround the human user with applications that should be tailored as possible to her/his preferences and the current situation. We propose to build them automatically and on the fly by composition of software components present at the time in the environment, but without prior expression of the user's needs or process specification or composition model. In order to produce knowledge useful for automatic composition in the absence of an initial guideline, we have developed a generic solution based on lifelong online reinforcement learning. It is decentralized within a multi-agent system where agents learn incrementally from user feedback to satisfy her/him. Different use cases have been experimented in which applications, adapted to the user and the situation, are composed and emerge automatically and continuously.

Model-Driven Engineering for End-Users in the Loop in Smart Ambient Systems

“…This way, the engine assures proactivity and runtime adaptation in a context of openness, dynamics, and unpredictability. How OCE works is detailed in [Younes et al, 2020]; it is out of scope of this article.…”

“…More precisely, it lists the bindings that have been modified, deleted, and added, and the components that have been removed or added to the assembly. This list is then used to give to OCE a positive (respectively negative) reinforcement signal for the bindings that have been accepted (respectively rejected) by the user [Younes et al, 2020]. Thanks to this signal, OCE will propose more pertinent applications in the future.…”

Quality-Based Reinforcement Learning in Intelligent Opportunistic Software Composition

“…Agents interact in order to locally decide on a correct and pertinent connection to realise. They use a four-step communication protocol: Advertise, Reply, Select and Agree [2]. An agent's decision is based on its local view of the ambient environment and on estimated values about neighbouring agents that have been computed by reinforcement from previous user feedback.…”

“…In this way, applications emerge from the environment, taking advantage of opportunities as they arise. However, the user is "in the loop" and keeps control on the environment: she/he finally decides on the relevance of the emergent application before it is deployed, and OCE learns from this user feedback to build future applications [2].…”

“…This work is part of the AILP (Assistance InteLligente et proactive en environnement Professionnel) project, which is supported by the French region Occitanie and the operational program FEDER-FSE Midi-Pyrénées et Garonne. A weakness of the current solution is OCE random decision in the presence of new components, i.e., components about which OCE has no knowledge: if such a component is a candidate to participate in an assembly, it may be selected by OCE with an arbitrarily set probability called novelty sensitivity coefficient [2]. This paper proposes an improvement to handle the appearance of new components more efficiently by considering component properties and exploiting knowledge about "similar" components: we enrich the OCE learning mechanism to generalise and transfer knowledge gained from past experiences with some components to components that have not yet been encountered.…”