Incentive Design for Temporal Logic Objectives

Savas, Yagiz; Gupta, Vijay; Ornik, Melkior; Ratliff, Lillian J.; Topcu, Ufuk

doi:10.1109/cdc40024.2019.9029287

Cited by 4 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Policy synthesis: The problem in ( 24) is a total undiscounted cost minimization problem subject to a constraint on the total undiscounted reward given in (20). We recently established the existence of optimal stationary deterministic policies for such problems in [30], where we also present an efficient method to synthesize optimal policies. For completeness, we provide the developed method below.…”

Section: An Approximation Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints

Savas¹,

Verginis²,

Hibbard³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study the synthesis of a policy in a Markov decision process (MDP) following which an agent reaches a target state in the MDP while minimizing its total discounted cost. The problem combines a reachability criterion with a discounted cost criterion and naturally expresses the completion of a task with probabilistic guarantees and optimal transient performance. We first establish that an optimal policy for the considered formulation may not exist but that there always exists a near-optimal stationary policy. We additionally provide a necessary and sufficient condition for the existence of an optimal policy. We then restrict our attention to stationary deterministic policies and show that the decision problem associated with the synthesis of an optimal stationary deterministic policy is NP-complete. Finally, we provide an exact algorithm based on mixed-integer linear programming and propose an efficient approximation algorithm based on linear programming for the synthesis of an optimal stationary deterministic policy.

show abstract

Section: An Approximation Algorithmmentioning

confidence: 99%

“…Proposition 4: [30] A policy π ⋆ D,app ∈Π SD (M) generated by the rule in (27) satisfies the condition in (24).…”

Section: An Approximation Algorithmmentioning

confidence: 99%

On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints

Savas¹,

Verginis²,

Hibbard³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…A proof of Proposition 1 can be found in [16]. Intuitively, the LP in (21a)-(21c) computes the minimum expected time to reach the set B with probability R max (M, B) and cost υ .…”

Section: A Behavior Modification Of a Dominant Type: Formulationmentioning

confidence: 99%

“…Consider an agent with a finite decision horizon N ∈N whose objective is to maximize the expected total reward it collects at the end of every N stages [16]. Such an agent's optimal policy is a sequence (D 1 , D 2 , .…”

Section: Appendix Amentioning

confidence: 99%

“…The second algorithm, on the other hand, computes a locally optimal solution by resorting to a variation [14] of the so-called convex-concave procedure [15] to solve a nonlinear optimization problem with bilinear constraints. Related Work: Some of the results presented in Section VII of this paper has previously appeared in [16] where we study a sequential incentive design problem in which the agent has a known intrinsic motivation. In this paper, we present an extensive study of the sequential incentive design problem for agents with unknown intrinsic motivation, which is a significantly different problem.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Complexity of Sequential Incentive Design

Savas¹,

Gupta²,

Topcu³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In many scenarios, a principal dynamically interacts with an agent and offers a sequence of incentives to align the agent's behavior with a desired objective. This paper focuses on the problem of synthesizing an incentive sequence that, once offered, induces the desired agent behavior even when the agent's intrinsic motivation is unknown to the principal. We model the agent's behavior as a Markov decision process, express its intrinsic motivation as a reward function, which belongs to a finite set of possible reward functions, and consider the incentives as additional rewards offered to the agent. We first show that the behavior modification problem (BMP), i.e., the problem of synthesizing an incentive sequence that induces a desired agent behavior at minimum total cost to the principal, is PSPACE-hard. Moreover, we show that by imposing certain restrictions on the incentive sequences available to the principal, one can obtain two NP-complete variants of the BMP. We also provide a sufficient condition on the set of possible reward functions under which the BMP can be solved via linear programming. Finally, we propose two algorithms to compute globally and locally optimal solutions to the NP-complete variants of the BMP.

show abstract

Improved Rates for Derivative Free Gradient Play in Strongly Monotone Games

Fazel

Ratliff

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

Incentive Design for Temporal Logic Objectives

Cited by 4 publications

References 12 publications

On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints

On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints

On the Complexity of Sequential Incentive Design

Improved Rates for Derivative Free Gradient Play in Strongly Monotone Games

Contact Info

Product

Resources

About