Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory

Kushner, Harold J.; Kleinman, Alan

doi:10.1109/tac.1968.1098939

Cited by 30 publications

(13 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We discretize (22) to get the difference equation (23) where (24) With the boundary condition and an initial approximate solution, we can determine the variables and (the control variables) by the rule (24), and update the numerical solution. The iterations converge to the exact solution to the difference equation (23), as can be proved by the method in [26]. We remark that there are general results concerning the convergence of this type of difference scheme to the solution of the original partial differential equation.…”

Section: A Numerical Schemementioning

confidence: 57%

“…We write (26) where and is a weight matrix. We penalize abrupt change of powers via since practical power control is exercised in a cautious manner and there exist basic limits for power adjustment rate.…”

Section: A Discounted Cost Function and The Hjb Equationmentioning

confidence: 99%

“…We penalize abrupt change of powers via since practical power control is exercised in a cautious manner and there exist basic limits for power adjustment rate. After subtracting the constant component from the integrand in (26), we get the cost function to be employed (27) where , are positive definite matrix, and vector, respectively, which are determined from (26). Write .…”

Section: A Discounted Cost Function and The Hjb Equationmentioning

confidence: 99%

See 2 more Smart Citations

Uplink Power Adjustment in Wireless Communication Systems: A Stochastic Control Analysis

Huang

Caines

Malhamé

2004

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

This paper considers mobile to base station power control for lognormal fading channels in wireless communication systems within a centralized information stochastic optimal control framework. Under a bounded power rate of change constraint, the stochastic control problem and its associated Hamilton-Jacobi-Bellman (HJB) equation are analyzed by the viscosity solution method; then the degenerate HJB equation is perturbed to admit a classical solution and a suboptimal control law is designed based on the perturbed HJB equation. When a quadratic type cost is used without a bound constraint on the control, the value function is a classical solution to the degenerate HJB equation and the feedback control is affine in the system power. In addition, in this case we develop approximate, but highly scalable, solutions to the HJB equation in terms of a local polynomial expansion of the exact solution. When the channel parameters are not known a priori, one can obtain on-line estimates of the parameters and get adaptive versions of the control laws. In numerical experiments with both of the above cost functions, the following phenomenon is observed: whenever the users have different initial conditions, there is an initial convergence of the power levels to a common level and then subsequent approximately equal behavior which converges toward a stochastically varying optimum.Index Terms-Dynamic programming, Hamilton-JacobiBellman (HJB) equations, lognormal fading channels, power control, quality of service.

show abstract

Section: A Numerical Schemementioning

confidence: 57%

Section: A Discounted Cost Function and The Hjb Equationmentioning

confidence: 99%

Section: A Discounted Cost Function and The Hjb Equationmentioning

confidence: 99%

See 1 more Smart Citation

Uplink Power Adjustment in Wireless Communication Systems: A Stochastic Control Analysis

Huang

Caines

Malhamé

2004

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

show abstract

“…See [ 1] for a discussion of a better iterative method. formulation and its generalizations, its useful to re • .,-rite (Pl) in an equivalent form.…”

Section: 1mentioning

confidence: 99%

“…By the method in [ 1), an approximating Markov chain (X n ) ( whose state space is the collection of nodes in Fig. 1 Px x+e, h ( u) = (a 2 +hu)12(a2 +hl x 2 1 )…”

Section: ' 'mentioning

confidence: 99%

Mathematical programming and the control of Markov chains†

Kushner

Kleinman

1971

International Journal of Control

View full text Add to dashboard Cite

Linear programming versions of some control problems on Ma.rkov chains are derived, and are studied under conditions which occur iii typical rroblems which arise by discretizing continuous time and state_ systems, or in discrete state -ystems. Control interpretations of the dual variables and simplex multipliers are given. The formulation allows the treatment of 'state space' like constraints which cannot be handled conveniently with dynamic programming. The relation between dyneaiuc programming on Markov chains, and the deterministic discrete maxirlum princip]e in explored, and some insight is c stained into the problem of singular s46l chastic controls (with respect to a stochastic maximum principle). 1 IntroductionThis paper is concerned with several problems occuring in the control of a Markov chain (Xn } on the state space (01p ... ,N) -S , , ,9 with transition probabilities pij (a), where a, a control, takes values in a set Ui.State 0 is a desired target state and Poo (a) = 1; once in state 0, ulways in state 0. The terms u = (ul)...,uN), U t U i , denotes a control vector. I.e., if the control vector u is always used, and X = i, then the value of a in p ii (a) is u(Y.n ) = ui . Let T denote the first time state 0 is attained, k(i,cx) the cost paid when the state is i and control u(X ) = u = u is used, and E the expectation operator Define problem (Pl): Let U i contain a finite Number of points (which, for convenience, we assume are al,...,aq), or let the n +1 dimensional set ( pi 1 (U i )' "'' piN (U i ) p k(i, U i ) ) be a convex polyhedron with extreme points included in ((pil(ar)'"''piN(ar),k(ifar)), r -1,...,q). As-ume (Al): In Section 2, a linear programming formulation of (Pl) will be given. Linear programming (L.P.) versions of many types of dynamic programming problems are well known ( see, e.g.., [3] -[51, [9]). Indeed, a L. P. version of (1 11) was given by Derman [6]. The variables in the L.P. form in [6] do not seem to have a simple physical interpretation. However, the form here seems more natural and has a more natural dual, namely the dynamic programming equations for (P1) N Vi s E pij (ar )Vj + k(i,ar ), all i,r. j =1While experience indicates that the linear programming algorithm (Simplex method) is generally inferior, in computational efficiency, to the available dynamic programming iterative methods (for the type of problems discussed !:ere), it is of interest since it is an alternative formulation which sheds further light on the Markov optimization problem and, in addition, the two important reasons:3 (a) There may be additional constraints on the probabilities P( Xn = i) ( Section 2) . The dynamic progrtuwdng is not d :reetly applicable, and the L.P. formulation yields useful insights into the optimization problem. Indeed, it i:-often desirable or necessary to add such constraints in Markov control problems. See Section 2 for example.(b) The L.P. formulation gives us insight into a form of a stochastic maximum principle (Section 3), and the singularity problem of the stochast...

show abstract

Bibliography

1994

Markov Decision Processes

View full text Add to dashboard Cite

Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory

Cited by 30 publications

References 4 publications

Uplink Power Adjustment in Wireless Communication Systems: A Stochastic Control Analysis

Uplink Power Adjustment in Wireless Communication Systems: A Stochastic Control Analysis

Mathematical programming and the control of Markov chains†

Bibliography

Contact Info

Product

Resources

About