The multi-armed bandit problem arises in sequentially allocating effcot to one of N prefects and sequentially asngning patients to cme of N treatmoits in dinical trials. Gittins ainl Jones (1974) have shown that oae. optintal policy Ua the JV-pn^ect problem, an A^dimensional discounted Maricov dedskm chain, is detennined by tiK following largest-index nde. There is an index for eadi state of eadi given project that depoids oidy on die data of that prcgect In each period one allocates effect to a prcgect with largest current index. The purpose (A this paper is to give a short pnx>f of this result and a new diaracterization of tbe index of a project in state i, viz., as the nmrimiim expected present value in state i for the restart-in-(problem in which, in eadi sute ai«i poiod, mie either amtinues allocating effc»t to the project or immediately restarts the project in state i. lAoKovei, it is shown that an approximate largest-index rule yields an approximatdy optinal parse transition matrices in large state qjaces than have been sugg^ted luretofore. By using a suitable inq>]ementation of successive a|>proximati<»is, a pcdicy whose expected present value is within 100c % of the maximum possible range ctf values of the indices can be found on-liiK with at most (N + T-l)TM q>erations where M is the number of operations required to calculate one approximation, T is the least integer nuycoizing the ratio In e/ln a and 0 < a < 1 is the discount factor.

No abstract

A complete computational approach for finding optimal (s, S) inventory policies is developed. The method is an efficient and unified approach for all values of the model parameters, including a non-negative set-up cost, a discount factor 0 \leqq \alpha \leqq 1, and a lead time. The method is derived from renewal theory and stationary analysis, generalized to permit the unit interval range of values for \alpha . Careful attention is given to the problem associated with specifying a starting condition (when \alpha

Many problems from inventory, production and capacity planning, and from network design, exhibit scale economies and can be formulated in terms of finding minimum-additive-concave-cost nonnegative network flows. We reduce the problem to an equivalent one in which the arc flow costs are nonnegative and give a dynamic-programming method, called the send-and-split method, to solve it. The main work of the method entails repeatedly solving set-splitting and minimum-cost-chain problems. In uncapacitated networks with n nodes, a arcs, and d + 1 demand nodes, i.e., nodes with nonzero exogenous demand, the algorithm requires up to n2−13d + s2d operations (additions and comparisons) where s = n log2n + 3a is the number of operations required to solve a minimum-cost-chain problem with nonnegative arc costs on the augmented graph formed by appending a node and an arc thereto from each node in the graph. If also the network is k-planar, i.e., the graph is planar with all demand nodes lying on the boundary of k faces, the method requires at most n2−kd3k + sd2k operations. The algorithm can be applied to capacitated networks because they can be reduced to equivalent uncapacitated ones. These results unify, significantly generalize (e.g., to cyclic problems), and sometimes improve upon (e.g., for tandem facilities) known polynomial-time dynamic-programming algorithms for Wagner and Whitin's (Wagner, H. M., Whitin, T. M. 1958. Dynamic version of the economic lot size model. Management Sci. 5 89–96.) dynamic economic-order-quantity problem, Zangwill's (Zangwill, W. I. 1969. A backlogging model and a multi-echelon model of a dynamic economic lot size production system—A network approach. Management Sci. 15 506–527.) generalization to tandem facilities, and Veinott's (Veinott, Jr., A. F. 1969. Minimum concave-cost solution of Leontief substitution models of multi-facility inventory systems. Oper. Res. 17 262–291.) increasing-capacity warehousing problem. The networks for the finite-period versions of these problems are each 1-planar. The method improves upon Zangwill's (Zangwill, W. I. 1968. Minimum concave cost flows in certain networks. Management Sci. 14 429–450.) related O(and) running-time dynamic-programming method for finding minimum-additive-concave-cost non-negative flows in circuitless single-source networks. We also implement the method to solve in polynomial time the (d + 1)-demand-node and k-planar versions of the minimum-cost forest and Steiner problems in graphs. The running time for the (d + 1)-demand-node Steiner problem in graphs is comparable to that of Dreyfus and Wagner's (Dreyfus, S. E., Wagner, R. A. 1971. The Steiner problem in graphs. Networks 1 195–207.) method.

A multiperiod single product nonstationary inventory problem is studied in which the system is reviewed at the beginning of each of a sequence of periods of equal length. The model has the following features. There are several classes of demand for the product in each period. The demands in different periods are independent but not necessarily identically distributed. The cost structure is nonstationary with the ordering cost being proportional to the amount ordered. Conditions are given that ensure that the base stock ordering policy is optimal and that the base stock levels in each period are easy to calculate. The results are based on the work given in the literature and on properties of stochastically ordered distributions that are developed in the paper. The case of a linear holding cost and a linear shortage cost is studied in detail.