Team Performance with Test Scores

Kleinberg, Jon; Raghu, Maithra

doi:10.1145/2764468.2764496

Cited by 16 publications

(31 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approximation factor decreases with the value of parameter r from value 1 for r = 1 to value 1/k as r goes to infinity. The limit value 1/k conforms to the approximation factor obtained for the best-shot function in Kleinberg and Raghu (2018).…”

Section: Mean and Quantile Test Scoressupporting

confidence: 80%

“…These algorithms are particularly appealing due to their simplicity and natural interpretation as their decisions are contingent only on individual item scores that are computed based on the distribution that captures the uncertainty in the respective item's performance. Although test score based methods have been studied in previous literature (Kleinberg and Raghu 2018), our work presents a new systematic framework for solving a broad class of stochastic combinatorial optimization problems by approximating complex set functions using simpler test score based sketch functions. By leveraging this framework, we show that it is possible to obtain good approximations under a natural extended diminishing returns condition, namely: (a) a constant factor approximation for the problem of maximizing a stochastic submodular function subject to a cardinality constraint, and (b) a logarithmic-approximation guarantee for the more general stochastic submodular welfare maximization problem.…”

Section: Discussionmentioning

confidence: 99%

“…Formally, a test score algorithm is a procedure that computes the test scores for each item in N according to some mapping h and uses only these scores to determine a feasible solution S for (1), e.g., by selecting the k items with the highest scores. Test score algorithms were first introduced by Kleinberg and Raghu (2018), who developed algorithms for a team formation problem for a specific function f . In this work, we propose a novel test score mechanism and utilize it to retrieve improved guarantees for a large class of naturally occuring functions.…”

Section: Can Algorithms That Assign Items To Groups Based On Individual Item Scores Achieve Nearoptimal Group Performance and If So Undermentioning

confidence: 99%

See 2 more Smart Citations

A Test Score-Based Approach to Stochastic Submodular Optimization

2021

View full text Add to dashboard Cite

We study the canonical problem of maximizing a stochastic submodular function subject to a cardinality constraint, where the goal is to select a subset from a ground set of items with uncertain individual performances to maximize their expected group value. Although near-optimal algorithms have been proposed for this problem, practical concerns regarding scalability, compatibility with distributed implementation, and expensive oracle queries persist in large-scale applications. Motivated by online platforms that rely on individual item scores for content recommendation and team selection, we study a special class of algorithms that select items based solely on individual performance measures known as test scores. The central contribution of this work is a novel and systematic framework for designing test score–based algorithms for a broad class of naturally occurring utility functions. We introduce a new scoring mechanism that we refer to as replication test scores and prove that as long as the objective function satisfies a diminishing-returns condition, one can leverage these scores to compute solutions that are within a constant factor of the optimum. We then extend these scoring mechanisms to the more general stochastic submodular welfare-maximization problem, where the goal is to partition items into groups to maximize the sum of the expected group values. For this more difficult problem, we show that replication test scores can be used to develop an algorithm that approximates the optimal solution up to a logarithmic factor. The techniques presented in this work bridge the gap between the rigorous theoretical work on submodular optimization and simple, scalable heuristics that are useful in certain domains. In particular, our results establish that in many applications involving the selection and assignment of items, one can design algorithms that are intuitive and practically relevant with only a small loss in performance compared with the state-of-the-art approaches. This paper was accepted by Chung Piaw Teo, optimization.

show abstract

Section: Mean and Quantile Test Scoressupporting

confidence: 80%

Section: Discussionmentioning

confidence: 99%

Section: Can Algorithms That Assign Items To Groups Based On Individual Item Scores Achieve Nearoptimal Group Performance and If So Undermentioning

confidence: 99%

See 1 more Smart Citation

A Test Score-Based Approach to Stochastic Submodular Optimization

2021

View full text Add to dashboard Cite

show abstract

“…Finally, the rise of collaborative platforms and MOOCs has led to a recent upsurge of interest in the use of testing for selecting teams. In this context, Kleinberg and Raghu (2018) look at the question of how test scores of multiple agents can be used to form teams whose output depends on a complex functiof of agents' joint utility profiles. On the other hand, Johari et al (2018) consider in some sense a dual question wherein a principal observes the scores of different teams, with each score being a complex function of the utility profile of the agents in a team, and must use this to try and rank the agents.…”

Section: Algorithmic Considerations For Ranking From Limited Datamentioning

confidence: 99%

Threshold Tests as Quality Signals: Optimal Strategies, Equilibria, and Price of Anarchy

Banerjee¹,

Kempe²,

Kleinberg³

2021

Preprint

View full text Add to dashboard Cite

We study a signaling game between two firms competing to have their product chosen by a principal. The products have (real-valued) qualities, which are drawn i.i.d. from a common prior. The principal aims to choose the better of the two products, but the quality of a product can only be estimated via a coarse-grained threshold test: given a threshold θ, the principal learns whether a product's quality exceeds θ or fails to do so.We study this selection problem under two types of interactions. In the first, the principal does the testing herself, and can choose tests optimally from a class of allowable tests. We show that the optimum strategy for the principal is to administer different tests to the two products: one which is passed with probability 1 3 and the other with probability 2 3 . If, however, the principal is required to choose the tests in a symmetric manner (i.e., via an i.i.d. distribution), then the optimal strategy is to choose tests whose probability of passing is drawn uniformly from [ 1 4 , 3 4 ]. In our second interaction model, test difficulties are selected endogenously by the two firms. This corresponds to a setting in which the firms must commit to their testing (quality control) procedures before knowing the quality of their products. This interaction model naturally gives rise to a signaling game with two senders and one receiver. We characterize the unique Bayes-Nash Equilibrium of this game, which happens to be symmetric. We then calculate its Price of Anarchy in terms of the principal's probability of choosing the worse product. Finally, we show that by restricting both firms' set of available thresholds to choose from, the principal can lower the Price of Anarchy of the resulting equilibrium; however, there is a limit, in that for every (common) restricted set of tests, the equilibrium failure probability is strictly larger than under the optimal i.i.d. distribution.

show abstract

“…As a rule, the best team of problem solvers need not consist of the most able individuals (Hong & Page, ; Page, ; Marcolino, Jiang, & Tambe, ). In fact, for some classes of problems no measure applied to individuals determines optimal team composition (Kleinberg & Raghu, ). The best person to add to a group will be the one most likely to apply a tool that is both novel and effective.…”

Section: Introductionmentioning

confidence: 99%

Optimal team composition for tool‐based problem solving

Bendor

Page

2018

Economics Manag Strategy

View full text Add to dashboard Cite

In this paper, we construct a framework for modeling teams of agents who apply techniques or procedures (tools) to solve problems. In our framework, tools differ in their likelihood of solving the problem at hand; agents, who may be of different types, vary in their skill at using tools. We establish baseline hiring rules when a manager can dictate tool choice and then derive results for strategic tool choice by team members. We highlight three main findings: First, that cognitively diverse teams are more likely to solve problems in both settings. Second, that teams consisting of types that master diverse tools have an indirect strategic advantage because tool diversity facilitates coordination. Third, that strategic tool choice creates counterintuitive optimal hiring practices. For example, optimal teams may exclude the highest ability types and can include dominated types. In addition, optimal groups need not increase setwise. Our framework extends to cover teamwork on decomposable problems, to cases where individuals apply multiple tools, and to teams facing a flow or set of problems.

show abstract

Team Performance with Test Scores

Cited by 16 publications

References 17 publications

A Test Score-Based Approach to Stochastic Submodular Optimization

A Test Score-Based Approach to Stochastic Submodular Optimization

Threshold Tests as Quality Signals: Optimal Strategies, Equilibria, and Price of Anarchy

Optimal team composition for tool‐based problem solving

Contact Info

Product

Resources

About