Personalized web services strive to adapt their services (advertisements, news articles, etc.) to individual users by making use of both content and user information. Despite a few recent advances, this problem remains challenging for at least two reasons. First, web service is featured with dynamically changing pools of content, rendering traditional collaborative filtering methods inapplicable. Second, the scale of most web services of practical interest calls for solutions that are both fast in learning and computation.In this work, we model personalized recommendation of news articles as a contextual bandit problem, a principled approach in which a learning algorithm sequentially selects articles to serve users based on contextual information about the users and articles, while simultaneously adapting its article-selection strategy based on user-click feedback to maximize total user clicks.The contributions of this work are three-fold. First, we propose a new, general contextual bandit algorithm that is computationally efficient and well motivated from learning theory. Second, we argue that any bandit algorithm can be reliably evaluated offline using previously recorded random traffic. Finally, using this offline evaluation method, we successfully applied our new algorithm to a Yahoo! Front Page Today Module dataset containing over 33 million events. Results showed a 12.5% click lift compared to a standard context-free bandit algorithm, and the advantage becomes even greater when data gets more scarce.

We study sequential decision making in environments where rewards are only partially observed, but can be modeled as a function of observed contexts and the chosen action by the decision maker. This setting, known as contextual bandits, encompasses a wide variety of applications such as health care, content recommendation and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strengths and overcome the weaknesses of the two approaches by applying the doubly robust estimation technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust estimation uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice in policy evaluation and optimization.Comment: Published in at http://dx.doi.org/10.1214/14-STS500 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Contextual bandit algorithms have become popular for online recommendation systems such as Digg, Yahoo! Buzz, and news recommendation in general. Offline evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "partial-label" nature. Common practice is to create a simulator which simulates the online environment for the problem at hand and then run an algorithm against this simulator. However, creating simulator itself is often difficult and modeling bias is usually unavoidably introduced. In this paper, we introduce a replay methodology for contextual bandit algorithm evaluation. Different from simulator-based approaches, our method is completely data-driven and very easy to adapt to different applications. More importantly, our method can provide provably unbiased evaluations. Our empirical results on a large-scale news article recommendation dataset collected from Yahoo! Front Page conform well with our theoretical results. Furthermore, comparisons between our offline replay and online bucket evaluation of several contextual bandit algorithms show accuracy and effectiveness of our offline evaluation method.

Extracellular vesicles (EVs) have stimulated considerable scientific and clinical interest, yet protein profiling and sizing of individual EVs remains challenging due to their small particle size, low abundance of proteins, and overall heterogeneity. Building upon a laboratory-built high-sensitivity flow cytometer (HSFCM), we report here a rapid approach for quantitative multiparameter analysis of single EVs down to 40 nm with an analysis rate up to 10 000 particles per minute. Statistically robust particle size distribution was acquired in minutes with a resolution and profile well matched with those of cryo-TEM measurements. Subpopulations of EVs expressing CD9, CD63, and/or CD81 were quantified upon immunofluorescent staining. When HSFCM was used to analyze blood samples, a significantly elevated level of CD147-positive EVs was identified in colorectal cancer patients compared to healthy controls (P < 0.001). HSFCM provides a sensitive and rapid platform for surface protein profiling and sizing of individual EVs, which could greatly aid the understanding of EV-mediated intercellular communication and the development of advanced diagnostic and therapeutic strategies.

In this review, the recent advances in inks, strategies, and the applications of inkjet-printed wearable electronics have been summarized.

This paper proposes KB-InfoBot 1 -a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typically need to interact with an external database to access real-world knowledge. Previous systems achieved this by issuing a symbolic query to the KB to retrieve entries based on their attributes. However, such symbolic operations break the differentiability of the system and prevent endto-end training of neural dialogue agents. In this paper, we address this limitation by replacing symbolic queries with an induced "soft" posterior distribution over the KB that indicates which entities the user is interested in. Integrating the soft retrieval process with a reinforcement learner leads to higher task success rate and reward in both simulations and against real users. We also present a fully neural end-to-end agent, trained entirely from user feedback, and discuss its application towards personalized dialogue agents.

Heat stroke is a life-threatening condition, featuring a high body temperature and malfunction of many organ systems. The relationship between heat shock and lysosomes is poorly understood, mainly because of the lack of a suitable research approach. Herein, by incorporating morpholine into a stable hemicyanine skeleton, we develop a new lysosome-targeting near-infrared ratiometric pH probe. In combination with fluorescence imaging, we show for the first time that the lysosomal pH value increases but never decreases during heat shock, which might result from lysosomal membrane permeabilization. We also demonstrate that this lysosomal pH rise is irreversible in living cells. Moreover, the probe is easy to synthesize, and shows superior overall analytical performance as compared to the existing commercial ones. This enhanced performance may enable it to be widely used in more lysosomal models of living cells and in further revealing the mechanisms underlying heat-related pathology.

No abstract