Abstract-Organizations rely on data analysts to model customer engagement, streamline operations, improve production, inform business decisions, and combat fraud. Though numerous analysis and visualization tools have been built to improve the scale and efficiency at which analysts can work, there has been little research on how analysis takes place within the social and organizational context of companies. To better understand the enterprise analysts' ecosystem, we conducted semi-structured interviews with 35 data analysts from 25 organizations across a variety of sectors, including healthcare, retail, marketing and finance. Based on our interview data, we characterize the process of industrial data analysis and document how organizational features of an enterprise impact it. We describe recurring pain points, outstanding challenges, and barriers to adoption for visual analytic tools. Finally, we discuss design implications and opportunities for visual analysis research.

In spite of advances in technologies for working with data, analysts still spend an inordinate amount of time diagnosing data quality issues and manipulating data into a usable form. This process of ‘data wrangling’ often constitutes the most tedious and time-consuming aspect of analysis. Though data cleaning and integration arelongstanding issues in the database community, relatively little research has explored how interactive visualization can advance the state of the art. In this article, we review the challenges and opportunities associated with addressing data quality issues. We argue that analysts might more effectively wrangle data through new interactive systems that integrate data verification, transformation, and visualization. We identify a number of outstanding research questions, including how appropriate visual encodings can facilitate apprehension of missing data, discrepant values, and uncertainty; how interactive visualizations might facilitate data transform specification; and how recorded provenance and social interaction might enable wider reuse, verification, and modification of data transformations

Though data analysis tools continue to improve, analysts still expend an inordinate amount of time and effort manipulating data and assessing data quality issues. Such "data wrangling" regularly involves reformatting data values or layout, correcting erroneous or missing values, and integrating multiple data sources. These transforms are often difficult to specify and difficult to reuse across analysis tasks, teams, and tools. In response, we introduce Wrangler, an interactive system for creating data transformations. Wrangler combines direct manipulation of visualized data with automatic inference of relevant transforms, enabling analysts to iteratively explore the space of applicable operations and preview their effects. Wrangler leverages semantic data types (e.g., geographic locations, dates, classification codes) to aid validation and type conversion. Interactive histories support review, refinement, and annotation of transformation scripts. User study results show that Wrangler significantly reduces specification time and promotes the use of robust, auditable transforms instead of manual editing.

New user interfaces can transform how we work with big data, and raise exciting research problems that span human-computer interaction, machine learning, and distributed systems.

BackgroundThis study aimed to evaluate the prevalence and predictors of AIDS-related complicated cryptococcal meningitis. The outcome was complicated cryptococcal meningitis: prolonged (≥ 14 days) altered mental status, persistent (≥ 14 days) focal neurologic findings, cerebrospinal fluid (CSF) shunt placement or death. Predictor variable operating characteristics were estimated using receiver operating characteristic curve (ROC) analysis. Multivariate analysis identified independent predictors of the outcome.ResultsFrom 1990-2009, 82 patients with first episode of cryptococcal meningitis were identified. Of these, 14 (17%) met criteria for complicated forms of cryptococcal meningitis (prolonged altered mental status 6, persistent focal neurologic findings 7, CSF surgical shunt placement 8, and death 5). Patients with complicated cryptococcal meningitis had higher frequency of baseline focal neurological findings, head computed tomography (CT) abnormalities, mean CSF opening pressure, and cryptococcal antigen (CRAG) titers in serum and CSF. ROC area of log2 serum and CSF CRAG titers to predict complicated forms of cryptococcal meningitis were comparable, 0.78 (95%CI: 0.66 to 0.90) vs. 0.78 (95% CI: 0.67 to 0.89), respectively (χ2, p = 0.95). The ROC areas to predict the outcomes were similar for CSF pressure and CSF CRAG titers. In a multiple logistic regression model, the following were significant predictors of the outcome: baseline focal neurologic findings, head CT abnormalities and log2 CSF CRAG titer.ConclusionsDuring initial clinical evaluation, a focal neurologic exam, abnormal head CT and large cryptococcal burden measured by CRAG titer are associated with the outcome of complicated cryptococcal meningitis following 2 weeks from antifungal therapy initiation.