What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-synchronous processing paradigm popularized by MapReduce, or even specialized graph-based execution that relies on graph representations of ML programs. The variety of approaches tends to pull systems and algorithms design in different directions, and it remains difficult to find a universal platform applicable to a wide range of ML programs at scale. We propose a general-purpose framework that systematically addresses data-and model-parallel challenges in large-scale ML, by observing that many ML programs are fundamentally optimization-centric and admit error-tolerant, iterative-convergent algorithmic solutions. This presents unique opportunities for an integrative system design, such as bounded-error network synchronization and dynamic scheduling based on ML program structure. We demonstrate the efficacy of these system designs versus well-known implementations of modern ML algorithms, allowing ML programs to run in much less time and at considerably larger model sizes, even on modestly-sized compute clusters.

Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in realtime applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ∼40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and also reveal their redundancy. Our work provides new ideas to efficiently apply deep transformer-based models to downstream tasks. Code is available at https://github.com/castorini/ DeeBERT.

Pooling plays an important role in generating a discriminative video representation. In this paper, we propose a new semantic pooling approach for challenging event analysis tasks (e.g. event detection, recognition, and recounting) in long untrimmed Internet videos, especially when only a few shots/segments are relevant to the event of interest while many other shots are irrelevant or even misleading. The commonly adopted pooling strategies aggregate the shots indifferently in one way or another, resulting in a great loss of information. Instead, in this work we first define a novel notion of semantic saliency that assesses the relevance of each shot with the event of interest. We then prioritize the shots according to their saliency scores since shots that are semantically more salient are expected to contribute more to the final event analysis. Next, we propose a new isotonic regularizer that is able to exploit the constructed semantic ordering information. The resulting nearly-isotonic support vector machine classifier exhibits higher discriminative power in event analysis tasks. Computationally, we develop an efficient implementation using the proximal gradient algorithm, and we prove new and closed-form proximal steps. We conduct extensive experiments on three real-world video datasets and achieve promising improvements.

Imaging the activity of neurons that are widely distributed across brain regions deep in scattering tissue at high speed remains challenging. Here, we introduce an open-source system with Dual Independent Enhanced Scan Engines for Large field-of-view Two-Photon imaging (Diesel2p). Combining optical design, adaptive optics, and temporal multiplexing, the system offers subcellular resolution over a large field-of-view of ~25 mm2, encompassing distances up to 7 mm, with independent scan engines. We demonstrate the flexibility and various use cases of this system for calcium imaging of neurons in the living brain.

The slow speed of BERT has motivated much research on accelerating its inference, and the early exiting idea has been proposed to make trade-offs between model quality and efficiency. This paper aims to address two weaknesses of previous work: (1) existing fine-tuning strategies for early exiting models fail to take full advantage of BERT; (2) methods to make exiting decisions are limited to classification tasks. We propose a more advanced fine-tuning strategy and a learning-toexit module that extends early exiting to tasks other than classification. Experiments demonstrate improved early exiting for BERT, with better trade-offs obtained by the proposed finetuning strategy, successful application to regression tasks, and the possibility to combine it with other acceleration methods. Source code can be found at https://github.com/ castorini/berxit.

Mitral cells (MCs) are a major class of principal neurons in the vertebrate olfactory bulb, conveying odor-evoked activity from the peripheral sensory neurons to olfactory cortex. Previous work has described the development of MC morphology and connectivity during the first few weeks of postnatal development. However, little is known about the postnatal development of MC intrinsic biophysical properties. To understand stimulus encoding in the developing olfactory bulb, we have therefore examined the development of MC intrinsic biophysical properties in acute slices from postnatal day (P)7-P35 mice. Across development, we observed systematic changes in passive membrane properties and action potential waveforms consistent with a developmental increase in sodium and potassium conductances. We further observed developmental decreases in hyperpolarization-evoked membrane potential sag and firing regularity, extending recent links between MC sag heterogeneity and firing patterns. We then applied a novel combination of statistical analyses to examine how the evolution of these intrinsic biophysical properties specifically influenced the representation of fluctuating stimuli by MCs. We found that immature MCs responded to frozen fluctuating stimuli with lower firing rates, lower spike-time reliability, and lower between-cell spike-time correlations than more mature MCs. Analysis of spike-triggered averages revealed that these changes in spike timing were driven by a developmental shift from broad integration of inputs to more selective detection of coincident inputs. Consistent with this shift, generalized linear model fits to MC firing responses demonstrated an enhanced encoding of high-frequency stimulus features by mature MCs.

No abstract

Mice use vision to navigate and avoid predators in natural environments. However, their visual systems are compact compared to other mammals, and it is unclear how well mice can discriminate ethologically relevant scenes. Here, we examined natural scene discrimination in mice using an automated touch-screen system. We estimated the discrimination difficulty using the computational metric structural similarity (SSIM), and constructed psychometric curves. However, the performance of each mouse was better predicted by the mean performance of other mice than SSIM. This high inter-mouse agreement indicates that mice use common and robust strategies to discriminate natural scenes. We tested several other image metrics to find an alternative to SSIM for predicting discrimination performance. We found that a simple, primary visual cortex (V1)-inspired model predicted mouse performance with fidelity approaching the inter-mouse agreement. The model involved convolving the images with Gabor filters, and its performance varied with the orientation of the Gabor filter. This orientation dependence was driven by the stimuli, rather than an innate biological feature. Together, these results indicate that mice are adept at discriminating natural scenes, and their performance is well predicted by simple models of V1 processing.