Global Pixel Transformers for Virtual Staining of Microscopy Images

Abstract

Visualizing the details of different cellular structures is of great importance to elucidate cellular functions. However, it is challenging to obtain high quality images of different structures directly due to complex cellular environments. Fluorescence staining is a popular technique to label different structures but has several drawbacks. In particular, label staining is time consuming and may affect cell morphology, and simultaneous labels are inherently limited. This raises the need of building computational models to learn relationships between unlabeled microscopy images and labeled fluorescence images, and to infer fluorescence labels of other microscopy images excluding the physical staining process. We propose to develop a novel deep model for virtual staining of unlabeled microscopy images. We first propose a novel network layer, known as the global pixel transformer layer, that fuses global information from inputs effectively. The proposed global pixel transformer layer can generate outputs with arbitrary dimensions, and can be employed for all the regular, down-sampling, and up-sampling operators. We then incorporate our proposed global pixel transformer layers and dense blocks to build an U-Net like network. We believe such a design can promote feature reusing between layers. In addition, we propose a multi-scale input strategy to encourage networks to capture features at different scales. We conduct evaluations across various fluorescence image prediction tasks to demonstrate the effectiveness of our approach. Both quantitative and qualitative results show that our method outperforms the state-of-the-art approach significantly. It is also shown that our proposed global pixel transformer layer is useful to improve the fluorescence image prediction results.

CleftNet: Augmented Deep Learning for Synaptic Cleft Detection from Brain Electron Microscopy

“…The self-attention mechanism [24], [33] has achieved great success in various domains, including natural language processing [34], [35] and computer vision [21], [22], [36]. Based upon the self-attention mechanism, we propose a novel method, known as the feature augmentor (FA), to improve cleft feature representations.…”

“…Next, we consider the query tensor Q in our proposed FA. Existing studies [21], [22], [36] use the same strategies to generate Q as K and V, making it to be input-dependent. In this work, we instead formulate Q ∈ R dq×hq×wq×cq as a learnable tensor containing free parameters, where d q , h q , w q , and c q denote the depth, height, width, and the number of feature maps, respectively.…”

“…Such patterns are important for learning structural topology and improving cleft feature representations. Existing studies [21], [22], [36] obtain the query tensor by imposing a linear transformation on the input tensor M. However, they suffer from two limitations when learning cleft features. First, the input-dependent query fails to capture the common patterns shared by all clefts.…”

“…Deep architectures have been intensively studied for dense prediction of biological images [15], [16], [21], among which networks based on the U-Net architecture are shown to achieve superior performance [16], [20], [21]. Essentially, the U-Net consists of an encoder and a decoder, which are connected by the bottom block and several skip connections.…”

A Light-Weight Practical Framework for Feces Detection and Trait Recognition

“…Oktay et al [ 22 ] incorporated prior anatomical knowledge into CNNs through a novel regularization model. Since fusion can improve the performance in many ways [ 23 , 24 , 25 ], Liu et al [ 26 ] proposed a novel network layer that effectively fuses the global information from the input, and a novel multi-scale input strategy that acquires multi-scale features. Li et al [ 27 ] proposed a novel 3D self-attention CNN for the low-dose CT denoising problem; the structure acquired more spatial information.…”