Scalable graph analysis tools for the connectomics community

Abstract

Neuroscientists now have the opportunity to analyze synaptic resolution connectomes that are larger than the memory on single consumer workstations. As dataset size and tissue diversity have grown, there is increasing interest in conducting comparative connectomics research, including rapidly querying and searching for recurring patterns of connectivity across brain regions and species. There is also a demand for algorithm reuse --- applying methods developed for one dataset to another volume. A key technological hurdle is enabling researchers to efficiently and effectively query these diverse datasets, especially as the raw image volumes grow beyond terabyte sizes. Existing community tools can perform such queries and analysis on smaller scale datasets, which can fit locally in memory, but the path to scaling remains unclear. Existing solutions such as neuPrint or FlyBrainLab enable these queries for specific datasets, but there remains a need to generalize algorithms and standards across datasets. To overcome this challenge, we present a software framework for comparative connectomics and graph discovery to make connectomes easy to analyze, even when larger-than-RAM, and even when stored in disparate datastores. This software suite includes visualization tools, a web portal, a connectivity and annotation query engine, and the ability to interface with a variety of data sources and community tools from the neuroscience community. These tools include MossDB (an immutable datastore for metadata and rich annotations); Grand (for prototyping larger-than-RAM graphs); GrandIso-Cloud (for querying existing graphs that exceed the capabilities of a single workstation); and Motif Studio (for enabling the public to query across connectomes). These tools interface with existing frameworks such as neuPrint, graph databases such as Neo4j, and standard data analysis tools such as Pandas or NetworkX. Together, these tools enable tool and algorithm reuse, standardization, and neuroscience discovery.

Data-driven motif discovery in biological neural networks

“…The ellipsoid body graph took a total of 1,125 CPU-days to run on the compute cluster (around 36,000 searches at µ = 45 minutes each). This runtime is dominated by the edge count and local topology of the host graph, rather than its vertex count, and so to our knowledge there is no straightforward way to estimate runtime prior to execution at this time [15]. To further reduce the wallclock execution time, we also provide a cloud-based software implementation that uses serverless compute (i.e., Amazon Web Services Lambda and DynamoDB) [15].…”

“…This runtime is dominated by the edge count and local topology of the host graph, rather than its vertex count, and so to our knowledge there is no straightforward way to estimate runtime prior to execution at this time [15]. To further reduce the wallclock execution time, we also provide a cloud-based software implementation that uses serverless compute (i.e., Amazon Web Services Lambda and DynamoDB) [15]. All results from both of these connectome graph searches, as well as our randomized network results, will be made publicly available.…”

“…Furthermore, connectomes are often one-of-a-kind, meaning that traditional statistical models are challenging or impossible to implement. Finally, exhaustive subgraph matching or counting can be computationally intractable for even relatively small connectomes [15, 16].…”

“…And, of course, there are a number of additional heuristics that can be used to improve computational feasibility. Some of the popular approaches include MAVisto, MFinder, FANMOD, and MODA [21, 22, 23, 24, 25], as well as our more recent contributions, DotMotif and GrandIso [10] and their cloud-scale equivalents [15].…”