THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

Abstract

Understanding object representations requires a broad, comprehensive sampling of the objects in our visual world with dense measurements of brain activity and behavior. Here we present THINGS-data, a multimodal collection of large-scale neuroimaging and behavioral datasets in humans, comprising densely-sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1,854 object concepts. THINGS-data is unique in its breadth of richly-annotated objects, allowing for testing countless hypotheses at scale while assessing the reproducibility of previous findings. Beyond the unique insights promised by each individual dataset, the multimodality of THINGS-data allows combining datasets for a much broader view into object processing than previously possible. Our analyses demonstrate the high quality of the datasets and provide five examples of hypothesis-driven and data-driven applications. THINGS-data constitutes the core public release of the THINGS initiative (https://things-initiative.org) for bridging the gap between disciplines and the advancement of cognitive neuroscience.

Visual object topographic motifs emerge from self-organization of a unified representational space

“…Recent open, large-scale condition-rich fMRI datasets are now available (e.g. NSD dataset, Allen et al, 2022; THINGS dataset, Hebart et al, 2019, 2022) which can enable the development of cortical topographic metrics beyond these macro- and meso-scale signatures probed for here. Thus, going forward, there is clear work to do towards mapping these computational models more directly to the cortex (c.f.…”

Cortical topographic motifs emerge in a self-organized map of object space

“…Finally, complementing these computational approaches, there is a clear need to develop quantitative metrics for comparing topographic activation similarity, which takes into account distance on a cortical sheet (e.g., Wasserstein distance). Recent open, large-scale condition-rich fMRI datasets are now available [e.g., NSD dataset (93) and THINGS dataset (94)], which can enable the development of cortical topographic metrics beyond these macro-and mesoscale signatures probed for here. Thus, going forward, there is clear work to do toward mapping these computational models more directly to the cortex and assessing how they succeed and fail at capturing the systematic response structure to thousands of natural images across the cortical surface.…”

Is my "red" your "red"?: Unsupervised alignment of qualia structures via optimal transport

“…While this study focused on comparing the similarity structures of colors as an initial tractable attempt, future research could explore other sensory modalities across a broader range of tasks (e.g., visual object similarity judgment tasks [20,21]). This could provide a more comprehensive understanding of the extent to which large language models accurately capture the similarity structures inherent in human perception.…”