Decoding Visual Information from High Density Diffuse Optical Tomography Neuroimaging Data

Tripathy, Kalyan; Fishell, Andrew K.; Markow, Zachary E.; Burns‐Yocum, Tracy M.; Newbold, Dillan J.; Tripathy, Pooja; Schlaggar, Bradley L.; Culver, Joseph P.

doi:10.1364/brain.2018.btu4c.3

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These include visual recall [11], visual imagery in wakefulness [11–15] and sleep [16, 17]. Last, advancements in fMRI-based decoding may be useful for other neuroimaging modalities that provide opportunities for effective brain-machine interfaces [18–20] or for diagnosing disorders of consciousness [21, 22].…”

Section: Introductionmentioning

confidence: 99%

“…These include visual recall [11], visual imagery in wakefulness [11][12][13][14][15] and sleep [16,17]. Last, advancements in fMRI-based decoding may be useful for other neuroimaging modalities that provide opportunities for effective brain-machine interfaces [18][19][20] or for diagnosing disorders of consciousness [21,22]. In the image reconstruction task, one attempts to decode natural images which were observed by a human subject from the induced brain activity captured by functional magnetic resonance imaging (fMRI).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity

Gaziv

Beliy

Granot

et al. 2020

Preprint

View full text Add to dashboard Cite

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs (image, fMRI) that span the huge space of natural images is prohibitive. We present a novel self-supervised approach for fMRI-to-image reconstruction and classification that goes well beyond the scarce paired data. By imposing cycle consistency, we train our image reconstruction deep neural network on many “unpaired” data: a plethora of natural images without fMRI recordings (from many novel categories), and fMRI recordings without images. Combining high-level perceptual objectives with self-supervision on unpaired data results in a leap improvement over top existing methods, achieving: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing); (ii) Large-scale semantic classification (1000 diverse classes) of categories that are never-before-seen during network training. Such large-scale (1000-way) semantic classification capabilities from fMRI recordings have never been demonstrated before. Finally, we provide evidence for the biological plausibility of our learned model. 1

show abstract