Summary of Neuropictor: Refining Fmri-to-image Reconstruction Via Multi-individual Pretraining and Multi-level Modulation, by Jingyang Huo et al.
NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation
by Jingyang Huo, Yikai Wang, Xuelin Qian, Yun Wang, Chong Li, Jianfeng Feng, Yanwei Fu
First submitted to arxiv on: 27 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach, NeuroPictor, is a novel method for generating images from functional magnetic resonance imaging (fMRI) signals. Unlike existing methods that focus on associating fMRI signals with specific conditions of pre-trained diffusion models, NeuroPictor directly modulates the generation process using fMRI signals. This approach consists of three steps: calibrated-encoding, multi-subject pre-training, and single-subject refining. The model extracts high-level semantic features from fMRI signals that characterize visual stimuli and fine-tunes the diffusion model with a low-level manipulation network to provide precise structural instructions. Training on about 67,000 fMRI-image pairs from various individuals enables superior decoding capacity, particularly in within-subject settings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary NeuroPictor is a new way to create images using brain scans. Instead of matching brain signals with pre-trained models, NeuroPictor directly changes the image-making process. This works by breaking down the process into three steps: getting ready for individual brains, learning from many people’s brain scans, and fine-tuning for one person. The model finds important details in brain scans that describe what we see and adjusts the image-making process to add more detail. By using lots of brain scan-image pairs, NeuroPictor can create great images. |
Keywords
* Artificial intelligence * Diffusion * Diffusion model * Fine tuning
