Paper review · October 21, 2026
Scaling PEFT (parameter-efficient fine-tuning) for brain encoding
Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
By adapting multimodal backbones of up to 104B parameters with LoRA, the authors achieved the best brain-response prediction performance on Algonauts 2025 while training about 5% of the parameters of the previous best model.
Problem
A brain encoding model takes stimuli such as video, sound and text as input and predicts the brain response at that moment. Until now, the dominant approach has been to freeze a pretrained model and train only the output module (readout). Brain data is expensive and scarce, which made it hard to train large models from scratch or to fully fine-tune them. As a result, adapting the backbone itself to brain data has been largely unexplored.
Method
The authors insert LoRA into a pretrained multimodal backbone and train it together with a lightweight readout (BrainLoRA). Taking advantage of the backbone's causal structure, they process a 149-second segment in a single forward pass instead of using sliding windows, obtaining features for all time points at once. A 60-second context is prepended to each segment. The readout is either linear or cross-attention, and after training the linear readout is refit with ridge regression. They also built a multi-LoRA kernel that trains multiple LoRAs together on a shared backbone, and used it to build an ensemble of more than 200 models.
Results
- Feature extraction cost was reduced by about 350x compared with the sliding windows of the previous SoTA (MIRAGE) (from about 700 to 2 GPU-hours on A800).
- Across all nine multimodal backbones ranging from 4.7B to 104B, LoRA raised Pearson r with a linear readout by 0.032–0.066 over a frozen backbone.
- BrainLoRA (cross-attn) on the Qwen3-Omni backbone scored 0.327 ID and 0.228 OOD, outperforming the single MIRAGE model (0.310/0.217), which has 1.1B trainable parameters, with 61.4M parameters.
- An ensemble of 261 models scored 0.343 ID and 0.248 OOD, higher than MIRAGE's 15-model ensemble (0.323/0.227), and the authors reported it was first on the public leaderboard at the time of writing.
- Full fine-tuning at 20.3B (ID 0.326, OOD 0.225) did not surpass LoRA. The authors reported that 97% of the improvement was retained when applied to subjects held out from training, and that applying it to another dataset (NNDb) without additional training yielded a +11.1 percentage-point gain relative to the noise ceiling.
Limitations
The main experiments focus on a single benchmark, Algonauts 2025 (4 subjects, fMRI recorded while watching TV dramas, 1,000 brain regions). The authors stated that differences in mean correlation below 0.00077 cannot be distinguished from seed variation. The correlation between model size and performance was weak (r=0.60–0.71). Because some first-stage modality encoders are bidirectional, the structure is not fully causal; the authors attributed the resulting discrepancy mainly to bf16 MoE numerical error. The provided text is cut off partway, so the later experiments could not be checked.
Why it matters
The paper provides evidence that large multimodal models can be adapted efficiently even with limited brain data. It makes PEFT a default option to consider in neuroscience modeling, in place of the usual 'frozen backbone + readout' practice. The single-pass feature extraction and multi-LoRA training kernel can also be applied to other large-scale adapter experiments.
Key terms
- Brain encoding model
- A model that takes stimuli such as video or sound as input and predicts the brain activity (fMRI, etc.) that occurs at that moment.
- LoRA
- A PEFT technique that adapts a model by keeping the original weights frozen and training only small added low-rank matrices.
- Readout
- An output module that converts the representation extracted by the backbone into actual response values for each brain region. It also aligns the time axes of the stimulus and the fMRI.
- ID / OOD evaluation
- ID is performance on stimuli of the same kind as in training (a different season of the same drama), and OOD is performance on completely different movie stimuli. OOD better reflects generalization ability.
Original
Authors: Zhuofan Josh Ying, Guanglong Sun, Nolan Ho, Kieran Liu, Rui Yang, Andrew Chen, Nikolaus Kriegeskorte, Pony Ma, et al. · Posted 2026-10-19 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.