Paper review · November 5, 2026
Structure, Dimensionality and Distributional Shape: What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides
Structure, Dimensionality, and Distribution Shape - What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which
Using TransferForge, a synthetic data generator that controls structure, dimensionality and distributional shape separately, the authors showed that which data properties matter for transfer is determined not by the modality but by the model's input normalization.
Problem
Time series foundation models are known to pretrain well on synthetic data alone. However, it is not known which properties of a pretraining corpus produce transfer, or whether they are the same across models. The same Gaussian synthetic data helped on one task and actually hurt on another.
Method
The authors describe a corpus along three axes: structure (mean absolute correlation between units), dimensionality (participation ratio of the correlation eigenvalues, i.e., effective rank), and per-unit distributional shape (skewness, kurtosis, SNR). TransferForge, which extends CauKer's GP+SCM structure, controls structure and dimensionality via the number of independent latent sources (n_roots) and per-unit noise weights, and matches distributional shape separately. They tested this on two pipelines: PatchTST (UCR classification), which applies z-score normalization to each input window, and POYO+ (visual stimulus orientation decoding), which uses unnormalized calcium ΔF/F. They also present a method that replaces only the neural signals in real session files with synthetic signals so the model loader can be used as is.
Results
- Accuracy was measured with PatchTST on 22 UCR datasets. An n_roots=4 synthetic corpus (about 10,000 windows) scored 0.842, higher than KernelSynth of the same size (0.831) and a real forecasting corpus (1,776 windows, 0.808). At the same size as the real corpus, there was no difference: 0.813 vs. 0.808 (p=0.60).
- In POYO+ calcium decoding (chance level 0.125), a synthetic corpus matched to calcium shape, with no real neural data, scored 0.407. This was not significantly different from pretraining on the full real data (0.386) (p=0.20), and was higher than no pretraining (0.302).
- With little real data (19 sessions), mixing in calcium-shaped synthetic data gave 0.343. Mixing in Gaussian synthetic data gave 0.220, lower even than no pretraining.
- With the model and data held fixed, changing only the input normalization changed the effect of manipulating distribution shape by a factor of 5.3.
Limitations
Few model–task combinations were tested (PatchTST/UCR, POYO+/calcium, and part of Chronos). Most results use 3 seeds, and the n_roots sweep uses a single seed. The Gaussian corpus varied widely across seeds, so the cause of its success (remaining structure) could not be isolated. Results in which synthetic data outperforms real data are either not statistically significant or appear only when the data scale is increased.
Why it matters
It proposes shifting pretraining data from something you 'collect' to something you 'design and produce' to fit the model's input pipeline. In fields where real data is expensive or sensitive, such as neural recordings or medical time series, it gives a basis for deciding before training what synthetic data to create.
Key terms
- Transfer-Invariance Principle
- The principle is that the same corpus property can be irrelevant to transfer under one input transformation and decisive under another. The claim is that which properties matter is determined by the model's input processing, not by the modality.
- Effective rank
- The number of independent modes of variation in the data, computed as the participation ratio of the eigenvalues of the correlation matrix.
- Distribution shape
- The shape of the marginal distribution of each unit's values. It is summarized by skewness, kurtosis and signal-to-noise ratio. It matters especially for sparse, heavy-tailed data such as calcium signals.
- Per-window normalization
- A step that z-scores each input window using its mean and standard deviation. It erases the scale and some of the shape information of the distribution.
Original
Authors: Sangyoon Bae, Jiook Cha · Posted 2026-11-03 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.