LLMs Don't Know What They Know: A Temporal Self-Knowledge Benchmark for Financial Backtesting

Problem

To backtest an LLM's financial forecasting ability on historical data, the model must not have already learned the actual outcomes it is asked to predict. In practice, people try to ensure this by trusting the knowledge cutoff stated by the provider, asking the model whether it knows, or instructing it to 'answer as of a past date'. However, whether these safeguards are actually reliable has rarely been tested.

Method

FinDendro consists of 2,226 US facts from 1999–2026: first-release values of 12 FRED macro indicators, month-end closing prices (before split adjustment) of 50 large-cap stocks, and Federal Reserve rate decisions. Four prompts are built for each fact: choosing the value (Task 1); Recall and Predict questions that pair each candidate value with a 'know/guess' claim (Task 2); and Predict with 'assume today is a specific past date' added (Task 3). Numeric facts that can be answered by repeating the previous month's value are excluded, and because 78% of rate decisions are 'hold', balanced accuracy is used to set chance accuracy at about 0.27. Four frontier models, two small open models and three families of point-in-time models were evaluated.

Results

• GPT-5 answered 76% of controlled facts correctly but claimed to know only 38%. Within its measured knowledge period, it also answered 75% of the facts it said it did not know correctly; after that period, the figure was 33%. The authors report that most frontier models under-report their own knowledge.
• Llama-3.3-70B's accuracy dropped to chance level after mid-2021, well before the provider's stated cutoff (December 2023), while its claims of knowing continued until early 2023. The authors report that claims of knowing can extend up to seven quarters beyond the measured boundary.
• When GPT-5 was instructed to answer as of a past date, its claims of 'already knowing' fell from 39% to 1%, but its accuracy stayed almost unchanged.
• The small point-in-time models had accuracy close to chance both before and after their training cutoffs.

Limitations

The facts are limited to US macro indicators, prices of 50 large-cap stocks and Fed decisions, and samples before 2021 are sparse. All questions are multiple choice, so leakage in free-form forecasts is not measured directly. Only four frontier models were evaluated. Some per-model detailed results cannot be confirmed within the provided text.

Why it matters

It shows with concrete numbers that backtest performance of LLM-based investment and forecasting strategies can be inflated by 'answers the model already knows'. Instead of relying on official cutoffs or prompt instructions, it proposes a practical procedure: measure the model's actual knowledge boundary behaviorally and choose the backtest period accordingly.

Key terms

Look-ahead bias (hindsight information leakage)

A problem in which the model contains future information that was not available at the time of prediction, making backtest performance look better than it really is.

Knowledge cutoff

The point at which a model's training data ends. This paper shows that the official cutoff, the actually measured boundary and the boundary the model claims for itself can differ.

Balanced accuracy

The average of per-class accuracy. It prevents a high score from always guessing the overwhelmingly common outcome, such as 'hold'.

Point-in-time model

A model trained only on data before a specific point in time, so that it has no information from after that point.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang, Zhuohan Xie, Anke Xu, Wenbo Cao, et al. · Posted 2026-11-04 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.