Large language models (LLMs) are increasingly used to analyze companies, rank securities and generate inputs for portfolio decisions. Taking a model's output at face value, however, introduces two systematic sources of bias. The first is memorization: the model may recognize a company from its data and draw on what it already knows about the company's subsequent performance. The second is elicitation bias: the answer depends on how the question is framed and on the order in which the inputs are presented. This article describes both mechanisms, quantifies them using the results of a recent study[1], and sets out a methodology to reduce them.

1. Context

AI has moved quickly from experiment to everyday tool across the investment industry: 70% of investors now use AI to support their investment decisions[3]. Applications range from summarizing earnings calls to automated pipelines in which model outputs feed directly into screening, ranking and allocation.

Methodological scrutiny has not kept pace. A February 2026 review of 164 academic papers on AI in finance identified five recurring biases that can distort reported results, and none of them was addressed in more than 28% of the studies[2]. Part of the reported performance of LLM-based approaches may therefore reflect these biases rather than genuine predictive ability.

The study summarized here measured two of these biases on real data from large US-listed companies, using two frontier models and two smaller open-weight models of the kind commonly deployed in internal tools.

2. Memorization and Look-Ahead Bias

Mechanism

LLMs are trained on large corpora of public text that include company filings, earnings coverage, news and market commentary spanning many years. A model can therefore associate a set of reported figures with a specific company, and the company with what happened to it afterwards. When such a model is evaluated on historical data, information from after the evaluation date can leak into its output. This is a form of look-ahead bias.

The effect is concentrated in backtesting. A strategy that appears to select outperformers from fundamentals may in fact be recognizing the companies and retrieving their known subsequent returns. The resulting backtest overstates the performance that could be achieved out of sample.

Measurement

Each model received real reported figures (revenue, net income, financial ratios and the reporting period) with no company name and no sector, and was asked to identify the company. The test was repeated under three input representations: the raw figures; the figures expressed relative to sector peers; and the peer-relative figures without any indication that a transformation had been applied.

Model Raw figures Compared to peers Compared to peers, unlabeled
Gemma 3 27B14.7%5.3%0.0%
Opus 512.6%4.2%1.1%
GPT-5.6 Sol11.6%3.2%0.0%
Qwen3 32B6.3%4.2%3.2%
Figure 1: Share of companies identified from their financial figures alone, by input representation

Results

Every model identified some companies from raw figures, with identification rates between 6.3% and 14.7%. Recognition was uneven across companies: Apple was identified every time and NVIDIA 75% of the time, while Microsoft was identified only 5% of the time and Tesla never. The pattern is consistent with the retrieval of specific memorized figures, not with a heuristic that favors the largest companies.

Anonymization by removing names is therefore insufficient. Expressing the figures relative to peers reduced identification by roughly a factor of two to four for three of the four models, and omitting the description of the transformation reduced it further. Even then, two of the four models still identified 1–3% of companies. Because a backtest compounds over many periods, a small number of recognized positions can be enough to bias its results materially.

3. Elicitation Bias: Framing and Ordering

Mechanism

An LLM does not hold a fixed internal ranking of the companies it is asked about. Its output is generated conditionally on the prompt, so the requested format (a rank, a weight or a score) and the position of each company in the input both influence the answer. If the model's judgments were stable, logically equivalent questions would produce the same ordering.

Measurement

Each model received sets of six companies and was asked for the same judgment in three formats, always with the objective of maximizing expected return:

Rank

Order the companies from best to worst.

Allocate

Split $100 across them.

Score

Rate each one from 0 to 100.

The three formats encode the same underlying preference, so the implied orderings should coincide. The example below, produced by one of the frontier models, shows that they do not.

The same six companies ordered three ways by Opus 5 (by rank, by dollar allocation and by score), showing different orderings depending on how the question is asked.
Figure 2: Same companies, three formats, three different orderings

Results

Across 10 sets of six companies, the frontier models were largely but not fully consistent: the three formats agreed on the top pick in 90% of sets for GPT and 70% for Opus. The smaller models were considerably less stable: ranking and scoring produced the same full ordering in only 25% of sets for Gemma and 15% for Qwen. For Qwen, reversing the order in which the companies were listed changed the top pick in 6 of 10 sets.

Left: share of sets where rank, allocate and score pick the same top company (Opus 70%, GPT 90%, Gemma 70%, Qwen 35%). Right: share of sets where rank and score give an identical ordering (Opus 75%, GPT 65%, Gemma 25%, Qwen 15%).
Figure 3: Agreement between formats, by model

A single model output therefore contains a component that is unrelated to the companies themselves and is determined by the prompt format and the input order. When outputs feed a portfolio directly, this component becomes noise in position selection and sizing, and the noise is largest for the smaller models most often embedded in automated pipelines.

4. A Methodology to Reduce Both Biases

Neither bias can be removed entirely, but both can be measured and substantially reduced. The methodology below treats the model as a measurement instrument whose errors are estimated and controlled before its outputs are used.

Step 1: Remove identifying information from the inputs

Removing names and sectors is a necessary first step, not a sufficient one. Raw financial figures act as fingerprints, so inputs are transformed into cross-sectional representations: each metric is expressed relative to a peer group at the same date, for example as a percentile rank or a standardized deviation from the peer median. Exact reporting dates are removed or coarsened, and the prompt does not describe the transformation. In the study, peer-relative, unlabeled inputs reduced identification from 6–15% to 0–3%.

Step 2: Test for residual memorization before any backtest

The identification test described in section 2 is run on the transformed inputs as a gate. If the model identifies companies at a rate meaningfully above chance, the representation is revised before the model is evaluated on returns. Where the model's training cutoff is known, backtests are also evaluated separately on periods after the cutoff, where memorized outcomes cannot exist. A material gap between pre-cutoff and post-cutoff performance is a direct estimate of the remaining look-ahead bias.

Step 3: Make outputs invariant to framing and ordering

Instead of relying on a single query, each decision is elicited several times: in more than one format (rank, allocation and score) and with the inputs presented in several random orders. The resulting orderings are aggregated into a consensus, for example by averaging each company's rank across all queries (a Borda count). This averages out positional and framing effects that are not tied to the companies themselves.

Step 4: Measure agreement and treat disagreement as information

The dispersion across queries is recorded alongside the consensus, using top-pick agreement and a rank correlation such as Kendall's tau. Low agreement indicates that the model's view of a set of companies is not stable. Such outputs are down-weighted or treated as having no signal, rather than being forced into a position.

Step 5: Fix and version the protocol

Prompt templates, model versions and sampling parameters are fixed and versioned, so that results are reproducible and comparable over time. Because bias levels differ widely across models and across versions of the same model, steps 2 and 4 are rerun whenever any component changes.

5. Summary

  • Memorization creates look-ahead bias

    All four models identified companies from anonymized figures, at rates up to 14.7%. Backtests built on raw figures can therefore embed knowledge of future outcomes.

  • Representation, not anonymization, controls it

    Peer-relative, unlabeled inputs reduced identification to 0–3%. An identification test on the final inputs, and evaluation after the training cutoff, measure what remains.

  • Single queries are unstable

    Equivalent questions produced different orderings, and input order alone changed results. Aggregating across formats and orderings, and measuring agreement, separates the model's judgment from prompt artifacts.

Sources

  1. LLMs for Financial Analysis: More Capabilities, More Biases (Sabr Research, 2026)
  2. Evaluating LLMs in Finance Requires Explicit Bias Consideration (arXiv 2602.14233, Feb 2026)
  3. Amundi, 2026 annual report