Quantitative Trading with Deep Learning: LSTM & Transformers

Quantitative trading with deep learning means training neural networks on past prices and other data, then letting them rank or time trades. The two architectures that matter most are LSTM networks, which read a price history step by step, and Transformers, which weigh every step against every other. Published tests show they can find patterns that simple linear models miss. They also show why most impressive backtests (simulated trading on historical data) fall apart once real money, costs, and new market conditions arrive.

Educational only, not financial advice. Investing involves risk of loss; past performance does not guarantee future results. Consult a licensed financial advisor before making investment decisions. This article explains mechanisms and published research. It does not recommend any strategy, fund, or product.

Key takeaways

• An LSTM study on S&P 500 stocks from 1992 to 2015 reported 0.46% average daily returns and a Sharpe ratio of 5.8, but both figures were measured before transaction costs.
• Machine learning gains in academic asset pricing come mostly from capturing nonlinear interactions among predictors, and the strongest signals are familiar ones such as momentum, liquidity, and volatility.
• Transformers are the dominant architecture in language modeling, yet a 2022 study found that very simple linear models beat Transformer forecasters on several long-horizon time series benchmarks.
• The main danger is not a weak model but a flattering test. Hundreds of published return factors, and millions of possible network settings, guarantee that some backtests look good by chance.
• Honest validation means walk-forward testing, purging overlapping labels, counting every strategy you tried, and charging realistic trading costs.

What linear models assume, and where markets break the assumptions

For decades, the standard toolkit for forecasting prices was a family of statistical models. ARIMA (autoregressive integrated moving average) predicts the next value as a weighted sum of recent values and recent forecast errors. It is linear: doubling yesterday's move doubles its predicted effect on tomorrow. GARCH (generalized autoregressive conditional heteroskedasticity) is a cousin that models volatility instead of direction, and it captures one real market feature, volatility clustering, meaning that calm days tend to follow calm days and wild days follow wild days.

These tools are not useless. They are transparent, cheap, and hard to overfit. But they carry assumptions that markets violate often enough to matter.

• Linearity. Real relationships can bend. A stock that fell sharply for five days may behave differently from one that fell mildly for five days, and the difference is not a simple multiple.
• Stable relationships. A model fit on one period assumes the same weights hold in the next. Interest rate regimes, market structure, and who is trading all change.
• Few interacting variables. A trader looking at momentum, trading volume, and volatility together may see a pattern that none of the three shows alone. A linear model adds their effects separately and misses the interaction.

Neural networks relax the first and third assumptions by design. They can represent bends and interactions without the researcher specifying them in advance. The price is that they have far more adjustable settings than there is trustworthy data to pin down, which is the root of nearly every failure described later in this article. For a broader primer on what a network is doing, see how neural networks work, and for the classical forecasting methods that deep learning is measured against, see how time series forecasting works.

How an LSTM reads a price history

An LSTM (long short-term memory network) is a neural network built for sequences. It was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997 to solve a specific problem: ordinary recurrent networks, which process one time step at a time and pass a summary forward, tend to forget distant inputs because the learning signal shrinks as it travels back through many steps.

The LSTM's fix is a memory cell with gates. Think of the cell as a notebook with an eraser and a pencil. At each time step, three small learned controls decide what happens to the notebook.

1. The forget gate decides how much of the existing notes to erase.
2. The input gate decides how much of the new observation to write down.
3. The output gate decides how much of the notebook to read aloud as the network's current summary.

Because the notebook is updated by gentle additions and erasures rather than rewritten from scratch each step, information that matters can survive across many days. What the network chooses to remember is learned from data, not set by the researcher.

What goes in and what comes out

A typical setup converts each trading day into a vector of numbers: the day's return, perhaps relative to the market, plus volume, volatility, and other features. A window of the most recent days, for example 240, is fed through the LSTM in order. The final summary feeds a small output layer that produces a prediction. The prediction might be the probability that a stock will beat the cross-sectional median return tomorrow, or a forecast of next-week volatility.

The choice of target matters as much as the architecture. Predicting the exact price is nearly hopeless because prices are dominated by noise. Predicting a direction, or a rank among many stocks, is a coarser task and the one most published work attempts. A small edge on a rank task, repeated across hundreds of stocks and thousands of days, is what a quantitative fund looks for.

A worked sizing example (illustrative)

Suppose a researcher uses daily data for 500 stocks over 24 years. With roughly 252 trading days a year, that is about 6,000 trading days per stock, or around 3 million stock-day rows. That sounds like a lot of data. It is less than it looks, for two reasons.

First, stocks move together. On a day when the whole market falls, 500 stocks falling provide something closer to one independent observation about the market than 500. Second, returns carry little signal per observation. The ratio of predictable structure to noise in daily stock returns is tiny. A network with hundreds of thousands of weights can therefore memorize noise without any difficulty, which is the situation regularization techniques, early stopping, and validation sets exist to manage.

What the published evidence shows

Claims about deep learning in trading are easy to make and hard to check, so it is worth anchoring on studies where the method, sample, and results are public.

The Fischer and Krauss LSTM study

Thomas Fischer and Christopher Krauss of the University of Erlangen-Nuremberg trained LSTM networks on the constituents of the S&P 500 from 1992 until 2015 (working paper, 2017). They took care to avoid survivorship bias (the error of testing only on companies that survived to the present) by rebuilding the index membership month by month from December 1989 to September 2015.

The networks predicted out-of-sample directional movements. Compared with a random forest, a standard deep neural network, and a logistic regression, the LSTM came out ahead in the authors' tests.

Model	Average daily return before transaction costs
LSTM network	0.46%
Random forest	0.43%
Standard deep neural net	0.32%
Logistic regression	0.26%

Source: Fischer and Krauss (2017), EconStor discussion paper 11/2017. Returns are before transaction costs.

The authors reported a Sharpe ratio of 5.8 before costs. The Sharpe ratio is average excess return divided by the volatility of returns, a measure of reward per unit of risk. A figure of 5.8 is extraordinary by the standards of any live fund, and the phrase "prior to transaction costs" in the abstract is the reason to treat it with caution. A strategy that trades a few hundred positions every day pays spreads and market impact on all of them, and daily-frequency edges of a fraction of a percent are exactly the kind that costs can consume.

The paper's second finding is more instructive than the headline. The authors looked inside the black box and found that the stocks the LSTM chose to trade shared a profile: high volatility, below-mean momentum, and extreme moves in the preceding days followed by a tendency to reverse. They then wrote a simple rules-based strategy that buys short-term extreme losers and sells short-term extreme winners. It earned 0.23% per day before costs and explained about 52.1% of the variance of the LSTM's long-side returns and 53.6% of its short-side returns. About 31 basis points per day of the LSTM's return remained unexplained by the simple rule, which the authors attribute to more complex patterns the network extracted.

The lesson is that the network largely rediscovered a known anomaly, short-term reversal, and added a smaller increment on top. That is a plausible and honest description of what deep learning does in markets: it refines known effects more often than it finds new ones.

Machine learning and the cross-section of stock returns

A second anchor is the 2020 Review of Financial Studies paper "Empirical Asset Pricing via Machine Learning" by Shihao Gu, Bryan Kelly, and Dacheng Xiu (NBER working paper 25398). It compared a range of machine learning methods for measuring asset risk premia, the extra return investors earn for bearing risk.

The authors report that machine learning forecasts can give investors large economic gains, in some cases roughly doubling the performance of leading regression-based strategies. The best-performing methods were tree-based models and neural networks. The authors attribute the advantage to their ability to capture nonlinear interactions among predictors. Equally important, all methods agreed on a common set of dominant signals, including variants of momentum, liquidity, and volatility.

That agreement is a useful sanity check. If very different algorithms keep pointing at the same few signals, the signals are probably real features of the data and not artifacts of one model's quirks. It also means the competitive edge in this field is thinner than the architecture debates suggest, because every serious participant can find the same dominant signals.

What a deep net is not guaranteed to do

The same Fischer and Krauss paper notes an earlier result from the same data by Krauss and co-authors: deep neural networks earned 0.33% per day before costs, below gradient-boosted trees at 0.37% and random forests at 0.43%. The authors call the weaker deep learning result surprising, given deep learning's record in speech and image recognition. Tabular financial data with a low signal-to-noise ratio is a different environment from images, and simpler tree models are often competitive. A reader who takes only one message from this literature should take this one: architecture fashion is a weak predictor of trading results.

What attention adds, and what it does not

The Transformer was introduced in the 2017 paper "Attention Is All You Need" by Ashish Vaswani and seven co-authors (arXiv:1706.03762). It was built for translation and reported 28.4 BLEU on English-to-German and 41.8 BLEU on English-to-French in the 2014 Workshop on Machine Translation benchmarks, with training for about 3.5 days on eight GPUs for the French model. The architecture dispensed with recurrence entirely and used attention instead.

Attention as a highlighter in a library

An LSTM reads a history like a person reading a book from page one, keeping a running summary in their head. Attention works like a researcher with the whole book open on the desk and a highlighter. For each position in the sequence, the model computes a relevance score against every other position, turns the scores into weights that sum to one, and builds its summary as a weighted blend. Positions that look relevant to the current question receive more weight, and the weights are recomputed for every prediction.

In a trading setting, the practical appeal is that the model can look directly at an earlier period that resembles the present, such as a prior high-volatility episode, instead of hoping a long chain of memory kept it. Attention also trains faster on modern hardware, because all positions are processed in parallel instead of one after another. If you want the language-model version of this idea in more detail, see how transformers pay attention to words.

Multiple attention heads run side by side, each free to learn a different kind of relevance. In a price model, one head might specialize in the last few days while another attends to the same calendar period in earlier years, though in practice what each head learns is found by inspection after training, not assigned in advance.

The evidence for Transformers in forecasting is mixed

Many practitioners assume Transformers must beat LSTMs on sequence data because they won in language. The evidence for time series is less one-sided. In 2022, Ailing Zeng and colleagues posted "Are Transformers Effective for Time Series Forecasting?" (arXiv:2205.13504). They introduced one-layer linear models they called LTSF-Linear and reported that these outperformed existing Transformer-based long-term forecasting models in all of their test cases, often by a large margin. They argued that self-attention, being permutation-invariant, loses temporal ordering information unless carefully compensated.

That study covered general forecasting benchmarks such as electricity load and traffic, not stock returns (it was later published at the AAAI 2023 conference). It still illustrates the central caution for finance: a more expressive architecture is not automatically a better forecaster, and a very simple baseline should always be in the comparison.

A side-by-side look at the model families

Model family	Strength	Weakness	Typical use in quant research
Linear regression, ARIMA	Transparent, hard to overfit	Misses interactions and bends	Baselines, risk models
GARCH	Captures volatility clustering	Models variance, not direction	Volatility forecasting
Random forest, gradient-boosted trees	Strong on tabular features, robust	Needs hand-built features	Cross-sectional stock ranking
LSTM	Reads sequences, learns what to remember	Slow to train, many settings	Short-horizon sequence signals
Transformer	Parallel training, direct access to any past step	Data-hungry, mixed forecasting evidence	Multi-asset and text-plus-price models

None of these rows is a ranking. In published comparisons, the winner depends on the sample, the target, and how carefully each model was tuned, and the cheaper models are frequently close behind.

Why backtests lie

A backtest is a simulation of what a strategy would have earned if it had been run in the past. It is the main evidence a quantitative researcher has, and it is also the easiest place to fool yourself.

Many tries, one reported result

Imagine 1,000 people each flip a fair coin 10 times and report their results. Several will report eight or nine heads. Nothing about those people is special; the large number of tries guaranteed that some would look skilled. Strategy research works the same way. A researcher tries different features, window lengths, network sizes, and thresholds, and reports the combination that looked best. The reported result is the winner of a tournament, and winners of tournaments overstate the typical case.

Campbell Harvey, Yan Liu, and Heqing Zhu examined this in the finance literature in "...and the Cross-Section of Expected Returns" (NBER working paper 20592). They describe a literature of hundreds of papers and hundreds of factors, and they argue that because of extensive data mining a newly proposed factor should clear a t-statistic hurdle of about 3.0 instead of the conventional 2.0. A t-statistic is a measure of how many standard errors an estimate sits from zero; a higher hurdle demands stronger evidence. The implication for deep learning is direct: a neural network trial is another draw in the same tournament.

Leakage and look-ahead

Financial data has a subtle trap that ordinary machine learning practice does not prepare people for. If the target is "return over the next 10 days," then labels for adjacent days overlap by nine days. A random shuffle that puts day 100 in the training set and day 101 in the test set lets the model see most of the answer. Researchers call this leakage, and it produces test results that look excellent and mean nothing.

Related errors include look-ahead bias (using information, such as a revised earnings figure, that was not known at the time) and survivorship bias (testing only on companies that still exist). Fischer and Krauss explicitly reconstructed historical index membership to avoid the second, which is the kind of care a credible study shows.

Costs and capacity

A backtest that ignores trading costs is a hypothesis about a frictionless market. Real trading pays the bid-ask spread, pays for market impact when orders are large relative to available liquidity, and sometimes pays to borrow shares for short positions. Strategies that trade often, which is what daily-frequency neural signals tend to do, are the most sensitive to this. A signal that earns a fraction of a percent per day before costs can disappear if costs consume most of it. Capacity matters too: a signal that works with a small amount of money may stop working when scaled, because the trades move the price.

Validating a model without fooling yourself

The remedy for most of these problems is process, not cleverness.

Walk-forward testing

Walk-forward testing respects the order of time. Train on an early block of years, test on the next block, then slide the window forward, retrain, and test again. A hypothetical schedule for a 20-year dataset might train on years 1 through 10 and test on years 11 through 13, then train on years 1 through 13 and test on years 14 through 16, and so on. Every test period is genuinely out of sample relative to its training period, and the stitched-together test results show how the model behaved across different market conditions. A strategy that earns its return in only one of the test blocks is a warning sign.

Purging and embargo

To handle overlapping labels, researchers remove from the training set any observation whose label period overlaps the test period (purging), and sometimes skip a buffer of additional days after the test block (an embargo). These techniques are associated with Marcos López de Prado's work on financial machine learning. They cost training data, which is why they are skipped, and skipping them is a reliable way to inflate results.

Count your trials

The more configurations you test, the higher the bar a result must clear. Methods such as the deflated Sharpe ratio, proposed by David Bailey and López de Prado, adjust a Sharpe ratio downward for the number of trials and for non-normal returns. The practical habit matters more than the formula: keep a log of everything tried, including the failures, and judge the final candidate against that log.

Keep a final holdout

Set aside the most recent period before any development begins and look at it once. If you tune the model after seeing the holdout, it stops being a holdout. This is the financial equivalent of not peeking at the exam answers, and it is violated constantly because the temptation is strong. The same logic applies to benchmark scores in AI generally, a theme explored in why AI benchmarks can be misleading.

Markets change: regimes, retraining, and the 2022 example

A model trained on one era learns that era's relationships. When the era ends, so does the validity of what it learned. Researchers call these periods regimes, and regime shifts are the standard explanation for why a strategy that worked for years can fail abruptly.

The 2022 interest rate cycle is a clean illustration. According to the Federal Reserve's published record (FOMC calendar page), the target range for the federal funds rate began 2022 at 0 to 0.25% and was raised seven times, on March 17, May 5, June 16, July 28, September 22, November 3, and December 15. The December increase brought the range to 4.25% to 4.50%. Several of the steps were 75 basis points, an unusually large size. A model trained almost entirely on the preceding decade of near-zero rates had never seen a tightening cycle of that speed in its training window, and any relationship it had learned between rates, valuations, and returns was untested.

Practitioners respond in a few ways. Retraining on a schedule lets the model absorb recent data, though retraining too often chases noise. Including regime indicators, such as the level and recent change of interest rates or measures of market stress, lets one model behave differently in different conditions, though it also adds more parameters to overfit. Combining several models with different training windows reduces dependence on any single view of the world. None of these is a cure, and the right retraining frequency is an empirical question for each strategy, not a rule of thumb.

When many models do the same thing

A single strategy's failure is a private loss. Many similar strategies failing together is a market event.

Amir Khandani and Andrew Lo analyzed the week of August 6, 2007, when several quantitative long/short equity funds suffered unprecedented losses (NBER working paper 14465). They hypothesized that a coordinated deleveraging of similarly constructed portfolios caused a temporary dislocation, and their simulations suggested the unwinding began in July 2007 and continued through the end of that year. They also found that a simple market-making strategy lost money that week but was profitable before and after, consistent with liquidity providers temporarily stepping back. Their conclusion pointed to significant systemic risks posed by the hedge fund industry.

The mechanism is the important part. If many firms hold similar positions because their models read the same signals, a forced sale by one pushes prices against all of them, which triggers risk limits and more selling. The more a signal is shared, the more crowded the exit. This is one reason a successful signal tends to weaken as it becomes known, and a reason regulators now study correlated model behavior. For more on the systemic side, see can AI predict the next financial crisis.

Operational risk is a separate failure mode that has nothing to do with prediction. In August 2012, Knight Capital lost more than $460 million. In the first 45 minutes after the market opened on August 1, its order router sent more than 4 million orders while trying to fill 212 customer orders, and the SEC found the firm lacked adequate safeguards on its market access (SEC press release 2013-222). Knight paid $12 million to settle the charges. The episode involved faulty deployed code, not a machine learning model, but it shows why serious firms spend as much effort on controls as on signals.

What the best-known quant fund tells us, and what it cannot

Renaissance Technologies' Medallion Fund is the standard example cited by people who believe markets can be beaten systematically. According to journalist Gregory Zuckerman's 2019 book The Man Who Solved the Market, as summarized on Wikipedia's Medallion Fund entry, the fund averaged about 66% annually before fees and about 39% after fees. The same entry reports that it has been closed to outside investors since 1993 and is open only to current and former employees and their families.

Two points follow. First, the figures come from a book and press reporting, not from audited public filings, so they should be treated as reported estimates. Second, and more important, Medallion's public record says almost nothing about whether deep learning works. The fund's methods are secret, its strategies are described as short-horizon and capacity-limited, and the fund is deliberately kept small. The existence of one extraordinary outlier tells a reader little about what an individual with a neural network and public data can expect, in the same way that one lottery winner tells you little about the lottery.

The takeaway

Deep learning has earned a real place in quantitative research. Studies such as Fischer and Krauss on LSTMs and Gu, Kelly, and Xiu on machine learning for risk premia show that nonlinear models can extract more from market data than linear ones. But the same studies show that the signals are mostly familiar ones, the gains are measured before costs, and simpler models often come close.

The harder skill is not building the network. It is building a test that the network cannot cheat, then accepting the answer when it is disappointing. Strategies decay, crowded trades unwind together, and a simulated Sharpe ratio is a claim, not a result. A reader who remembers one principle should remember that the more impressive a backtest looks, the more of its history deserves scrutiny.

Frequently asked questions

Q: Do LSTM models actually predict stock prices? A: Published research shows LSTMs can produce statistically significant directional signals in some samples, such as the 1992 to 2015 S&P 500 study by Fischer and Krauss, which reported 0.46% daily returns before transaction costs. That is not the same as reliably predicting prices. The results are measured before costs, depend on the sample period, and largely reflect a short-term reversal pattern that simpler rules can also capture.

Q: Are Transformers better than LSTMs for trading? A: There is no settled answer. Transformers dominate language tasks, and their direct access to any past step is appealing for markets. But a 2022 study by Zeng and colleagues found simple linear models beat Transformer forecasters on several general time series benchmarks. For financial data with a low signal-to-noise ratio, results depend heavily on the dataset, target, and tuning, so a fair comparison needs the same validation for each model.

Q: What is overfitting in a trading model? A: Overfitting means the model has learned the noise in its training data instead of a repeatable pattern, so it performs well on history and poorly on new data. It is common in finance because the signal is weak and the model has many adjustable parameters. The usual defenses are walk-forward testing, a final holdout period, regularization, and counting how many strategies were tried.

Q: What is walk-forward validation? A: Walk-forward validation trains a model on an earlier block of time, tests it on the following block, then rolls both forward and repeats. It respects the order of time, so the model is never tested on data that preceded its training data. Consistent performance across many blocks is stronger evidence than a single backtest over the whole sample.

Q: Why do published trading strategies stop working? A: Several causes overlap. Some results were produced by data mining and never existed. Others were real but get competed away as more participants trade the same signal. Markets also shift regimes, such as the 2022 rise in interest rates. Harvey, Liu, and Zhu argue that because of extensive data mining, new factors should meet a higher statistical bar than the conventional one.

Q: Can an individual investor use deep learning to beat the market? A: The research reviewed here does not support that expectation. Individuals face higher trading costs, less data, and less infrastructure than the institutions that publish these results, and even institutional studies report returns before costs. Anyone experimenting should treat it as education, use paper trading, and remember that a good backtest is not evidence of future profit.

Sources

1. Fischer, T., and Krauss, C. (2017). "Deep learning with long short-term memory networks for financial market predictions." FAU Discussion Papers in Economics 11/2017. EconStor
2. Gu, S., Kelly, B., and Xiu, D. "Empirical Asset Pricing via Machine Learning." NBER Working Paper 25398; published in The Review of Financial Studies 33(5), 2020. NBER
3. Vaswani, A., et al. (2017). "Attention Is All You Need." arXiv:1706.03762
4. Zeng, A., Chen, M., Zhang, L., and Xu, Q. (2022). "Are Transformers Effective for Time Series Forecasting?" arXiv:2205.13504
5. Harvey, C., Liu, Y., and Zhu, H. "...and the Cross-Section of Expected Returns." NBER Working Paper 20592. NBER
6. Khandani, A., and Lo, A. (2008). "What Happened to the Quants in August 2007?" NBER Working Paper 14465. NBER
7. Federal Reserve, Federal Open Market Committee, 2022 target range decisions. federalreserve.gov
8. U.S. Securities and Exchange Commission, press release 2013-222 on Knight Capital. sec.gov
9. Medallion Fund return figures, citing Zuckerman (2019). Wikipedia

---

Read next:

• Algorithmic Trading & Neural Networks: How AI Makes Billions Per Second
• Reinforcement Learning for Portfolio Optimization: Teaching AI to Invest
• Crypto Trading Bots and Machine Learning: Separating Hype from Reality