Reasonable data size for backtest

fr · Post by fr » Thu Aug 30, 2012 2:35 am

I'm backtesting a strategy which makes a reasonable return after transaction costs. The asset class is index futures. The strategy is relatively low capacity for a single future, and I still haven't had time to test whether returns are comparable using other contracts, which would allow larger capacity.

The strategy would trade at most once per day and it currently decides to trade in ~40% of the possible trading days (so approx 100 days each year). I've done my tests with 1 year's worth of data, but I feel it's not enough. If you were to risk your money in my strategy, how long would you want the backtest to be in order to be comfortable? (I could use data for 10 years and still overfit, but assume that's not the case and that all you are trying to decide now is the minimum backtest length.)

One possible answer would be "use all the historical data that is available for the instrument", but some people argue that markets are not the same today as a few years ago (even though this strategy should not be affected much by market microstructure).

Thanks

functor · Post by **functor** » Thu Aug 30, 2012 3:03 am

If you assume you're not overfitting, then you can get standard errors on sharpe ratios (see the paper by Lo). Unfortunately, the fact that you are not overfitting is going to be the primary issue. On a single future market it is ridiculously easy to overfit and get good backtested equity curves for the number of trades you mention, so the p-values are meaningless.

HankScorpio · Post by **HankScorpio** » Thu Aug 30, 2012 3:30 am

Start by performing a search in the Trading phorum. This was discussed many moons ago and FDAXHunter (amongst others) provided some good advice.

ricko · Post by **ricko** » Mon Sep 03, 2012 2:22 pm

It is hard for anyone to answer your questions unless they know:

1. The exact instrument you are trading

2. The timeframe of the data and the backtest results

3. The distribution of trades and profits

Overfit reduces substantially with sample size. Those who appeal to changing conditions have a reason doing that and often it is the fact that their system fails as they increase the data history.

Have you done an out-of-sample test of significant sample size? Why don't you post the backtest results here?

fr · Post by fr » Sun Sep 16, 2012 12:43 pm

I'm going to agree with functor about the risk of overfitting with the number of trades I mention. In particular, I'd be more confident in the results if the backtest included data from the 'quant meltdown' of 2007.