Bootstrap resampling applied to fund returns

[intradaybill]

What do you think about the analysis in this blog:



http://www.priceactionlab.com/Blog/2010/11/the-bootstrap-method-for-hypothesis-testing/



Does it make sense? Specifically, is it possible from resampling monthly returns to determine whether the edge of a fund is fading away?

[aaron]

I don't care for it.



First, it doesn't mention the one crucial assumption for using a bootstrap, that the data are i.i.d. When people question the track record of a hedge fund, it is usually because (a) they don't believe the published returns, (b) they think the fund was selected from a larger group, (c) they think there is some small probability of a large loss absent from the data or (d) they believe the distribution is non-constant (for example that the manager doubles up after losses, or smooths returns) . None of these are tested by the bootstrap and, in any case, mean the future won't be like the past.



Second, for this application, there is rarely significant difference between the bootstrap answer and a Normal hypothesis test. If there is a significant difference, that is more important than wondering whether or not the mean is positive. Suppose, for example, the fund's returns consist of 99 1% up months and 1 down 50% month. A Normal test gives a 46% significance, a bootstrap gives 74%. But would any reasonable person look at either one of these? Instead you'd look at the 50% down month to see what happened, how likely it was to recur and how big the loss could have been.



Third, although this is minor, there's no need to resample for this problem. You can easily compute the outcome rather than simulating.



Fourth, another minor point, testing whether the expected return is positive is not useful. I don't like hypothesis tests in the first place (confidence intervals are preferable for almost all real decisions) but at least you should test a conclusion that would be actionable.



Finally, eliminating some years to see what the effect is on the bootstrap makes no sense to me. Why calendar years for one thing? And a p-value is a measure of certainty, not of return. That eliminating data makes it harder to form a strong conclusion does not mean the removed data prove that the fund is bad. The logic here escapes me entirely.

[veroniq]

I'm with Aaron on this. Bootstrap assumes iid, without it the whole exercise is meaningless. Even if we try to relax the iid condition, not correcting for different volatility regimes (whatever that means) invalidates the results, which is btw beautifully shown there by the difference in the outcomes if we take different time periods. And I would reiterate Aaron's point about p-values being the measures of certainty, where 0.05 is almost arbitrarily chosen.

Conclusion: the guy doesn't really understand statistical methods he's trying to use and applies them in a way that doesn't make sense.



The whole idea of resampling is quite sound though, but you need to think about necessary conditions and whether they will be attainable. Trying to use it on monthly returns over short periods is just using very small non-homogeneous sample. Meaningless.

[intradaybill]

As far as the i.i.d. I think the guy explicitly stated the assumption: "The bootstrap method of computational statistics offers a way of reproducing the actual distribution of a population and it can be used with any sample statistic without relying on any assumptions about the data other than the basic assumption that they are distributed similarly to the population data."




So I wonder whether you guys actually read the whole post or you jumped to the results.



As far as 20 years of monthly returns being a short period I do not think so. I think it captures a wide spectrum of market conditions. The real question is whether this analysis can eliminate selection bias in my opinion. I believe statistically the study is sound and it is not fair to say that the guy does not understand it because he actually says that: "It appears that the question of randomness is not an easy one to answer within the context of naive statistical hypothesis testing. "



Thus, I think you guys, and especially Veroniq, did not read the study carefully. It appears that the guy tries to claim the opposite of what you think i.e. that bootstrap resampling cannot provide valid clues against randomness. This is how I interpreted the study.

[Maggette]

Beeing "similary distributed to the population data" says nothing about i.i.d. ! It just says, that your sample is a representative one...not that the process that generates the population is iid.



So, as always, aaron and veroniq are right.

Wink

[FatChoi]

20 years a lot of data?



Suppose the fund has genuinely IID gaussian returns drawn from a distribution with a fixed mean -- it really is a geometric brownian motion with fixed parameters -- and suppose that we know, not just estimate but know, that the annual volatility is 14% so that it has a monthly volatility of 4%. What is the standard deviation of the sample mean monthly return? 4%/sqrt(240) = 0.25% or 6% annualised and that's in log terms! That is in the best possible case. If the returns are not lognormal, heteroskedastic or autocorrelated you know much less.



For a real fund, what makes returns now related to returns in 1990? People, models, processes and market conditions are all different. What does behaviour in the dot.com boom have to do with behaviour in a credit crisis? How would the Madoff fund look?



Beyond that you can't look at a fund in isolation. There are tens of thousands of funds so even if all performance is totally random you would expect hundreds to pass the test applied in the article. The analysis described is probably marginally better than applying the same reasoning to the sample variance but the limitations of the approach are less transparent. Doing the analysis more carefully is quite a lot of work -- see here for example (I think they still omitted to compensate for the lost degree of freedom in their regression model mind). Not that this approach guarantees results.

[bluelou]

Focusing on boostrapping and iid data for a moment...



Assuming the data are dependent to some degree couldn't you run a block bootstrap with random block sizes to mitigate this? Alternatively, wouldn't it make sense to analyse the ACF and use a block length > the # of lags found in the ACF?



Any thoughts on using either of these 2 approaches?

[bluelou]

...

[intradaybill]

@Maqqete,



"Beeing "similary distributed to the population data" says nothing about i.i.d. ! It just says, that your sample is a representative one...not that the process that generates the population is iid.



So, as always, aaron and veroniq are right. "



What about de Finetti's theorem? Exchangeability may suffice for bootstrap validity?