How different are your backtesting and production systems. i.e vectorized vs event loop

goldorak · Post by **goldorak** » Tue Jan 17, 2012 9:48 am

In our history of development we tended to have a common software for statistical analysis but completely different softwares for portfolio construction/management. We got bad surprises in the past. One example is that we were implementing portfolio construction algorithms that you could not really backtest and discovered flawss in the logic years after.

We then decided to follow one single rule: the backtesting program should be able to run exactly the same once used in live trading. This limits certain optimizations one may have introduced, but all in all our experience so far is very positive.

To add to Praetorian's post, the use of the backtesting system as a live trading system definitely brings to light a lot of bugs/errors/mistakes. Running the system from date A to B as a backtesting system and then again from date B to C as a live trading system can be compared to a run between date A to C as a backtesting system. Logic flaws in you programs are far more easier to spot that way. On top of that the amount of data involved in our software makes it a very good platform to test improbable events/collisions, events you would not even have thought of while developing.

Finally 95% of our softwares are written in perl. Not really the most efficient language you may find out there I admit, but definitely good at what we need it for, which in the end is nothing more than data mining. The very few pieces of function where real numerical efficiency is required are replaced with inline C code or calls to the Gnu Scientific Library.

To conclude on the topic, I just want to add that R/Matlab are not really necessary as strategy development tools. The problem being that with 10 researchers looking for good strategies at the same time, trying a lot of ideas, and worse optimizing the results, data snooping becomes a real issue. This is why this research process is delegated to the software itself which is then told how to control for data snooping. A real plus. I would rather consider R/Matlab as high level research tools to study the inner working of the markets, not to develop strategies to take advantage of these "mechanics" in term of pennies!

William_Weaver · Post by **William_Weaver** » Tue Jan 24, 2012 8:03 pm

Praetorian has a few great points. Only to add to the list of languages used, I use a language built from APL (which it seems no one uses anymore, at least in the States) that was originally created by a futures trader and PhD mathematician. It's nice because the entire language is written around what I want to test/implement, so for the most part any basic testing (example downloading or referencing minute data on 3000 stocks and ranking with a momentum strategy along with compiling an equity curve of the top n) can be tested in less than 8 lines of code, and in only a few seconds. The program is designed without drop down menus to avoid retracing steps and testing the same things over and over again, but because there is a language underneath this application, if I do need to add functionality I can go directly to the original APL code (which is the fastest code available for processing large arrays - good for finance). This way there is no separation between development and production. But the downside is that having separation, although that allows for bugs, doesn't force you to re-test your hypothetical versus actual results, so one just needs to be diligent about that.

jslade · Post by **jslade** » Wed Jan 25, 2012 6:16 am

WW: are you using K? J? Q? A+? Don't be shy!

I've screwed around in Q and liked it a lot, except for the price.

ralph · Post by **ralph** » Tue Jan 31, 2012 10:47 am

"...I use a language built from APL"

I echo jslade, which language in particular do you speak of? I'm very interested in this regard. And yes, there are a few of us left in the US that use APL and its offspring.

jslade: Have you seen An interesting intro video to Kona, and an example of doing mapreduce in Kona.

jslade · Post by **jslade** » Sat Sep 08, 2012 1:30 am

Missed this post Ralph; thanks for the pointer to Kona. Wish WW had replied... I might pinch my nose and have a stab at one of these APL things again.