Trading system architecture

goldorak · Post by **goldorak** » Fri Jul 29, 2011 7:47 am

> Some would be surprised by the degree of computational complexity required nowadays to discover promising strategies. Strategy guys that can regularly generate unique Sharpe 3.0+ strategies with validation done right are always short on computational power.

I tend to call this over-fitting or data-snooping. Validation 'done right' is usually your own definition of 'done right'. Unfortunately we usually all miss knowledge when it comes to that kind of strategy developments.

dgn2 · Post by **dgn2** » Fri Jul 29, 2011 12:13 pm

Most automated trading systems have conditions under which they may need to reinitialize. In this case one may also need to incorporate a backtesting component.

CUDA is useful for automating parameter selection particularly if you use walk forward. I use a framework where I select parameters based on a large set of price path scenarios derived from the data (bootstrapping and other scenario generating schemes). Since parameters evolve, massive computation is necessary, particularly for a large number of instruments.

prophet · Post by **prophet** » Fri Jul 29, 2011 3:44 pm

>> Some would be surprised by the degree of computational complexity required nowadays to discover promising strategies. Strategy guys that can regularly generate unique Sharpe 3.0+ strategies with validation done right are always short on computational power.

> I tend to call this over-fitting or data-snooping. Validation 'done right' is usually your own definition of 'done right'. Unfortunately we usually all miss knowledge when it comes to that kind of strategy developments.

So some computational power is good, but more is bad? All resources (capital, manpower, data, cpu) require skill, experience and self-discipline to be used effectively. The fact that many misuse a resource does mean the resource is the problem.

We live and die by managing over-fitting. I say manage because over-fitting can never truly be eliminated. Over-fitting can be quantified in various ways. The strategies themselves can also be designed to avoid over-fitting. Such techniques are an essential skill for modern trading strategy design, just as important as knowing what broad classes of strategies are still profitable.

dgn2 · Post by **dgn2** » Fri Jul 29, 2011 6:23 pm

Basically what I am trying to say is that testing for robustness is computationally intensive and this activity is likely to continue even after a strategy is deployed.

Praetorian · Post by **Praetorian** » Fri Jul 29, 2011 8:41 pm

Interesting points of view. I never thought about adding the backtesting component to the "trading system" for live trading, because I am used to wait for half an hour for a backtest of only one trading day. To backtest the strategy for all historic data takes several days. That's stuff that runs at night and not during the day.

I always reasoned that if I need to do "complex stuff" during the day I would introduce lag that can not be reproduced but must be approximated in the backtest. I figured that including some form of computation time in the backtest would be a mess.

If you recalibrate/backtest during the day how do you guys deal with the lag of computation? If you need CUDA for performance it seems that there is non negligible computation time involved. Do you recalibrate at fixed intervals and don't trade at these intervals?

dgn2 · Post by **dgn2** » Fri Jul 29, 2011 9:31 pm

I recalibrate at a fixed interval. This is as much to check that the system is behaving as it should as it is to find "new" parameters. Robust parameter selection should not lead to parameters that bounce around excessively in my opinion. In fact, if there isn't either 1) some dependence in your parameter estimates from walk-forward period to walk-forward period or 2) they aren't constant after accounting for expected variation (i.e. distributed in a well behaved way) then there may be a problem with the way parameters are selected (IMHO).

Think about implied vols. They are a model parameter arrived at by cranking your model backwards. They are useful because they are long range dependent. It doesn't matter what they "mean" only that you can use them to do feedforward. They represent the market state and having a view about them can be easily converted into prices / corresponding bets.

There are many similar measurements that can be used to do feedforward control to enhance your risk management or signals or both depending on the type of strategy. Most trading systems are highly path dependent like options. I crank mine backwards to get a set of implied parameters in such a way as to get a market condition that can be used in controls.

goldorak · Post by **goldorak** » Sat Jul 30, 2011 6:10 am

@prophet

Sharpe 3.0+ strategies ARE the result of over-fitting and data-snooping. If not it simply means sharpe ratio is not the right measure or that you have annualized the measure using the square root of time while you maybe should not have. One of our strats has been running with Sharpe ratio around 21 for about 6 months while in backtesting Sharpe ratio was below 2...

Sorry, I did not intend to hijack the thread. Radiation

Just found back this very interesting plot published on World Beta blog. For what it is worth.

[img]/User%20Files/329/ctas2-1024x639.jpg[/img]

macrotrader · Post by **macrotrader** » Sat Jul 30, 2011 12:23 pm

prophet, thanks for the example. What other classes / threads would you have around the strategy class for "manage positions, orders, react to fills, transmit trades, etc.". For instance order timing and routing are features which most if not all strategies will have in common. As I understand these will be not in their own thread (besides a GUI). I thought that maybe having a server which receives and validates the strategies actions might be used as a firewall for malfunctions.

Praetorian · Post by **Praetorian** » Sat Jul 30, 2011 3:15 pm

If you trade at fixed heartbeats it is handy to have a trade aggregator to eventually save some transaction costs and impact.

prophet · Post by **prophet** » Sat Jul 30, 2011 3:29 pm

> because I am used to wait for half an hour for a backtest of only one trading day. To backtest the strategy for all historic data takes several days.

How much market data is involved? I favor strategies that can backtest at a rate of 1 to 20 strategy*instrument*years per second. Performance will always vary wildly with the nature of the strategy. I've found that intelligent caching of computational intermediates is essential.

> I always reasoned that if I need to do "complex stuff" during the day I would introduce lag that can not be reproduced but must be approximated in the backtest.

That's why in backtesting I intentionally lag my trade signals enough to allow for real-time computation lag and data feed delays.

> If you recalibrate/backtest during the day how do you guys deal with the lag of computation?

If recalibrate means re-optimize, I have not re-optimized during the day. Lots of strategies re-calibrate constantly by design, within limits. Such re-calibration should not be computationally complex or slow enough to impact executions. For example, online / moving window covariances can be used instead of periodically calculating a covariance.

> Sharpe 3.0+ strategies ARE the result of over-fitting and data-snooping.

Not in general. Data-snooping is a problem for inexperienced developers. Over-fitting is indeed a huge problem. Market regime-change is another. Many times regime-change is mistakenly believed to be over-fitting.

Nice chart. It fits with experience.

> What other classes / threads would you have around the strategy class for "manage positions, orders, react to fills, transmit trades, etc.". For instance order timing and routing are features which most if not all strategies will have in common. As I understand these will be not in their own thread (besides a GUI).

I haven't personally used a class or library that manages positions, trades, fills, etc. But many developers this to significant advantage.

> I thought that maybe having a server which receives and validates the strategies actions might be used as a firewall for malfunctions.

Excellent point. A seperate server might be overkill, at least starting out. I would suggest using a library to manage your trades/executions. You need to test this library throughly and prove the functions are correct.

Then ensure your strategy code cannot have any data-snooping problems. One simple approach is to encapsulate your entire strategy into one large function. Strategy state is stored in statically declared variables or arrays provided by the calling function (but never modified by the caller). You then call the strategy function and provide it with a new bar of market data. Now the strategy function can never see future bars. Obviously this has computational inefficiencies.

That pseudo-code I provided earlier demonstrates another method to avoid data-snooping, provided you have certain guarantees. As long as every thread receives the same timestamps (to break the inner loop on a bar close), the market data itself has no causality violations and that inner loop is never skipped, no thread can fall behind or effectively snoop on future data. I would also include various causality checks to be extra careful. Data snooping is a real time waster.