Distributed Cache and Back Testing

[dcworthi]

Hello all,



I work for a company that produces middleware for in-memory data grids. It also has a Map/Reduce capability w/ parallel speed up.



I recently spoke with a "quant" about its usefulness in back testing, and feel that it may be a good fit. But that was only one person's opinion.



Is this something that HFC or hedge fund companies would be receptive to? My feeling is that we are a better for back testing than production systems.



Any thoughts/recommendations would be appreciated.



Regards,



David

[spaghetti_o_o]

It depends. For HFT strategies I doubt moving data in and out of machines through the network would bring any advantage given the latency constraints. Even for backtesting I wonder if any HFT strategy requires that much computing power. Most of the time you still need to serially process recorded data, parallel processing is limited.



You might have a better case in data mining and statistical computing (correlation matrix, regressions, monte carlo, etc). Mathlab is junk when it comes to large data sets.



In general the problem I find with vendors is that they are urged to cover as many functionalities as possible, turning a good idea into a mediocre product that fits it all. Do one thing but do it well, knowing all the details and issues, offering as many parameters as possible to tune the product to each customer needs. I guess high quality is not where the big $$$ are in this industry yet.

[dcworthi]

Thank you. We are totally focused on middleware rather than end user financial services applications. So, we are quite specialized within the in-memory data grid space, but not within any particular vertical industry.

[MadMax]

what's the name of your company?

[dcworthi]

Hello. The company is ScaleOut Software.

[intradaybill]

I don't know enough about distributed in-memory computing but your idea about using it with backtesting is an excellent one in the case of system that do that on-the-fly.



Most traders are used to systems that are backtested off-line and then programmed for automation in real-time. In that mode the systems access limited amount of information needed to calculate their indicators.



There is another class of trading systems that need backtesting on the fly. An example from a program I use is an indicator whose value is calculated using backtesting.



http://www.priceactionlab.com/Literature/pIndicator.pdf



In addition, the search and scan of the progam could be used with thousands of securities to run real-time. These function are outlined here:



http://www.priceactionlab.com/index.html



Another one that comes to mind in TeleChart real-time. This program can benefit from this technology by speeding up its real-time calculations. This program does some serious scanning



www.worden.com

[dcworthi]

@intradaybill Interesting post - thank you. I'd be interested in speaking with you further. E-mail me at Davew (at) scaleoutsoftware (dot) com if you are so inclined.