hadoop bleg

astar · Post by **astar** » Fri Dec 03, 2010 10:51 pm

We are considering using it to speed up things like historical backtest, greek calculations on a bunch of books, and so on. I am hoping other phorum members have done something similar and can suggest pointers, possible gotchas; or even alternatives. We are a small shop, and we are still building up things. So we have some operational independence re: installed code base and so on.

thanks in advance

ast4 · Post by **ast4** » Tue Dec 21, 2010 12:25 am

I'm fairly ignorant on the subject, however FWIW I recently went through a round of interviews with a prominent social-networking site in their data analytics departnebt. They did a lot of work in hadoop, and the bulk of their analysis was done through hive and mapreduce.

Those would probably be worth looking into.

Praetorian · Post by **Praetorian** » Tue Dec 21, 2010 1:01 am

I thought about using hadoop on the amazon ec2 clusters for backtesting but then skipped the idea. It totally depends on your application: hadoop is basically fairly usable for parallelization but uses the file system as input and output. That is where your application comes in: If you can store everything in memory (or even lets say 50%) you can build your application memory-centric and will be way faster than going by the hadoop hd i/o.

I would not "blindly" use hadoop only to wear the fancy pants. Totally depends on your needs.

numerical · Post by **numerical** » Tue Dec 21, 2010 3:15 am

Praetorian: last time I did research, ec2 was a pretty terrible choice for large-scale calculations: it was very slow CPU-wise, and had bad IO. I believe many of the alternatives were faster, but I don't remember exactly which. I'd recommend you do a lot of research and testing before ever using ec2 for something non-hosting-related.

Praetorian · Post by **Praetorian** » Tue Dec 21, 2010 4:50 am

yeah i know. Each node there is not state of the art (even if they constantly add new node types - meanwhile with infiniband support). To profit from a handy speedup on ec2 your application must avoid as much communication between nodes as possible. It is useful for settings where you load data once up to each machine, and do some heavy number crunching in memory there and then scale over the number of cpus. To cope with these limitations required quite some reengineering of the backtest.

To decide for or against hadoop (or whatever other technology) one should do this kind of analysis.

MadMax · Post by **MadMax** » Tue Dec 21, 2010 10:16 am

Re EC2:

They offer cluster compute instances and GPU compute instances with 10 Gigabit Ethernet network. If there is demand for infiniband they will offer it. The only issue is if you want to load large amounts of data onto there, then you get hit with slow internet transfer. In that case you have to use their AWS Import/Export service where you ship data on a device.

I would still do some research to check if it suitable for one's application.

Some HPC cloud alternatives: Sgi Cyclone, Penguin On Demand, IBM's Compute On Demand.

astar · Post by **astar** » Thu Dec 23, 2010 4:50 pm

Thanks guys, appreciate it.

Over the next week or so, I will code up a few test cases that used to take me the better part of a day to run and see how it goes. One of the other products that matlab is trying to sell us is their parallelization toolkit and they will give us a test license to try it out as well. I'll try out some of the other "cloud-like" products mentioned.

For the immediate types of backtesting, map-reduce type coding might be pretty decent to work with. Essentially running intraday strategies with the same parameter sets, but over different days, so the parallelization is trivial and there isn't much of a data dependency either. I *was* wondering about the file system based data transfers and am curious how that will play out, tho I am quite optimistic.