Datamining and the hunt for returns

[pj]

you, kids!

Big Smile

[braincell]

@artemiso



thanks for the post, and I don't think you said anything you should apologize for either. I appreciate your answer, and it did help me focus on a few elements. The myths you talk about, at least I feel, need to be dispelled on a per-user basis, in other words I had to work on the few concepts a little more to believe what you were saying. My question(s) was a little broad and unfocused, but each time somebody like me attempts to do just that (ie understand it in context of my work) it might require a slightly nuanced approach which causes me to think - maybe I'm doing something slightly different, so it's worth a shot. Obviously, thus far I have reached some of the same conclusions as you did, but I still have ways to go, simply to convince myself. Would I be the first to do "research" I wholeheartedly expect to fail just to convince myself? I think not. I've found dissuading people against wasting time equally difficult (and self-destructive for them) on subjects I've already tried, but it's a very useful learning mechanism for them.



Also the first 3 items you implied I was saying, I didn't say or think. Especially not the R-R philosophy, I just used that term to illustrate positive skew.



Either way, I think my point is: even what Steve is posting here requires more detail to be of any use for us to give advice. I find that these subjects very quickly become demanding in terms of the details needed to describe them. Just a slightly different application of any concept can mean a huge difference between 'useful' and 'useless', so unless we all start writing 11-page references, posting questions about broad subjects is very unproductive. I see some of the members post straight questions and get straight answers, that's because they have more experience in communicating these things, and I (and Steve perhaps) should probably learn that too.



So perhaps we should ask Steve for a little more detail and then tell him the things he needs to know to most quickly realise it's all not going to work, but without telling him it's not going to work.



So Steve, any more details on what'cha trying to do there? Blue vs Red suggests two concepts, 2 data sets with 4 subcategories?

[Steve Castle]

right, the colors are screwing this up. Actually this is an artifact of trying to plot multipe signals and distributing then over the spectrum evenly, and matplotlib dropping the yellows and greens because they are all nan.



3 were dropped due to happening while the asset's market was closed, so my amatureish program set them to nan instead of filtering them.



so i take away two points:

a) clean my data better

b) make sure the resulting plot doesn't look like there are artificial groups.



both useful

[braincell]

Glad I could help! 0_o



I hope you're looking at the order book at some depth, if you have the data, for those time-frames trades (and lvl1) don't tell you much.

[astar]

pj,

Too bad that googling Lorfing-Klett does not return just one result.

[jslade]

Kantorovich refers to the convergence properties of Newton's method. I figure artemiso was making a bit of a joke; I didn't need Kantorovich to tell me when Newton's method converges.

The other one he refers to is probably Ralf Klett's ideas, with some other guy's name pegged to it.

[artemiso]

Yeah I saw ephedyn make that up somewhere else lol. Lorfing-Klett and Kantorovich are just fictitious names. Big Smile



@braincell



Good, I'm happy to see that you took this in the right direction and I'm glad you don't think I was trolling you.

[jslade]

Big Smile



Serves me right for showing off my google skills. A textbook case of overfitting to noise, which brings us back to the original topic.

[tbrown122387]

so are you trying to approximate the density of returns given all six factors, or are you trying to come up with an additive model.



if its the second one, and you're using simple technical indicators, then weights might not be a good idea (relationship isnt linear).



if the second one: what's that... like.. decision trees? im not too up on the whats what of machine learning techniques.

[Steve Castle]

I was trying do more the second, help me wade through a bunch of results and find interesting ones. Jslade doesn't approve .



This is pure research, it's not going to be traded on. It's L1 data, and if anyone was interested, they'd (ideally) give us the L2 data to continue the research. We're trying to bootstrap our way to more free data, we got the signal and L1 data for free by proposing this study. But all your comments are helpful, and I appreciate the criticisms, as mentioned, this is new territory for me. I'm mostly doing the technical pieces.