Selecting input vectors in a learner

[astar]

tabris, this is true of the whole problem domain jslade asked about. anything you do wont be automatic and risk of over-fitting is large. What's the best of a set of not-so-good possibilities?

[tabris]

I would disagree that the job can't be "automated" to a certain extent. Lets take the simple example of variable ranking by correlation. This can be done with the least amount of computation time, takes a few loops and correlation calculation, and you have your features selected where the learner can distinguish relevancy/irrelevancy. This opens another can of worms but I am just representing a simple case...



Yes overfitting is a big problem but using CART expose you to locally overfitting in in-sample data/window. There are many ways to correct this if you must use CART but this increases computation time and even then exposes you to other problems. I am just merely pointing out faults of the method. I am actually a big fan of a specialized version of CART (as another NP member would attest to) but just not for feature selection and not at its basic level.

[quantz]

I also find this topic intriguing and perplexing at the same time. particularly with time series data there seem to be almost unlimited nontrivial ways you could TRANSFORM the data to reveal different information. for instance in addition to n lags you could look at changes between ith and jth periods, differences in rates of changes over different periods, etc. this goes into pattern matching etc, which could be totally spurious or highly relevant as in the case of language processing. i would like to read up more on this topic to see what smart people have to say about it ...



it seems to me that as your input space reaches infinite dimension that in addition to curse of dimensionality problems, you are also guaranteed to predictors that are seemingly very accurate despite whatever kind of cross-validation/multiple in-sample out-sample safeguards you put in place. but then these factors (moon phases, astral cycles...) still have no predictive value and are just due to chance... (or are they?!)



so maybe in the end intuition still needs to guide you...

[MrMagoo]

I also think that intuition should play a key role.



All approaches suggested here are valid, and theres always a cost/benefit tradeoff.



If you want a very easy and simple toolbox to select a multiple linear model among many possible variables, take a look at Matlab Stepwise regression toolbox. ( but its only locally optimal ).

[jslade]

Quantz: ya, I've been thinking about this as well. Practically speaking, it is a small number of sensible candidate regressors. Lags and differences in multiples of the forecast horizon seem to work just fine. I might throw some wavelet shyeah at this problem as well, but again, the forecast horizon kind of defines what might be good here. I'm not looking for the world's most awesomest forecaster: I just want the forecast to make some sense and have a similar spectrum to the actuals (the learner takes care of that).

The real problem for me is, the generating process is very different in the class of gizmos generating my time series (of which there are 10^6 or so). Only some are financial; others are more like monkeys flipping a light switch in their houses.