Addressing parameter instability when using rolling WLS?

[haakhtar]

So I've been working on a model using WLS and one of the problems I run in to is unstable estimates of the regression parameters. I was wondering if anyone had any idea about ways to mitigate this?

[gaj]

A few possibilities off the top of my head, 1) there's not enough data, 2) the features are bad, 3) you have highly correlated features.



1 and 2 are kind of related. If you have bad features but you have a lot of data, the regression should give zero weights to those features. You can check the p-values and confidence intervals of the regression parameters to diagnose this problem. You should get rid of features that have no predictive power. You can also use shrinkage methods like ridge regression to make the coefficients more conservative and hence more stable.



For 3, if you have several features that are very highly correlated, the coefficients will blow up. The ideal solution is to construct the features to be uncorrelated in the first place. Sometimes this is not possible, so you could try some dimension reduction techniques like PCA.

[bullero]

Most likely, you have a near singular correlation matrix.



notes: http://www.stat.cmu.edu/~larry/=stat401/lecture-17.pdf

[haakhtar]

Thanks for the info bullero & gaj, I'll do some more reading according to your suggestions.