Mutually orthogonal approximations?

[lexx]

nikol and kuebiko - many thanks for the interesting discussion, now see the problem from different perspective

[ronin]

> Solution definitely exists, since I can regress one against the other and use residual,

> but it's not exactly the same as above





I think it is the same as above. Wlog set X*=0.



Then your problem is min X^2 + (Y*-Y)^2 subject to X.Y=0. It's just an orthogonal projection.

[frolloos]

Why can't you use Gram-Schmidt orthogonalisation? I am likely missing something here, but if the hard constraint is that they are orthogonal then you can use Gram-Schmidt, can't you? Whether the mean square error is minimised I don't know, and I am not sure you can do both at once.



https://en.wikipedia.org/wiki/Gram–Schmidt_process

[frolloos]

dbl post.

[nikol]

the problem with G-S is that u1=v1, i.e. first (arbitrary) vector, v1, always goes alongside with u1 generated orthogonal set and, therefore, u1-v1 = 0.



as far as I understood the initial problem, lexx wants {u}'s to be equidistant from {v}'s.

[kuebiko]

So it’s definitely bi-convex, meaning you can fix y and solve for x, then fix x and solve for y, and repeat in alternating fashion until convergence. But in general bi-convexity doesn’t guarantee a global optimum. So you’d have to play around with it and see if it’s sensitive to your choice of initialization. This is a simple problem so somebody who knows more about this may know whether or not there’s a global solution here.

[ETwode]

I could be wrong but I think it isn't quite as simple as orthogonal projection. Consider the case in 2d where X* and Y* are the standard basis vectors rotated inward toward each other by the same angle. We expect that the solution is to symmetrically push them both back outward (projecting onto the axes), and can show that eg. regressing X* on Y* and taking X to be the residual gives larger error than this. Every case where ||X*|| = ||Y*|| reduces to this situation.



Manipulating the KKT conditions suggests that the general solution is:

X = (1-u^2)^-1 [X* - uY*], Y = (1-u^2)^-1 [Y* - uX*], with

u = (s - sqrt(s^2 - 4r^2)) / 2r, s = ||X*||^2 + ||Y*||^2, r = X*'Y*

which passes some quick/basic sanity checks, but I haven't convinced myself the reasoning is 100% correct.

[lexx]

Wow! thanks for the discussion!

Didn't expect it



Wanted to share initial motivation for the problem :



I have 2 factors - equity characteristics, say value and quality.

And want to combine them together, based on how they perform.

They often do become highly correlated, so better results I could get if to

orthogonalize them - which I can do by regressing one against the other and using residuals instead of the original factor. But results depend on which one is regressed

and which is kept as is (tried different factors). Residualized factor is dominated by the one kept as-is (ETwode's point regarding errors)

So the posed problem would avoid that, and be "fair" to both factors - nikol's point of equi-distance.

G-S procedure I believe creates orthogonal basis consisting of linear combinations of the vectors, if i'm not mistaken. I would like to avoid that, since interpretation becomes difficult.

Min their respective errors, somewhat preserves the "nature" (to some degree) of approximating factors.