Generalized Linear Regression for Multiple Targets

Building on generalized linear regression, this lesson introduces techniques for simultaneously modeling several output variables. Previously, the optimal parameters were a vector ($w$); now, the optimal parameters are represented by a weight matrix ($W$). Each input vector $x_i \in R^d$ now corresponds to a target vector $y_i \in R^m$, where $m$ is the number of outcomes we are predicting. The model uses the form $f_{W}(x) = W^T\phi(x)$, where $\phi(x)$ still allows for non-linear feature transformation. This lesson focuses on formulating this multi-output prediction task as a single, efficient matrix minimization problem, often termed multi-target Ridge Regression, to derive the corresponding closed-form solution.

Multiple targets

Consider a regression dataset $D=\{(x_1,y_1),(x_2, y_2),\dots,(x_n,y_n)\}$, where $x_i \in R^d$ and $y_i \in R^m$. A function $f_W(x) = W^T\phi(x)$ is a generalized linear model for regression for any given mapping $\phi$ of the input features $x$, and $W$ is a matrix with $m$ columns, one for each target. Note that $f_W(x) \in \R^m$, meaning the model produces a vector of $m$ predicted values for each input $x$, with each component corresponding to one of the multiple target variables. This allows the generalized linear model to simultaneously predict all target outputs in a single evaluation.

The optimal parameters $W^*$ can be determined by minimizing a regularized squared loss (multi-target ridge regression) as follows:

$$W^*=\argmin_{W}\bigg\{\sum_{i=1}^n \|W^T\phi(\bold x_i)-\bold y_i\|_2^2 + \lambda \|W\|_F^2\bigg\}$$

Here, $\sum_{i=1}^n \|W^T\phi(x_i)-y_i\|_2^2$ ...