The previous few research-oriented posts (e.g. 1, 2, 3) have been fairly critical of Tikhonov regularization in a specific machine learning application being developed. This post explains the source of the problems with integrating this form of regularization into the algorithm, and demonstrates its successful application.

In previous versions of the modal analysis algorithm performed the following regression:

$$w_{lin} = (X^TX)^{+}X^T y$$

which expands to the form:

$$w_{lin} = ((X^TX)^T(X^TX))^{-1}(X^TX)^T y$$

This is itself a form of regularization, because it protects against the case where $X^TX$ cannot be inverted. The concomitant exchange of bias for stability occurs in this case.

However, when Tikhonov Regularization is integrated into this already regularized regression, it produces:

$$w_{reg} = (X^TX + \alpha I)^{+}X^T y$$

which expands to the form:

$$w_{reg} = ((X^TX)^T(X^TX) + \alpha I)^{-1}(X^TX)^T$$

which provides no improvement to generalization and only degrades accuracy.

If, instead, we perform the regression:

$$w_{reg} = (X^TX + \alpha I)^{-1}X^T y$$

where $\alpha$ is a number very close to zero, we improve the accuracy of the algorithm while still ensuring stability.

Current work involves this regularization technique, and explores the benefits of higher dimensional measurements for retrieving information about the forcing function. Still other current work thoroughly examines various distance measures for eigenvalue estimates in the context of forcing function estimation.