Stanley L. Sclove

By Stanley L. Sclove

The problem of choosing the number of PCs to retain is analyzed in the context of model selection, using so-called model selection criteria (MSCs). For a prespecified set of models, indexed by k=1,2,…,K, these model selection criteria (MSCs) take the form MSC_k=_nLL_k+a_nm_k, where, for model k, LL_k is the maximum log likelihood, m_k is the number of independent parameters, and the constant a_n is a_n=lnn for BIC and a_n=2 for AIC. The maximum log likelihood LL_k is achieved by using the maximum likelihood estimates (MLEs) of the parameters. In Gaussian models, LL_k involves the logarithm of the mean squared error (MSE). The main contribution of this chapter is to show how to best use BIC to choose the number of PCs, and to compare these results to ad hoc procedures that have been used. Findings include the following. These are stated as they apply to the eigenvalues of the correlation matrix, which are between 0 and p and have an average of 1. For considering an additional PC_k + 1, with AIC, inclusion of the additional PC_k + 1 is justified if the corresponding eigenvalue λ_k+1 is greater than exp(−2/n). For BIC, the inclusion of an additional PC_k + 1 is justified if λ_k+1 > n^1/n, which tends to 1 for large n. Therefore, this is in approximate agreement with the average eigenvalue rule for correlation matrices, stating that one should retain dimensions with eigenvalues larger than 1.

Part of the book: Advances in Principal Component Analysis