On 2021-07-15 04:53:29, user Derek Beaton wrote:
Overview of “On stability of Canonical Correlation Analysis and Partial Least Squares with application to brain-behavior associations”
Derek Beaton, PhD<br />
Director, Advanced Analytics <br />
Data Science & Advanced Analytics (DSAA)<br />
St. Michael’s Hospital, Unity Health Toronto
This manuscript provides an in-depth look at reliability and stability of CCA and PLS through the use of a generative modelling approach with synthetic data (and their software gemmr), and subsequently show CCA and PLS applied to large and modern brain-behavior data sets (HCP, UKBB). The manuscript also provides multiple perspectives: (1) assessment of brain-behavior CCA & PLS when sample sizes change for the number of features, (2) a meta-analysis/review of brain-behavior CCA studies, and (3) tools, suggestions, and advice on how to approach interpretation of CCA & PLS-based studies for brain-behavior neuroimaging studies. There is a substantial amount of work and the contributions of the manuscript are quite valuable. Overall I think this is a strong manuscript and there are many good things about this paper and the software.
However I focus my review on my concerns. I think if some of these are clarified or responded to, then the paper would possibly be stronger and clearer. Below I first bullet point my primary concerns with the manuscript, and how those concerns relate to the overall conclusions and generalizability of the work. Following that, I provide my other concerns generally in order of appearance in the manuscript.
My first major concern is that the manuscript generally reads as potential limitations of CCA and PLS. However, only these two methods are discussed and, I believe, that the core issues of stability (and generalizability, replicability, etc…) in neuroimaging are because of (1) small samples, and (2) noisy measurements. So are the issues presented exclusive to CCA/PLS? Or should we expect to see the same effects in other techniques (e.g., standard GLMs, multivariate regressions, statistical/machine learning approaches such as SVM or random forests)?
While comparing CCA and PLS is (very, very) useful for many fields, especially neuroimaging, I believe that some of the comparisons here are in effect unfair. In particular, CCA doesn’t really work without extra preprocessing to data when those data have more variables than samples. CCA effectively requires us to reduce the dimensionality of data so that we have more samples than variables or to allow us to invert X’X and/or Y’Y. However, PLS does not require additional preprocessing in order to work (correctly). The pipelines for the data were designed around the limitations of CCA but applied to both PLS and CCA. How does PLS perform when these extra steps are not taken? Effectively, how does PLS vs. PLS with CCA-friendly data vs. CCA compare? Though I comment on it more later, I believe that the observed “bias towards the first principal components” in the PLS results may be due to this.
Taken together, I think the general conclusion to take away from the manuscript is that these are the behaviors and limitations of CCA/PLS under these specific conditions, but not necessarily any condition. I expand on this in additional comments and provide some references throughout.
Abstract:
You’ve noted that the “Application of CCA/PLS to high-dimensional datasets raises critical questions about reliability and interpretability”. Perhaps a small but important distinction here is that these techniques provide a lot of things to interpret, but comparatively are relatively easy to interpret (they are interpreted like PCA). I think there should be a de-emphasis of interpretability and most of the emphasis on reliability and stability. To note: these techniques are still easy to interpret even when results are not reliable (which is, perhaps, a drawback of their use).
I apologize for the following comment as it will be repeated a few more times, but I believe that “For PLS [there is a] bias toward leading principal component axes.” is more likely an artifact of how the data were prepared for use in PLS and not strictly a drawback of PLS. If both X and Y data sets are principal components (which include their subsequently decreasing variance), then PLS will (correctly) pick up on those “variables” (components). This is particularly true if/when data submitted to PLS are not normed or scaled in some way (which principal components are likely not, as that destroys the inherent variance in the principal components).
Introduction:
I think “the dominant latent patterns of association linking individual variation in behavioral features to variation in neural features” would be better rephrased as “the dominant common latent patterns shared between behavioral and neural features”. Or something along these lines as it’s a bit clearer and doesn’t emphasis linking one thing to another thing (as this sounds a bit directional, where CCA and this flavor of PLS is symmetric)
When you say “[...] a number of open challenges exist regarding [CCA/PLS] stability in characteristic regimes of dataset properties”, I wonder if it’s more appropriate to also discuss the open challenges of the data themselves. Noisy instruments and measurements are difficult to analyze with most approaches, and this isn’t a problem for just CCA and PLS. In effect, do we have data that are stable and reliable?
I find the mixtures of terminology difficult to follow. Could you provide a clearer set of definitions for terminology, and then stick specifically to certain terms? You’ve mentioned both the SVD and eigendecompositions. It might make things clearer to connect CCA & PLS terminology directly to SVD/eigen results, and just use those terms instead. For one particularly confusing example: “weights”. I’m not sure what “weights” are to mean here, especially because “weights” has so many meanings in stats/machine learning.
I think the discussions of stability rely too heavily on relatively older literature (e.g., references 10-12) which are also generally from other domains. The same points from those are likely still true (or even more so in larger and noisier data) but I think more modern works that directly discuss high dimensional problems would be helpful. Furthermore, these generally discuss CCA and not PLS. So additional literature on PLS here would be good.
For reference 13, the manuscript says “cross-validated association strengths that are markedly lower than in-sample estimates”. Isn’t that expected based on this (and other) work? Should we not expect the smaller sample sizes (e.g., folds) to produce lower (or less stable) estimates?
To echo a previous point: most of the literature discussing (in)stability is for CCA and not PLS. This should be clarified or further supported.
Though this work is important and well done, I don’t think it’s fair to say “to our knowledge, no framework exists [...]”. There has been a lot of work on the systematic assessment of these techniques, and the SVD/eigen in general. Could you clarify this a bit more? Or instead show that this is an additional element in our understanding of CCA/PLS behaviors? The field of chemometrics in particular has an extensive literature on the stability of PLS (although typically the regression flavor, not the PLSC flavor here).
I think this is misleading and possibly incorrect: “CCA and PLS differed in their dependences and robustness, in part due to PLS exhibiting a detrimental bias of weights toward principal axes”. PLS may exhibit this behavior under these data processing conditions (which are required for CCA, but not for PLS).
Another repeated point: the manuscript says that “typical CCA/PLS studies in neuroimaging are prone to instability”. Is this because of CCA/PLS? Are other techniques also unstable? Is this because of the data?
Results:
“Number of features” as the additive number between X and Y is strange, because each set has a different number of features. And the sizes of X and Y (as well as their internal covariance structures) can have substantial influence on the results. For example, if X were only 1 or 2 (strongly correlated) measures and Y had many 100s or 1000s of measures, then the (joint) solution is fairly limited and (to a degree) constrained by X.
The finding of the “average of the cross-validated and in-sample” results struck me, especially given that the bootstrapped results didn’t converge to the expected estimate (but the previous average did). I didn’t expect this, but I think it’s a positive finding. Could you provide more details on these procedures, and could you possibly explain these behaviors/findings in more detail?
Why are you quantifying error as the greater of the two errors (X and Y) from their true weights? Why not present them separately? That would tell us if/how CCA/PLS can estimate one set but perhaps not the other.
I don’t follow what the authors did to get around the sign-flips in the results. The manuscript says “it is chosen to obtain a positive between-set correlation”, but I’m not sure what this means here.
To repeat a previous point about terminology: the term “loadings” has many meanings, too. Here it seems the authors used the correlation between datasets and scores, correct? These correlation loadings are one type of loading, where, say, the singular/eigen vectors are another type of loading.
Why switch between Spearman and Pearson correlations for the distance estimate for the various scores? Why not both in both cases or choosing one?
I find Figure 3---in particular panels A and B---unclear. First, it’s not entirely clear to me what “weights” and “feature id” convey here. Figure 3B seems to show that PLS weights are spherical. This is not what I would expect from PLS. Could you explain these results in more detail?
A reiterated point: The description of what it means for PLS to converge to “the first principal component” is unclear. The first principal component of what? There are two data sets (X, Y) that are sets of PCs (if I am understanding correctly).
I think the permutation tests may be too conservative and/or incorrect (as described in CCA/PLS analysis of empirical data). While it is typical to permute just the rows of one matrix vs. the other, this is potentially problematic for CCA/PLS. That’s because each X & Y has an internal covariance structure. If at least one of those structures is strong, then the results will resemble the strong internal structure. This is particularly true when, for example, behavioral data are already very correlated. So a more appropriate permutation may be within each column of the data matrices. However, this is only appropriate in the original data matrices. Permutation should not be done on the PC scores (I am presuming that was the case, but please correct me if I am wrong).
For the line that starts with “After modality-specific preprocessing (see Methods)”, I will reiterate and expand on one of my sticking points. CCA requires invertible or rank reduced matrices when there are too many variables but PLS does not. So to reduce specifically to 100 PCs is a limitation of CCA. PLS does not require this. How would the results change if PLS were run directly on the data? Furthermore, 100 principal components is not informative nor a meaningful choice. How many total components were there? How much variance did 100 components explain? Could just 10 or 20 components explain almost as much variance as 100? For analyses based on PCs, it is important to select based on something meaningful: that could be explained variance or by performing tests on the PCs themselves for selection. Though almost any approach is somewhat arbitrary, to select 100 is seemingly unmotivated or unguided.
In Figure 4, how are you computing 95% CIs from permutations? Permuations are for null distributions, not distributions around the effects (CIs). I would expect other resampling approaches (e.g., bootstrap) to provide CIs.
By the time I get to Figure 4, I’m wondering why are the CCA and PLS results not directly compared? As in, why not present, for examples, correlations or other similarities between the CCA & PLS results? I think it would be important to directly quantify the similarity between CCA & PLS results.
Later in the manuscript, you indicate that you “considered reducing the data to different numbers of principal components than 100.” While this is certainly a benefit, the description of the results is unclear. You indicate that “Retaining more than 10 behavioral PCs lead to marginal increases [...]”. But 10 here is not informative. How much variance was explained by those 10? By the 100? How much is explained by 1 PC? The total number of PCs is not particularly informative, rather, the amount of (cumulative) explained variance, the number of retained components, and the total number of possible components makes for something more informative.
Discussion:
The authors mention that CCA is (more) attractive (than PLS) because it’s scale invariant, which is nice when measures are not commensurate. However, when data are normalized or scaled (e.g., z-scored), then data are commensurate. Did you use normed or scaled data for PLS? How would that change the conclusions about commensurate scales and CCA’s scale invariance?
You mention in limitations that you “assume data are described in a PC basis” and then you “expect that a dataset whose features have been rotated into a new coordinate system by an orthogonal transformation matrix to have the same sample size requirements as the untransformed dataset.” In this particular case for PLS: you don’t need to assume that. You can run the same pipelines you have with the untransformed data to see how CCA vs. PLS vs. (untransformed) PLS compare. This would provide a very interesting case regardless of the results (whether the sample size requirements are the same or different).
You say that the generative model points out the pitfalls of CCA and PLS. Could you also apply this generative approach to other techniques, even simple linear models? Do the pitfalls also exist there? Are these pitfalls of the methods, or are these pitfalls reflective of the kinds of data we analyze?
You note that there are regularized versions of CCA and PLS to “mitigate the problem of small sample sizes”. I have two issues (one small, one a bit bigger) with this statement. Regularized (and penalized, and sparsified, etc…) methods are not necessarily designed to allow for small sample sizes. Rather they help with mitigating overfitting (which sometimes could be due to too small of sample). My second issue is that the line between CCA and PLS becomes especially blurred, and even disappears, when it comes to regularized techniques. In particular, we should look to Witten et al.’s penalized approach for CCA. Witten et al., note that “[in] high dimensional problems, treating the covariance matrix as diagonal can yield good results” where they reframe their CCA equation (4.2) and in a different way, where their “penalized CCA criterion, [they] substitute in the identity matrix” for X’X and Y’Y in their equation 4.3. Witten et al., then further note that their CCA “is simply [eq. 2.7] with X replaced with X'Y”. That means that when it comes to penalized CCAs, most drift towards or even start out as PLS. This can make any suggestions as to which is better (CCA or PLS) moot as in the penalized approaches, they are effectively much closer to one another than in the standard approaches. (Furthermore, using a subset of PCs for each data set is, effectively, a soft form of regularization.)
Though brief, I think you’ve placed too much emphasis on PLS regression as being “conceptually different from PLSC/PLS-SVD” because in virtually all implementations of PLS regression, the first component/latent variable is identical to PLSC’s first component/latent variable. This is because both approaches model X’Y and (in most cases) use the SVD to do so. It’s just that PLSC is one pass of the SVD (so effectively a PCA of X’Y) where as PLSR is iterative, deflates X and Y in each iteration, and (asymmetrically) emphasizes certain properties for X (e.g., orthogonal latent variables for X, but not necessarily Y).
Methods:
The approach to the behavioral data is not particularly realistic when it comes to studies, is it? In most cases some form of imputation would be used and the behavioral data in particular would be directly used, not a projection (PCs) of the data. Would the behavioral PCs change substantially in your pipeline if you were to impute instead of using the method you did?
References and literature:
Below I provide some references and literature to supplement some of my points and to help strengthen some of the points you’ve made in the paper. Please note that some are mine. I’m not providing my (or the other) citations because I want them to be or am expecting them to be cited, rather these are for reference. Furthermore, these articles also provide quite a bit of citations that are worth looking into.
These two articles provide more unified perspectives on PLS, CCA, and many related techniques. The Borga et al., article is quite a good one. I provide my article moreso for the supplemental materials (https://www.biorxiv.org/con... "https://www.biorxiv.org/content/10.1101/598888v3.supplementary-material)"). In my supplemental materials, I further unify and generalize more approaches like the Borga article. Both of these show (at least algebraically) that these techniques can be thought of as variations of one another, and in some cases not very different.
Borga, M., Landelius, T., & Knutsson, H. (1997). A unified approach to pca, pls, mlr and cca. Linköping University, Department of Electrical Engineering.
Beaton, D., Saporta, G., & Abdi, H. (2019). A generalization of partial least squares regression and correspondence analysis for categorical and mixed data: An application with the ADNI data. bioRxiv, 598888.
To further emphasize why CCA/PLS can be very different or very similar, please see another one of my articles (see below). Like above, most of this article can just be skipped. Starting in section 4 on Page 22, I show CCA, PLS, and reduced rank regression (RRR) because they are all variants of one another. In Figure 5 the data are centered and scaled, and each technique produces comparable results. In Figure 7, however, the data are only centered and produce different results. This highlights that when norming/scaling, CCA and PLS can in fact be more similar than different:
Beaton, D. (2020). Generalized eigen, singular value, and partial least squares decompositions: The GSVD package. arXiv preprint arXiv:2010.14734.
Some recent work has been published to show what happens to results when sample sizes are small and as sample sizes change:
Grady, C. L., Rieck, J. R., Nichol, D., Rodrigue, K. M., & Kennedy, K. M. (2021). Influence of sample size and analytic approach on stability and interpretation of brain-behavior correlations in task-related fMRI data. Human brain mapping, 42(1), 204-219.
The above article is an interesting companion to yours because it shows that there is an advantage to multivariate over univariate techniques because multivariate approaches provide consistent (stable) results. However, Grady et al., concluded that small samples wouldn’t be sufficient to get reliable results, regardless of approach.
These would be more suitable PLS articles to reference, especially for neuroimaging:
Krishnan, A., Williams, L.J., McIntosh, A.R., & Abdi, H. (2011). Partial Least Squares (PLS) methods for neuroimaging: A tutorial and review. NeuroImage, 56, 455-475.
Abdi, H. (2010). Partial least square regression, projection on latent structure regression, PLS-Regression. Wiley Interdisciplinary Reviews: Computational Statistics, 2, 97-106.
McIntosh, A. R., & Mišic, B. (2013). Multivariate statistical analyses for neuroimaging data. Annual review of psychology, 64, 499-525.
McIntosh, A. R., & Lobaugh, N. J. (2004). Partial least squares analysis of neuroimaging data: applications and advances. Neuroimage, 23, S250-S263.
McIntosh, A. R., Bookstein, F. L., Haxby, J. V., & Grady, C. L. (1996). Spatial pattern analysis of functional brain images using partial least squares. Neuroimage, 3(3), 143-157.
Additional PLS & CCA articles:
Gatius, F., Miralbés, C., David, C., & Puy, J. (2017). Comparison of CCA and PLS to explore and model NIR data. Chemometrics and Intelligent Laboratory Systems, 164, 76-82.
Goodhue, D. L., Lewis, W., & Thompson, R. (2012). Does PLS have advantages for small sample size or non-normal data?. MIS quarterly, 981-1001.
To determine the number of PCs especially when detecting the space to interpret (which applies to PLS and CCA):
Peres-Neto, P. R., Jackson, D. A., & Somers, K. M. (2005). How many principal components? Stopping rules for determining the number of non-trivial axes revisited. Computational Statistics & Data Analysis, 49(4), 974-997.