A Guide to Multicollinearity & VIF in Regression

Multicollinearity in regression analysis occurs when two or more predictor variables are highly correlated to each other, such that they do not provide unique or independent information in the regression model. Multicollinearity exists whenever an independent variable is highly correlated with one or more of the other independent variables in a multiple regression equation. Multicollinearity is a problem because it will make the statistical inferences less reliable. However, the Variance Inflation Factor (VIF) can provide information about which variable or variables are redundant, and thus the variables that have a high VIF can be removed. High multicollinearity demonstrates a correlation between multiple independent variables, but it is not as tight as in perfect multicollinearity. Not all data points fall on the regression line, but it still signifies data is too tightly correlated to be used.

What Is Multicollinearity in Regression?

In linear regression analysis, no two variables or predictors can share an exact relationship in any manner. Thus, when multicollinearity occurs, it negatively affects the regression analysis model, and the researchers obtain unreliable results. Therefore, detecting such a phenomenon beforehand saves researchers time and effort. Data-based multicollinearity arises purely from the dataset used, rather than from inherent relationships in the model. It often appears when data collection methods inadvertently create correlations between independent variables.

Effects on coefficient estimates

When using a scatter plot, one plots independent variable values for each data point against one another. If the scatter plot reveals a linear correlation between the chosen variables, then some degree of multicollinearity may be present. This figure illustrates multicollinear data in a scatter plot using the Montgomery et al. delivery dataset example. In some cases, one may use the squared or lagged values of independent variables as new model predictors. Of course, these new predictors will share a high correlation with the independent variables from whence they were derived.10 This is structural multicollinearity.

How can multicollinearity be detected?

• Similarly, trying many different models or estimation procedures (e.g. ordinary least squares, ridge regression, etc.) until finding one that can “deal with” the collinearity creates a forking paths problem.
• In technical analysis, indicators with high multicollinearity have very similar outcomes.
• Here we provide an intuitive introduction to the concept of condition number, but see Brandimarte (2007) for a formal but easy-to-understand introduction.
• Multicollinearity only affects the predictor variables that are correlated with one another.
• Perform an analysis that is designed to account for highly correlated variables such as principal component analysis or partial least squares (PLS) regression.
• Perfect multicollinearity occurs when one independent variable is an exact linear combination of another.

At its core, multicollinearity affects the precision and reliability of regression analysis, making it a significant barrier to predicting outcomes based on multiple variables. One method for detecting whether multicollinearity is a problem is to compute the variance inflation factor, or VIF. This is a measure of how much the standard error of the estimate of the coefficient is inflated due to multicollinearity. From the last column, we can see that the VIF values for height and shoe size are both greater than 5.

More Commonly Misspelled Words

• There is a collinearity situation in the above example since the independent variables directly correlate with the results.
• Multicollinearity can manifest in several forms, each affecting regression analysis differently.
• If that does not work or is unattainable, there are modified regression models that better deal with multicollinearity, such as ridge regression, principal component regression, or partial least squares regression.
• This is due to wider confidence intervals (larger standard errors) that can lower the statistical significance of regression coefficients.
• This doesn’t seem to make sense, considering we would expect players with larger shoe sizes to be taller and thus have a higher max vertical jump.
• To reduce the amount of multicollinearity found in a statistical model, one can remove the specific variables identified as the most collinear.

To address the high collinearity of a dataset, variance inflation factor can be used to identify the collinearity of the predictor variables. In statistics, multicollinearity or collinearity is a situation where the predictors in a regression model are linearly dependent. Instead, the analysis must be based on markedly different indicators to ensure that the market is analyzed from independent analytical viewpoints. For example, momentum and trend indicators share the same data, but they will not be perfectly multicollinear or even demonstrate high multicollinearity. These two indicators have different outcomes based on how the data was manipulated.

A VIF between 1 and 5 generally suggests a moderate level of multicollinearity, while values above 5 may warrant further investigation or corrective measures. It’s important for analysts to consider the context of their specific analysis, as different fields may have different thresholds for acceptable VIF multicollinearity meaning levels. In addition to causing numerical problems, imperfect collinearity makes precise estimation of variables difficult. In other words, highly correlated variables lead to poor estimates and large standard errors.

To illustrate, let’s consider a hypothetical regression analysis aiming to predict real estate prices based on factors like square footage, age of the property, and proximity to the city center. When multicollinearity is present, the precision of the estimated coefficients is reduced, which in turn clouds the interpretative clarity of the model. This section explores the adverse effects of multicollinearity on coefficient estimates and outlines why addressing this issue is essential in data analysis. Of course, this polynomial equation aims to measure and map the correlation between Y and Xn. In an ideal predictive model, none of the independent variables (Xn) are themselves correlated.

Understanding the nuances between perfect, high, structural, and data-based multicollinearity is essential for effectively diagnosing and remedying this condition. Linearly combine the predictor variables in some way, such as adding or subtracting them from one way. By doing so, you can create one new variables that encompasses the information from both variables and you no longer have an issue of multicollinearity. Multicollinearity only affects the predictor variables that are correlated with one another. If you are interested in a predictor variable in the model that doesn’t suffer from multicollinearity, then multicollinearity isn’t a concern.

This often occurs when variables are either redundant or are measuring similar underlying phenomena. For example, in economic studies, indicators like per capita income and poverty rates may be inversely related, and including both in a regression model can lead to multicollinearity. Perfect multicollinearity occurs when one independent variable is an exact linear combination of another. For example, if in a financial model, ‘total assets’ is always the sum of ‘current assets’ and ‘fixed assets,’ then using all three variables in a regression will lead to perfect multicollinearity. This scenario makes it impossible to estimate the regression coefficients uniquely, as the model cannot distinguish the individual contributions of these correlated variables. As mentioned, simple fixes for multicollinearity range from diversifying or enlarging the sample size of training data to removing parameters altogether.