The bias — variance tradeoff is one of the foundations of machine learning algorithms, but like most people it eluded me for the longest time. Hopefully this article makes it easy for you to differentiate the two.
What is bias and variance?
Bias essentially occurs when a model is “too simple” or underfitting. It is the difference between our predicted value and true value. The algorithm will continuously miss the mark with predictions no matter how many data points we use. If we were to change the data set multiple times, the fit doesn’t really change drastically.
Variance occurs when a model is “too complicated” or overfitting. In other words, it is the amount that the estimate of the target will change given a different data set. The algorithm is very sensitive to the specific sets of data. The models will mistake noise in the data as signals.
To get good predictions, you need to find a balance between the two that minimizes the total error which makes this tradeoff extremely important for algorithms.
High Bias vs High Variance
A model with high bias — low variance is one that is generally consistent but inaccurate. A high bias algorithm is not very sensitive to changes in the data.
A model with high variance — low bias is one that is typically accurate but inconsistent. A high variance algorithm is very sensitive to even the slightest change in the data.
Algorithms Prone to High Bias
Certain ML algorithms are more susceptible to one error versus the other. High bias models are generally simpler which include linear regression, logistic regression and Naive Bayes models. Some algorithms can be both depending on the parameters. For example, a larger KNN model will be high bias.
Algorithms Prone to High Variance
High variance algorithms tend to be more complicated on the other hand. Decision trees and smaller KNN models are examples of such algorithms. It is worth noting that decisions trees can fall into both error categories too depending on the pruning.
Total Error of Algorithm
Total Error = Bias² + Variance + Irreducible Error
Bias and variance can be reduced by your algorithm choice but irreducible error cannot. There will always be inherent randomness or incomplete feature set.
Summary (Easy way to remember the difference)
Thanks for making it through the article but you could’ve just skipped to the summary.
An easy way to remember high bias vs high variance is a high bias model will always underfit (too general that it can’t find signals) while a high variance model will always overfit (be too specific to that dataset).
