A Different Way to Think About Belief
Most of what we've done so far has been frequentist statistics: we treat unknown quantities as fixed but unknown, collect data, and ask how surprising the data would be under some hypothesis. P-values, confidence intervals, hypothesis tests — all frequentist.
Bayesian statistics flips the question. Instead of "given the hypothesis, how surprising is the data?", it asks "given the data, how should I update my belief about the hypothesis?" It starts with a prior belief, observes data, and produces a posterior belief that combines both.
This is a brief introduction — full Bayesian methods can fill a whole course. But you should understand the basics for two reasons. First, Bayesian thinking shows up throughout ML: probabilistic models, Bayesian neural networks, uncertainty quantification. Second, it's a common interview topic in data science roles!
Bayesian in ML Practice
You'll meet Bayesian thinking in ML in several places:
- Bayesian optimization for hyperparameter tuning — maintains a probabilistic model of the objective function and updates it as trials come in.
- Probabilistic classifiers like Naive Bayes.
- Bayesian neural networks — model uncertainty in weights.
- Thompson sampling in reinforcement learning and multi-armed bandits.
- Bayesian A/B testing — reports the full posterior distribution over the lift, rather than a binary reject/fail-to-reject decision.