STATISTICS AND PROBABILITY FOR MACHINE LEARNING
Statistics helps us describe data, measure variation, and quantify relationships. Probability gives us a framework for reasoning about uncertainty. Together, they form an important mathematical foundation for machine learning.
STATISTICS AND PROBABILITY
FOR MACHINE LEARNING
The Mathematical Foundation of ML
Statistics helps us describe data, measure variation, and quantify relationships. Probability gives us a framework for reasoning about uncertainty. Together, they form an important mathematical foundation for machine learning.
Learning Objectives
- Understand population, sample, variable, and observation.
- Calculate and interpret mean, median, mode, weighted mean, variance, and standard deviation.
- Use percentiles, quartiles, and the interquartile range to understand data distribution.
- Understand probability, conditional probability, independence, and Bayes' theorem.
- Recognize common discrete and continuous probability distributions.
- Understand expectation, sampling, the Central Limit Theorem, covariance, and correlation.
- Understand the basic purpose of hypothesis testing and confidence intervals.
- Connect statistical concepts to exploratory data analysis, feature engineering, model evaluation, and ML systems.
1. Why Statistics and Probability Matter in ML
Machine learning rarely operates on perfectly predictable data. Customer behavior, sensor measurements, transactions, images, text, and operational metrics contain noise and variation. Statistics helps us summarize and investigate that variation, while probability helps us model uncertainty and reason about possible outcomes.
2. Population, Sample and Observation
A population is the complete group we want to understand. A sample is a subset collected from that population. An observation is one individual record or measurement.
3. Types of Data
4. Measures of Central Tendency
Mean
Mean is the arithmetic average:
x̄ = (x₁ + x₂ + ... + xₙ) / n
The mean uses every value, so it can be strongly affected by extreme observations.
Median
Median is the middle value after sorting the observations. For an even number of observations, it is the average of the two middle values. It is often more robust than the mean when a feature is skewed.
Mode
Mode is the most frequently occurring value. It can be useful for categorical variables as well as discrete numerical data.
5. Weighted Mean
A weighted mean assigns different importance to observations. It is useful when observations do not contribute equally.
Weighted mean = Σ(wᵢxᵢ) / Σwᵢ
Example: If model performance is evaluated across datasets of different sizes, weighting each dataset equally may not represent the overall population as well as weighting by the number of observations.
6. Measures of Dispersion
Central tendency alone is not enough. Two datasets can have the same mean but very different spread.
7. Variance and Standard Deviation
Population variance:
σ² = (1/n) Σ(xᵢ − μ)²
Sample variance:
s² = (1/(n−1)) Σ(xᵢ − x̄)²
The sample formula uses n−1 in the denominator because it provides an unbiased estimator of population variance under the standard random-sampling framework.
Standard deviation is the square root of variance:
σ = √σ² or s = √s²
8. Percentiles and Quartiles
A percentile indicates a position in an ordered dataset. The 50th percentile corresponds to the median. Quartiles divide ordered data into four parts: Q1, Q2 (median), and Q3.
Interquartile Range (IQR) = Q3 − Q1
IQR is useful for understanding the spread of the central portion of a dataset and is commonly used in outlier analysis.
9. Probability Fundamentals
Experiment: a process that produces an outcome.
Sample space: the set of possible outcomes.
Event: a subset of outcomes of interest.
Probability: a value from 0 to 1 representing the likelihood of an event under a specified probability model.
10. Basic Probability Rules
Complement: P(not A) = 1 − P(A)
Addition rule: P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
Multiplication rule: P(A ∩ B) = P(A)P(B|A)
If A and B are independent, P(B|A) = P(B), so P(A ∩ B) = P(A)P(B).
11. Conditional Probability
Conditional probability describes the probability of A given that B has occurred:
P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0.
This concept is central to probabilistic ML because the information available to the model can change the probability assigned to an outcome.
12. Independence
Two events are independent when knowledge that one occurred does not change the probability of the other. Independence is different from being mutually exclusive: mutually exclusive events with positive probability cannot occur together, while independent events can occur together.
13. Bayes' Theorem
Bayes' theorem connects conditional probabilities:
P(A|B) = [P(B|A)P(A)] / P(B)
In classification and diagnosis-style problems, Bayes' theorem highlights why base rates matter: a strong-looking signal does not by itself determine the posterior probability.
14. Random Variables
15. Common Probability Distributions
16. Normal Distribution
The normal distribution is continuous and symmetric around its mean. Its shape is determined by its mean and standard deviation.
Standardization (z-score): z = (x − μ) / σ
A z-score expresses how far a value is from the mean in standard-deviation units. Not every real-world ML feature is normally distributed, so normality should be checked rather than assumed.
17. Expectation
Expectation is the probability-weighted average of a random variable's possible values. For a discrete random variable:
E[X] = Σ x P(X = x)
Expectation is useful when reasoning about average outcomes, expected loss, and probabilistic models.
18. Sampling
Sampling bias occurs when the sampling process systematically produces a sample that does not adequately represent the target population. Selection bias can affect both statistical conclusions and ML model performance.
19. Central Limit Theorem — Practical View
Under appropriate conditions, the sampling distribution of a suitably normalized sample mean approaches a normal distribution as sample size grows. This result supports many statistical procedures.
Important: the Central Limit Theorem does not say that every raw dataset becomes normally distributed.
20. Covariance
Covariance describes how two variables vary together. Positive covariance indicates that higher values of one variable tend to be associated with higher values of the other under the observed data; negative covariance indicates the opposite tendency.
Its magnitude depends on the units of the variables, which makes covariance harder to compare across differently scaled features.
21. Correlation
Pearson correlation standardizes linear association to a range from −1 to +1.
Correlation does not establish causation. A low Pearson correlation also does not prove that two variables have no relationship; the relationship could be nonlinear or otherwise structured.
22. Statistical Significance and Hypothesis Testing
Null hypothesis (H₀): a reference claim used for the test.
Alternative hypothesis (H₁): a competing claim.
Test statistic: a quantity calculated from the observed data.
p-value: under the null model, the probability of observing a result at least as extreme as the one obtained, according to the test definition.
Significance level (α): a preselected threshold used to define a rejection rule.
A p-value is not the probability that the null hypothesis is true, and a small p-value alone does not measure practical importance.
23. Confidence Intervals
A confidence interval provides a range produced by a statistical procedure to communicate uncertainty around an estimate. In the frequentist repeated-sampling interpretation, a 95% confidence procedure produces intervals that contain the true parameter in 95% of repeated samples under the stated assumptions.
Confidence intervals should be interpreted together with the sampling design, assumptions, and practical context.
24. Statistics in Machine Learning
25. Example: Customer Churn
Suppose a company wants to predict customer churn. Statistics can be used before model training to understand the dataset:
- Compare mean and median monthly spend.
- Measure standard deviation and percentiles to understand customer variability.
- Compare churn rates across customer segments.
- Inspect relationships among tenure, spend, support calls, and churn.
- Check class imbalance before selecting evaluation metrics.
- Use appropriate train/validation/test splits to estimate generalization.
- Interpret model metrics with uncertainty and business context.
26. Python Example
import numpy as np
import pandas as pd
values = np.array([10, 12, 15, 18, 20])
print("Mean:", values.mean())
print("Median:", np.median(values))
print("Standard deviation:", values.std())
df = pd.DataFrame({
"age": [25, 31, 42, 28, 36],
"income": [40000, 55000, 90000, 48000, 70000]
})
print(df.describe())
print(df.corr(numeric_only=True))
27. Common Mistakes
- Confusing population parameters with sample statistics.
- Using the mean without checking extreme values and skew.
- Assuming every feature follows a normal distribution.
- Interpreting correlation as causation.
- Ignoring base rates when interpreting probabilities.
- Treating a p-value as the probability that the null hypothesis is true.
- Ignoring uncertainty around estimates.
- Applying statistical tests without considering their assumptions.
- Using biased samples and expecting unbiased conclusions.
- Choosing an evaluation metric without considering class balance and the real objective.
28. Interview Questions
- What is the difference between a population and a sample?
- What is the difference between a parameter and a statistic?
- When would you prefer the median over the mean?
- How does an outlier affect the mean and standard deviation?
- What is variance?
- Why is sample variance commonly calculated with n−1?
- What is standard deviation?
- What is the IQR?
- What does a percentile represent?
- What is conditional probability?
- What is the difference between independent and mutually exclusive events?
- State Bayes' theorem and explain its components.
- What is a random variable?
- What is the difference between discrete and continuous variables?
- What is a normal distribution?
- What is a z-score?
- What is the Central Limit Theorem?
- What is covariance?
- What does Pearson correlation measure?
- Why does correlation not imply causation?
- What is a p-value?
- What is a confidence interval?
- What is sampling bias?
- How does class imbalance affect ML evaluation?
- Where are statistics and probability used in an ML workflow?
29. Hands-on Exercises
- Use NumPy to calculate mean, median, variance, and standard deviation for a small dataset.
- Calculate Q1, Q2, Q3, and IQR for customer spending data.
- Plot a histogram and describe its center, spread, and skew.
- Build a correlation matrix for numerical features.
- Simulate coin tosses and compare empirical probability with theoretical probability.
- Create a small Bayes' theorem example using a diagnostic/classification scenario.
- Add an extreme outlier to a dataset and compare mean versus median before and after.
- Create an imbalanced binary classification dataset and discuss appropriate metrics.
- Draw repeated random samples and examine the distribution of sample means.
30. What You Should Be Able to Do After This Article
- Explain core statistical terminology used in ML.
- Summarize numerical data using central tendency and dispersion.
- Interpret probability and conditional probability.
- Explain Bayes' theorem and base-rate effects.
- Recognize common probability distributions.
- Explain sampling and the practical meaning of the Central Limit Theorem.
- Interpret covariance and correlation correctly.
- Explain p-values and confidence intervals without common misconceptions.
- Use Python to perform basic statistical exploration.
- Connect statistical reasoning to ML data preparation and evaluation.
BITCODEMATRIX • Practical Engineering Learning