ai

STATISTICS AND PROBABILITY FOR MACHINE LEARNING

Statistics helps us describe data, measure variation, and quantify relationships. Probability gives us a framework for reasoning about uncertainty. Together, they form an important mathematical foundation for machine learning.

By Shubhankar Tripathi • • 5 min read

STATISTICS AND PROBABILITY

FOR MACHINE LEARNING

The Mathematical Foundation of ML

Statistics helps us describe data, measure variation, and quantify relationships. Probability gives us a framework for reasoning about uncertainty. Together, they form an important mathematical foundation for machine learning.

Learning Objectives

  • Understand population, sample, variable, and observation.
  • Calculate and interpret mean, median, mode, weighted mean, variance, and standard deviation.
  • Use percentiles, quartiles, and the interquartile range to understand data distribution.
  • Understand probability, conditional probability, independence, and Bayes' theorem.
  • Recognize common discrete and continuous probability distributions.
  • Understand expectation, sampling, the Central Limit Theorem, covariance, and correlation.
  • Understand the basic purpose of hypothesis testing and confidence intervals.
  • Connect statistical concepts to exploratory data analysis, feature engineering, model evaluation, and ML systems.

1. Why Statistics and Probability Matter in ML

Machine learning rarely operates on perfectly predictable data. Customer behavior, sensor measurements, transactions, images, text, and operational metrics contain noise and variation. Statistics helps us summarize and investigate that variation, while probability helps us model uncertainty and reason about possible outcomes.

Concept

Typical ML Connection

Mean / Median

Data summarization and robust analysis

Variance / Standard Deviation

Feature scaling and understanding spread

Probability

Classification probabilities and uncertainty

Distributions

Modeling assumptions and simulation

Sampling

Training, validation, experiments

Correlation / Covariance

Feature relationships and analysis

Bayes' Theorem

Probabilistic inference

Confidence Intervals

Uncertainty around estimates

Hypothesis Testing

Experiments and statistical comparisons

2. Population, Sample and Observation

A population is the complete group we want to understand. A sample is a subset collected from that population. An observation is one individual record or measurement.

Term

Meaning

Example

Population

Entire group of interest

All customers of a company

Sample

Subset used for analysis

10,000 selected customers

Observation

One record/measurement

One customer's transaction

Parameter

Numerical property of a population

Population mean μ

Statistic

Numerical quantity computed from a sample

Sample mean x̄

3. Types of Data

Type

Description

Examples

Numerical

Quantitative values

Age, income, temperature

Categorical

Groups or labels

City, plan type, device

Discrete

Countable values

Number of purchases

Continuous

Measured values on a continuum

Weight, latency

Ordinal

Categories with meaningful order

Low / Medium / High

4. Measures of Central Tendency

Mean

Mean is the arithmetic average:

x̄ = (x₁ + x₂ + ... + xₙ) / n

The mean uses every value, so it can be strongly affected by extreme observations.

Median

Median is the middle value after sorting the observations. For an even number of observations, it is the average of the two middle values. It is often more robust than the mean when a feature is skewed.

Mode

Mode is the most frequently occurring value. It can be useful for categorical variables as well as discrete numerical data.

Measure

Strength

Limitation

Mean

Uses all values

Sensitive to outliers

Median

Robust to extreme values

Does not use distances between all observations

Mode

Works naturally with categories

May be non-unique or absent

5. Weighted Mean

A weighted mean assigns different importance to observations. It is useful when observations do not contribute equally.

Weighted mean = Σ(wᵢxᵢ) / Σwᵢ

Example: If model performance is evaluated across datasets of different sizes, weighting each dataset equally may not represent the overall population as well as weighting by the number of observations.

6. Measures of Dispersion

Central tendency alone is not enough. Two datasets can have the same mean but very different spread.

Measure

Meaning

Range

Maximum − Minimum

Variance

Average squared deviation from the mean, with sample/population definitions

Standard Deviation

Square root of variance; expressed in the original units

IQR

Q3 − Q1; spread of the middle 50% of observations

7. Variance and Standard Deviation

Population variance:

σ² = (1/n) Σ(xᵢ − μ)²

Sample variance:

s² = (1/(n−1)) Σ(xᵢ − x̄)²

The sample formula uses n−1 in the denominator because it provides an unbiased estimator of population variance under the standard random-sampling framework.

Standard deviation is the square root of variance:

σ = √σ²    or    s = √s²

8. Percentiles and Quartiles

A percentile indicates a position in an ordered dataset. The 50th percentile corresponds to the median. Quartiles divide ordered data into four parts: Q1, Q2 (median), and Q3.

Interquartile Range (IQR) = Q3 − Q1

IQR is useful for understanding the spread of the central portion of a dataset and is commonly used in outlier analysis.

9. Probability Fundamentals

  • Experiment: a process that produces an outcome.

  • Sample space: the set of possible outcomes.

  • Event: a subset of outcomes of interest.

  • Probability: a value from 0 to 1 representing the likelihood of an event under a specified probability model.

10. Basic Probability Rules

Complement:  P(not A) = 1 − P(A)

Addition rule:  P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

Multiplication rule:  P(A ∩ B) = P(A)P(B|A)

If A and B are independent, P(B|A) = P(B), so P(A ∩ B) = P(A)P(B).

11. Conditional Probability

Conditional probability describes the probability of A given that B has occurred:

P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0.

This concept is central to probabilistic ML because the information available to the model can change the probability assigned to an outcome.

12. Independence

Two events are independent when knowledge that one occurred does not change the probability of the other. Independence is different from being mutually exclusive: mutually exclusive events with positive probability cannot occur together, while independent events can occur together.

13. Bayes' Theorem

Bayes' theorem connects conditional probabilities:

P(A|B) = [P(B|A)P(A)] / P(B)

Term

Meaning

Prior

P(A): belief/probability before observing B

Likelihood

P(B|A): probability of evidence given A

Evidence

P(B): overall probability of the evidence

Posterior

P(A|B): updated probability after observing B

In classification and diagnosis-style problems, Bayes' theorem highlights why base rates matter: a strong-looking signal does not by itself determine the posterior probability.

14. Random Variables

Type

Description

Example

Discrete random variable

Takes countable values

Number of failed transactions

Continuous random variable

Can take values over an interval

Response time

15. Common Probability Distributions

Distribution

Type

Typical Use

Bernoulli

Discrete

One binary trial

Binomial

Discrete

Number of successes in fixed independent trials

Poisson

Discrete

Counts of events in a fixed interval under suitable assumptions

Uniform

Discrete/Continuous

Outcomes with equal density/probability over a specified range

Normal

Continuous

Symmetric continuous modeling and many statistical procedures

Exponential

Continuous

Waiting-time modeling under suitable assumptions

16. Normal Distribution

The normal distribution is continuous and symmetric around its mean. Its shape is determined by its mean and standard deviation.

Standardization (z-score):  z = (x − μ) / σ

A z-score expresses how far a value is from the mean in standard-deviation units. Not every real-world ML feature is normally distributed, so normality should be checked rather than assumed.

17. Expectation

Expectation is the probability-weighted average of a random variable's possible values. For a discrete random variable:

E[X] = Σ x P(X = x)

Expectation is useful when reasoning about average outcomes, expected loss, and probabilistic models.

18. Sampling

Sampling Method

Description

Simple Random

Every eligible unit has a specified random selection opportunity

Stratified

Population divided into groups and samples taken from strata

Systematic

Select observations using a fixed interval after a starting point

Cluster

Select groups/clusters and sample within selected groups depending on design

Sampling bias occurs when the sampling process systematically produces a sample that does not adequately represent the target population. Selection bias can affect both statistical conclusions and ML model performance.

19. Central Limit Theorem — Practical View

Under appropriate conditions, the sampling distribution of a suitably normalized sample mean approaches a normal distribution as sample size grows. This result supports many statistical procedures.

Important: the Central Limit Theorem does not say that every raw dataset becomes normally distributed.

20. Covariance

Covariance describes how two variables vary together. Positive covariance indicates that higher values of one variable tend to be associated with higher values of the other under the observed data; negative covariance indicates the opposite tendency.

Its magnitude depends on the units of the variables, which makes covariance harder to compare across differently scaled features.

21. Correlation

Pearson correlation standardizes linear association to a range from −1 to +1.

Correlation

Typical Interpretation

Near +1

Strong positive linear association

Near 0

Little or no linear association

Near −1

Strong negative linear association

Correlation does not establish causation. A low Pearson correlation also does not prove that two variables have no relationship; the relationship could be nonlinear or otherwise structured.

22. Statistical Significance and Hypothesis Testing

  • Null hypothesis (H₀): a reference claim used for the test.

  • Alternative hypothesis (H₁): a competing claim.

  • Test statistic: a quantity calculated from the observed data.

  • p-value: under the null model, the probability of observing a result at least as extreme as the one obtained, according to the test definition.

  • Significance level (α): a preselected threshold used to define a rejection rule.

A p-value is not the probability that the null hypothesis is true, and a small p-value alone does not measure practical importance.

23. Confidence Intervals

A confidence interval provides a range produced by a statistical procedure to communicate uncertainty around an estimate. In the frequentist repeated-sampling interpretation, a 95% confidence procedure produces intervals that contain the true parameter in 95% of repeated samples under the stated assumptions.

Confidence intervals should be interpreted together with the sampling design, assumptions, and practical context.

24. Statistics in Machine Learning

ML Activity

Statistical Role

EDA

Summarize distributions, spread, missingness, and relationships

Feature Analysis

Understand scale, skew, outliers, and association

Sampling

Build representative train/validation/test data where appropriate

Classification

Reason about class proportions and probabilistic outputs

Anomaly Detection

Identify observations that are unusual under a reference distribution/model

Experimentation

Quantify uncertainty and compare alternatives

Evaluation

Interpret metrics across samples and populations

25. Example: Customer Churn

Suppose a company wants to predict customer churn. Statistics can be used before model training to understand the dataset:

  • Compare mean and median monthly spend.
  • Measure standard deviation and percentiles to understand customer variability.
  • Compare churn rates across customer segments.
  • Inspect relationships among tenure, spend, support calls, and churn.
  • Check class imbalance before selecting evaluation metrics.
  • Use appropriate train/validation/test splits to estimate generalization.
  • Interpret model metrics with uncertainty and business context.

26. Python Example

import numpy as np
import pandas as pd

values = np.array([10, 12, 15, 18, 20])

print("Mean:", values.mean())
print("Median:", np.median(values))
print("Standard deviation:", values.std())

df = pd.DataFrame({
    "age": [25, 31, 42, 28, 36],
    "income": [40000, 55000, 90000, 48000, 70000]
})

print(df.describe())
print(df.corr(numeric_only=True))

27. Common Mistakes

  • Confusing population parameters with sample statistics.
  • Using the mean without checking extreme values and skew.
  • Assuming every feature follows a normal distribution.
  • Interpreting correlation as causation.
  • Ignoring base rates when interpreting probabilities.
  • Treating a p-value as the probability that the null hypothesis is true.
  • Ignoring uncertainty around estimates.
  • Applying statistical tests without considering their assumptions.
  • Using biased samples and expecting unbiased conclusions.
  • Choosing an evaluation metric without considering class balance and the real objective.

28. Interview Questions

  1. What is the difference between a population and a sample?
  2. What is the difference between a parameter and a statistic?
  3. When would you prefer the median over the mean?
  4. How does an outlier affect the mean and standard deviation?
  5. What is variance?
  6. Why is sample variance commonly calculated with n−1?
  7. What is standard deviation?
  8. What is the IQR?
  9. What does a percentile represent?
  10. What is conditional probability?
  11. What is the difference between independent and mutually exclusive events?
  12. State Bayes' theorem and explain its components.
  13. What is a random variable?
  14. What is the difference between discrete and continuous variables?
  15. What is a normal distribution?
  16. What is a z-score?
  17. What is the Central Limit Theorem?
  18. What is covariance?
  19. What does Pearson correlation measure?
  20. Why does correlation not imply causation?
  21. What is a p-value?
  22. What is a confidence interval?
  23. What is sampling bias?
  24. How does class imbalance affect ML evaluation?
  25. Where are statistics and probability used in an ML workflow?

29. Hands-on Exercises

  • Use NumPy to calculate mean, median, variance, and standard deviation for a small dataset.
  • Calculate Q1, Q2, Q3, and IQR for customer spending data.
  • Plot a histogram and describe its center, spread, and skew.
  • Build a correlation matrix for numerical features.
  • Simulate coin tosses and compare empirical probability with theoretical probability.
  • Create a small Bayes' theorem example using a diagnostic/classification scenario.
  • Add an extreme outlier to a dataset and compare mean versus median before and after.
  • Create an imbalanced binary classification dataset and discuss appropriate metrics.
  • Draw repeated random samples and examine the distribution of sample means.

30. What You Should Be Able to Do After This Article

  • Explain core statistical terminology used in ML.
  • Summarize numerical data using central tendency and dispersion.
  • Interpret probability and conditional probability.
  • Explain Bayes' theorem and base-rate effects.
  • Recognize common probability distributions.
  • Explain sampling and the practical meaning of the Central Limit Theorem.
  • Interpret covariance and correlation correctly.
  • Explain p-values and confidence intervals without common misconceptions.
  • Use Python to perform basic statistical exploration.
  • Connect statistical reasoning to ML data preparation and evaluation.

BITCODEMATRIX • Practical Engineering Learning