Day 40 of ML Series: MLE Issue | #MachineLearning

Maximum Likelihood Estimation (MLE) is a fundamental technique in statistics and machine learning for estimating the parameters of a probabilistic model. In this post, we’ll walk through a simple example of using MLE to estimate the parameters of a normal distribution. We’ll use Python for the implementation and visualization.

Problem Statement

Suppose we have a set of data points that we believe are drawn from a normal distribution with unknown mean μ and variance sigma². Our goal is to estimate the parameters μ and sigma² using MLE.

Generating Sample Data

First, let’s generate some sample data from a normal distribution with a known mean and variance.

import numpy as np
import matplotlib.pyplot as plt
# Set seed for reproducibility
np.random.seed(42)
# True parameters
mu_true = 5
sigma_true = 2
# Generate sample data
data = np.random.normal(mu_true, sigma_true, 1000)
# Plot histogram of the data
plt.hist(data, bins=30, density=True, alpha=0.6, color='g')
plt.title('Histogram of Sample Data')
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.show()

Output:

Implementing MLE for Normal Distribution

To find the MLE estimates for the parameters of the normal distribution, we need to:

1. Define the likelihood function.
2. Take the log of the likelihood function (log-likelihood).
3. Derive the expressions for the MLE estimates by maximizing the log-likelihood.

For a dataset {x1, x2,…, xn} assumed to be drawn from a normal distribution with mean μ and variance sigma², we aim to find the parameter values that maximize the likelihood of observing this data.

1. Probability Density Function

The probability density function (PDF) of a normal distribution is:

This function gives the likelihood of a single observation x given the parameters μ and sigma².

2. Likelihood Function

For a dataset of n independent observations {x1, x2, …, xn}, the joint likelihood function L(μ, σ2 ∣ x1, x2,…, xn) is the product of the individual probabilities:

Substituting the PDF of the normal distribution, we get:

3. Log-Likelihood Function

To simplify the calculations and avoid numerical underflow (since the product of many small probabilities can be very small), we take the natural logarithm of the likelihood function. The log-likelihood function is:

Using properties of logarithms, this becomes:

Further simplifying:

This is the log-likelihood function for a normal distribution.

For a normal distribution, the log-likelihood is given by:

The MLE estimates for μ and sigma² are:

For Sigma 2:

Let’s implement this in Python.

# Calculate MLE estimates
mu_mle = np.mean(data)
sigma_mle = np.std(data, ddof=0)  # ddof=0 for MLE estimate of standard deviation
print(f'MLE estimate of mean (mu): {mu_mle}')
print(f'MLE estimate of standard deviation (sigma): {sigma_mle}')

Visualizing the Results

We can now visualize the results by plotting the histogram of the data and overlaying the estimated normal distribution.

from scipy.stats import norm
# Plot histogram of the data
plt.hist(data, bins=30, density=True, alpha=0.6, color='g', label='Data')
# Plot the true normal distribution
x = np.linspace(min(data), max(data), 1000)
plt.plot(x, norm.pdf(x, mu_true, sigma_true), 'r--', label='True Distribution')
# Plot the estimated normal distribution
plt.plot(x, norm.pdf(x, mu_mle, sigma_mle), 'b-', label='Estimated Distribution')
plt.title('Histogram and Normal Distributions')
plt.xlabel('Value')
plt.ylabel('Density')
plt.legend()
plt.show()

Output:

In this post, we demonstrated how to use Maximum Likelihood Estimation (MLE) to estimate the parameters of a normal distribution. We generated sample data, calculated the MLE estimates for the mean and standard deviation, and visualized the results. This simple example illustrates the power and utility of MLE in statistical modeling and inference.

If you like the article and would like to support me make sure to:

👏 Clap for the story (as much as you liked it 😊) and follow me 👉
📰 View more content on my medium profile
🔔 Follow Me: LinkedIn | Medium | GitHub | Twitter