---
aliases:
  - kl divergence
date: '2024-12-12'
description: also called relative entropy or I-divergence
id: Kullback-Leibler divergence
modified: 2026-06-05 15:08:06 GMT-04:00
tags:
  - math
  - probability
title: Kullback-Leibler divergence
created: '2024-12-12'
published: '2024-12-12'
pageLayout: default
slug: thoughts/Kullback-Leibler-divergence
permalink: https://aarnphm.xyz/thoughts/Kullback-Leibler-divergence.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
_denoted as_ $D_{\text{KL}}(P \parallel Q)$

> \[!definition\] Definition 1.
>
> The <mark>statistical distance</mark> between a model probability distribution $Q$ difference from a true probability distribution $P$:
>
> $$
> D_{\text{KL}}(P \parallel Q) = \sum_{x \in \mathcal{X}} P(x) \log (\frac{P(x)}{Q(x)})
> $$

Alternative form [^discrete]:

[^discrete]: For _discrete probability distribution_ $P$ and $Q$ defined on the same sample space.

$$
\begin{aligned}
\text{KL}(p \parallel q) &= E_{x \sim p}(\log \frac{p(x)}{q(x)}) \\
&= \int_x P(x) \log \frac{p(x)}{q(x)} dx
\end{aligned}
$$

For relative entropy if $\forall x > 0, Q(x) = 0 \implies P(x) = 0$ _absolute continuity_

For distribution $P$ and $Q$ of a continuous random variable, then KL divergence is:

$$
D_{\text{KL}}(P \parallel Q) = \int_{-\infty}^{+ \infty} p(x) \log \frac{p(x)}{q(x)} dx
$$

where $p$ and $q$ denote probability densities of $P$ and $Q$