profile pic
⌘ '
raccourcis clavier
Search

Kullback-Leibler divergence

also called relative entropy or I-divergence.

  • Étiquette

    • publié à

    • modifié à

    • durée

      1 min de lecture

    • source

      llms.txt

    denoted as DKL(P∥Q)D_{\text{KL}}(P \parallel Q)

    definition

    The statistical distance between a model probability distribution QQ difference from a true probability distribution PP:

    DKL(P∥Q)=∑x∈XP(x)log⁡(P(x)Q(x))D_{\text{KL}}(P \parallel Q) = \sum_{x \in \mathcal{X}} P(x) \log (\frac{P(x)}{Q(x)})

    alternative form:

    KL(p∥q)=Ex∼p(log⁡p(x)q(x))=∫xP(x)log⁡p(x)q(x)dx\begin{aligned} \text{KL}(p \parallel q) &= E_{x \sim p}(\log \frac{p(x)}{q(x)}) \\ &= \int_x P(x) \log \frac{p(x)}{q(x)} dx \end{aligned}

    Vous pourriez aimer ce qui suit

  • PageRank
  • Menus
  • feed-forward neural network
  • Autograd
  • Technical Design
  • Capitalism and Freedom
  • Socrates
    • autoencoders
    feature extraction from neural network
    Quantization
    See also: this talk I gave at Hack the North 2023. reduce computational and memory costs of running inference with representing the weight and activations with low-precision data type int16 - half precision bfloat16 int8 fp8 Note This also applies to post-training quantization, where the methodology is applied after the model has been trained, instead of during load-time.