profile pic
⌘ '
raccourcis clavier
Search

Foundational large models

  • Étiquette

    • publié à

    • modifié à

    • durée

      1 min de lecture

    • source

      llms.txt

    Popularized through LLMs, GPT-3 paper,

    See also: 7.1 of The Little Book of Deep Learning

    Though, it should be thought as Intelligence amplification rather than “artificial intelligence” system.

    Scaling laws

    Initial work from OpenAI

    Distributed serving of large models requires cost-efficient methods1

    • Petals: a decentralized system that run Llama 2 over internet

    large world models

    LWM: implementation of RingAttention

    visions

    Remarque

    1. Distributed Inference and Fine-tuning of Large Language Models over the Internet ↩

    Vous pourriez aimer ce qui suit

  • Turing-complete Transformers
  • dundurn.
  • data
  • Root locus
  • Fisher-Yates
  • Stochastic gradient descent
  • Sequential programming
    • LLMs
    large language models, often implemented as autoregressive transformers models. GPTs and friends Most variants of LLMs are decoder-only (Radford et al., 2019) Have “capabilities” to understand natural language.
    Turing-complete Transformers
    Turing Complete Transformers: Two Transformers Are More Powerful Than One"We prove transformers are not Turing complete, propose a new architecture that is Turing complete, ...