---
date: '2024-01-08'
description: and compression of human knowledge
id: large models
modified: 2026-06-05 15:08:20 GMT-04:00
tags:
  - ml
title: Foundational models
created: '2024-01-08'
published: '2024-01-08'
pageLayout: default
slug: thoughts/large-models
permalink: https://aarnphm.xyz/thoughts/large-models.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
Popularized through [[thoughts/LLMs]], GPT-3 paper \[@brown2020languagemodelsfewshotlearners\]

Though, it should be thought as [[thoughts/Intelligence amplification]] rather than “artificial intelligence” system.

## Scaling laws

Initial [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) \[@kaplan2020scalinglawsneurallanguage\]  from OpenAI

Distributed serving of large models requires cost-efficient methods[^1]

- [Petals](https://petals.dev/): a decentralized system that run Llama 2 over internet

### large world models

[LWM](https://github.com/LargeWorldModel/LWM): implementation of [[thoughts/Attention#RingAttention|RingAttention]]

## visions

[^1]: [Distributed Inference and Fine-tuning of Large Language Models Over The Internet](https://arxiv.org/abs/2312.08361) \[@borzunov2023distributedinferencefinetuninglarge\]