---
date: '2025-09-07'
description: Python and Rust
id: implementation
modified: 2026-06-05 15:08:32 GMT-04:00
tags:
  - seed
  - technical
  - ml
title: implementation of tokenization
created: '2025-09-07'
published: '2025-09-07'
pageLayout: default
slug: thoughts/tsfm/lecture-2-exercise/implementation
permalink: https://aarnphm.xyz/thoughts/tsfm/lecture-2-exercise/implementation.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
- training logic: [[thoughts/tsfm/lecture-2-exercise/src/minibpe/core.py]]
- Python: [[thoughts/tsfm/lecture-2-exercise/src/minibpe/impl.py]]
- Rust: [[thoughts/tsfm/lecture-2-exercise/rust/fastbpe/src/bpe.rs]]