---
date: '2025-09-18'
description: and GPUs, TPUs, accelerators, hardware et al.
id: '4'
modified: 2026-06-05 15:08:32 GMT-04:00
seealso:
  - '[[thoughts/GPU programming|GPU programming]]'
  - '[[thoughts/TPU|TPU]]'
  - '[[thoughts/XLA|XLA]]'
socials:
  link: https://tsfm.ca/lecture-four
tags:
  - ml
  - tsfm
title: lecture four
created: '2025-09-18'
published: '2025-09-18'
pageLayout: default
slug: thoughts/tsfm/4
permalink: https://aarnphm.xyz/thoughts/tsfm/4.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
$$
A(Q, K, V) = \operatorname{softmax}(\frac{Q \cdot K^{T}}{\sqrt{d}})V
$$

> \[!question\]+ Number of FLOPs in matmul?
>
> Given $A \in \mathbb{R}^{m\times k}, B \in \mathbb{R}^{k\times n}$, the matmul ops $C=A\otimes B$ requires how many ops?
>
> > \[!tip\]- Answer
> >
> > $2 \times K \times M \times N$

## IEEE754