Series
GPU and Kernel Optimization
From CUDA thread blocks and shared memory to custom Triton kernels, fused operators, and profiling with nsys — the implementation layer behind fast LLM inference.
- 1
CUDA Fundamentals for ML Engineers
CUDA exposes GPU parallelism through a three-level thread hierarchy: grid, block, and warp. Understanding how these map to hardware — SMs, register files, shared memory — is the prerequisite for writing fast kernels.
2024-06-20
- 1
CUDA Fundamentals for ML Engineers
CUDA exposes GPU parallelism through a three-level thread hierarchy: grid, block, and warp. Understanding how these map to hardware — SMs, register files, shared memory — is the prerequisite for writing fast kernels.
2024-06-20
- 2
Writing Fused Kernels with Triton
Kernel fusion eliminates the HBM round-trips between chained operations. Triton makes this practical in Python. This post builds a fused online softmax from scratch, then extends it to a fused RMSNorm + linear projection — the kind of kernel that actually speeds up LLM inference.
2024-06-20
- 2
Writing Fused Kernels with Triton
Kernel fusion eliminates the HBM round-trips between chained operations. Triton makes this practical in Python. This post builds a fused online softmax from scratch, then extends it to a fused RMSNorm + linear projection — the kind of kernel that actually speeds up LLM inference.
2024-06-20