Blog Posts

LoRA implementation in PyTorch

9/9/2024

Introduction and implementation for LoRA in PyTorch.

PyTorchLoRAMachine Learning

RoPE (Rotary Position Embedding)

9/10/2024

Math + implementation of RoPE in PyTorch

PyTorchRoPEEmbeddings

Attention to RWKV

10/1/2024

Math explained for scaled dot product to RWKVv7

RWKVAttentionMath

RWKV models explained

12/8/2024

Math and Code for RWKV 5.2 . More to come

RWKVModel ExplanationMath

CUDA Flash Attention Kernel

1/24/2025

Forward pass for Flash Attention

CUDAAttentionOptimization

CUDA Softmax Kernel

1/27/2025

SoftMax Kernel in CUDA beats PyTorch

CUDASoftMaxOptimization

RoPE Kernel Optimization in CUDA

2/4/2025

Introduction and implementation for optimizing RoPE kernel in CUDA.

CUDARoPEOptimization