Writing · 9 posts
GPU kernels
- MatmulA tiled CUDA matmul kernel with shared-memory caching, benchmarked against Torch.
- 2D WorkloadsMapping images onto CUDA thread blocks: row-major memory, 2D indexing, and a grayscale conversion kernel.
- Fused Softmax::P3::Cuda KernelA CUDA softmax kernel taken from a naive per-row loop to shared memory, float4 loads, and warp-shuffle reductions.
- Fused Softmax::P2::Triton optimizationDebugging a Triton softmax that falls off a cliff at large row sizes, and what block size has to do with it.
- Fused Softmax::P1::Naive & Triton ImplementationSoftmax in Torch and a first Triton kernel, starting from the numerically stable formulation.
- Vector Addition::P4::OptimizingVectorized memory access and a grid-stride loop bring the CUDA vector-add kernel up to Triton and Torch speed.
- Vector Addition::P3::BenchmarkingBenchmarking the CUDA, Triton, and Torch vector-add kernels on a GTX 1080, RTX 5060, A100, and H100.
- Vector Addition::P2::Cuda KernelThe same vector addition written in raw CUDA: grids, blocks, threads, and host-to-device memory transfer.
- Vector Addition::P1::Triton KernelA first Triton kernel: vector addition, with pointers, blocks, and masking explained line by line.