Writing · 5 posts
benchmarking
- Fused Softmax::P3::Cuda KernelA CUDA softmax kernel taken from a naive per-row loop to shared memory, float4 loads, and warp-shuffle reductions.
- Fused Softmax::P2::Triton optimizationDebugging a Triton softmax that falls off a cliff at large row sizes, and what block size has to do with it.
- Fused Softmax::P1::Naive & Triton ImplementationSoftmax in Torch and a first Triton kernel, starting from the numerically stable formulation.
- Vector Addition::P4::OptimizingVectorized memory access and a grid-stride loop bring the CUDA vector-add kernel up to Triton and Torch speed.
- Vector Addition::P3::BenchmarkingBenchmarking the CUDA, Triton, and Torch vector-add kernels on a GTX 1080, RTX 5060, A100, and H100.