Blog

Notes on how LLMs work under the hood, and a hands-on series on writing GPU kernels.

#benchmarking #cuda-kernel #image-processing #kernels #llm #positional embedding #rope #triton-kernel

RoPE: The Phase Geometry Behind Long-Context Transformers
How rotating query and key vectors turns absolute position into relative attention
Jun 6, 2026 · #llm
From Greedy to Nucleus, How LLMs Choose the Next Token
Decoding algorithms and sampling techniques explained
Dec 22, 2025 · #llm
Matmul
A tiled CUDA matmul kernel with shared-memory caching, benchmarked against Torch.
Dec 1, 2025 · #kernels
2D Workloads
Mapping images onto CUDA thread blocks: row-major memory, 2D indexing, and a grayscale conversion kernel.
Nov 22, 2025 · #kernels
Fused Softmax::P3::Cuda Kernel
A CUDA softmax kernel taken from a naive per-row loop to shared memory, float4 loads, and warp-shuffle reductions.
Nov 16, 2025 · #kernels
Fused Softmax::P2::Triton optimization
Debugging a Triton softmax that falls off a cliff at large row sizes, and what block size has to do with it.
Nov 9, 2025 · #kernels
Fused Softmax::P1::Naive & Triton Implementation
Softmax in Torch and a first Triton kernel, starting from the numerically stable formulation.
Nov 2, 2025 · #kernels
Vector Addition::P4::Optimizing
Vectorized memory access and a grid-stride loop bring the CUDA vector-add kernel up to Triton and Torch speed.
Oct 26, 2025 · #kernels
Vector Addition::P3::Benchmarking
Benchmarking the CUDA, Triton, and Torch vector-add kernels on a GTX 1080, RTX 5060, A100, and H100.
Oct 12, 2025 · #kernels

#include <cuda_runtime.h>
/*
 * __global__ = This is a CUDA keyword that tells the compiler:
 *    "This function runs on the GPU and can be called from CPU!"
 *
 * Think of this as a tiny worker that gets copied thousands of times across
 * the GPU. Each copy (thread) handles ONE element of the arrays.
 *
 * Params:

Vector Addition::P2::Cuda Kernel

The same vector addition written in raw CUDA: grids, blocks, threads, and host-to-device memory transfer.

Oct 5, 2025 · #kernels

import torch
import triton
import triton.language as tl
DEVICE = triton.runtime.driver.active.get_active_torch_device()
@triton.jit
def add_kernel(x_ptr,  # *Pointer* to first input vector.
               y_ptr,  # *Pointer* to second input vector.
               output_ptr,  # *Pointer* to output vector.
               n_elements,  # Size of the vector.

Vector Addition::P1::Triton Kernel

A first Triton kernel: vector addition, with pointers, blocks, and masking explained line by line.

Sep 28, 2025 · #kernels