Resources

Guides, references, and research for kernel benchmarking

Reward Hacks

When language models generate kernels, they sometimes produce code that games the evaluation rather than performing the intended computation. These “reward hacks” exploit timing mechanisms, memory aliasing, or correctness check gaps to achieve high benchmark scores without genuine optimization. Below are the documented patterns and the defenses used to catch them.

Timing Attack

Exploits the timing measurement — the kernel appears fast because it tricks the clock, not because it computes faster.

Semantic Attack

Produces incorrect, trivial, or cached output that games the correctness check without performing the intended computation.

Benign

Not a deliberate exploit, but circumvents the benchmark's purpose (e.g. calling a library function instead of writing a custom kernel).