Many reinforcement learning problems can be seen from the point of view of stochastic approximation. Unfortunately, classic stochastic approximation algorithms, such as Robbins-Monro, may have an infinite asymptotic variance. The class of “zap” algorithms aim at solving that problem. We then examine the application of zap algorithms to reinforcement learning, with the example of zap Q-learning.