The article presents an interactive visualization of reinforcement learning using a simulated agent called a "spark" with a four-neuron brain: two sensory neurons detecting light brightness (b_L, b_R) and two motor neurons controlling flipper speeds (v_L, v_R). The neural circuitry is defined by three parameters — straight connection weight (w_s), crossed connection weight (w_c), and drive (d) — governed by ReLU-activated equations: v_L = σ(w_s b_L + w_c b_R + d) and v_R = σ(w_c b_L + w_s b_R + d). The spark learns via a weight-perturbation variant of the REINFORCE algorithm (Williams, 1992): each trial, parameters are randomly perturbed (ε), the spark moves for 500 timesteps from a random arena edge, and reward R is the average path brightness. Parameters update as ΔW = α ε (R - R̄), where α is the learning rate and R̄ a running average reward. Over 2,000 trials, sparks converge from random initializations to expert light-seeking, visualized as trajectories in the three-dimensional parameter space converging to a stable attractor. Changing the reward function produces distinct behaviors: fleeing, freezing, or fighting. The piece draws parallels to biological reinforcement learning, such as dopamine-guided song learning in birds, and notes the method's relation to evolution strategies used by OpenAI (2017) for Atari games.
Read original
hackernews