In this experiment, the author demonstrates that outcome-grounded retrieval can outperform a random baseline in the Gymnasium CartPole-v1 environment without traditional reinforcement learning training. The methodology involves logging 200 episodes of random agent behavior using the vaas-x Bootstrap API. Each state is converted into a discrete text description via a custom "bucket" function that categorizes observations into labels like "cart_left" or "pole_falling_right." These states are paired with the actions taken and stored in a remote memory. To execute the policy, the agent performs a similarity query for the current state, retrieving the top $k=5$ most similar historical states. Crucially, the retrieval is biased toward successful outcomes using the `prefer_success=True` parameter. The agent then uses a majority vote of the actions associated with these successful historical states to determine its next move. While the author notes that the CartPole environment can be noisy and results may vary by seed, the goal is to prove that biasing action selection toward previously successful state-action pairs via memory retrieval provides a measurable performance lift over a purely stochastic policy.

Read original