Reinforcement Learning Snake AI

The Goal: Achieving Superhuman Foresight

The game of Snake, while simple in its rules, is a profound challenge in long-term planning. A purely greedy approach (always moving towards the apple) inevitably leads to self-trapping. My objective was to move beyond simple heuristics and create an AI that could learn to play with true foresight, aiming to fill the entire board by building its own understanding of strategy through reinforcement learning.

Architecture Inspired by AlphaZero

This project's architecture is a direct adaptation of the principles laid out in DeepMind's groundbreaking AlphaZero paper. The core of the system is a powerful combination of a deep neural network and a Monte Carlo Tree Search (MCTS) algorithm. The neural network learns to evaluate board positions and suggest moves, while the tree search explores future possibilities to find the optimal path.

The Core Components

The AI's "brain" was built on three interconnected components that work in a continuous loop of learning and refinement:

• Self-Play Reinforcement Learning: The foundation of the training process. The AI plays thousands of games against itself to generate data. In each game, it learns from its successes and failures, gradually improving its strategy from a state of complete randomness to one of deep tactical understanding. This generated data is crucial for training the neural network.
• A Dual-Headed Neural Network: This is the heart of the learned intuition. The network takes the current game state (the board) as input and produces two outputs:
  • A Policy Head, which outputs a probability distribution over possible next moves. It essentially answers the question, "What are the most promising moves to explore from here?"
  • A Value Head, which outputs a single scalar value from -1 to 1. This value estimates the expected outcome of the game from the current position (e.g., win/loss or a score proxy), answering, "How good is this board state for me?"
• Monte Carlo Tree Search (MCTS) with Alpha-Beta Pruning: For each move during a game, the AI doesn't just trust the neural network's initial guess. It performs a sophisticated lookahead search. The MCTS builds a search tree of future moves, guided by the network's policy. The value output from the network is used to evaluate the leaves of this tree. To make the search more efficient, I implemented alpha-beta pruning, a technique that dramatically reduces the number of nodes the search needs to evaluate by ignoring branches that are provably worse than a path already found.

Learning and Results

The training process was a remarkable thing to watch. The agent started by bumping into walls and its own tail, but through self-play, it quickly learned to survive, then to hunt apples efficiently, and finally, to master complex, board-filling maneuvers. The synergy between the learned intuition of the neural network and the deliberate, "what-if" analysis of the tree search allowed the AI to develop strategies that were not explicitly programmed.

This project was a deep dive into the practical application of modern reinforcement learning. It was a challenging and rewarding experience to adapt the abstract concepts from the AlphaZero paper into a concrete, functioning system that could learn and master a complex task from scratch.