Seeking Insights: Enhancing a Connect Four AI Project

[walt-neb]

I've been working on a Connect Four AI project, developed primarily as a learning opportunity to explore reinforcement learning (RL) in a two-agent gameplay environment. The project is structured around training and testing AI models to master Connect Four, using a Python-based setup.

Initially, I approached the problem by feeding a flattened 6x7 array representing the board into a multi-layer fully connected neural network (FC NN). Realizing the need for improvement, I integrated a convolutional neural network (CNN) layer to process the grid before passing it into the fully connected layers, aiming to better capture spatial relationships.

Despite experimenting with both large and small network architectures, the performance of the trained models has been underwhelming, and I'm currently unable to pinpoint the exact causes of this limitation. Each training session is managed through a hyperparameters.txt file, which not only sets the initial conditions but also logs the outcomes, ensuring transparency and traceability of what inputs lead to specific results.

The GitHub repository for this project is available here: Connect Four AI Project. I am eager to get feedback or suggestions on how to improve the model's performance to reach a master-level capability in Connect Four.

If you have experience with deep reinforcement learning, especially in game environments, or have insights into how I might enhance the training process or model architecture, I would greatly appreciate your input.
The repo is located here: https://github.com/walt-neb/connect_four

Below is an example output from the training session:

----Episode 500 of 20000--------
 0  1  2  3  4  5  6
 | X | . | X | . | . | X | . |
 | O | O | X | . | O | O | . |
 | O | X | X | X | X | O | . |
 | O | X | O | O | O | X | O |
 | X | O | O | X | X | O | X |
 | O | O | X | O | X | X | X |

Episode 500 Step 35
Agent 1 (X) action: 3
Agent 1 wins
Agent 1: 269
Agent 2: 231, Draws: 1
Agent 1 epsilon: 0.8127695141087002
Agent 2 epsilon: 0.8127695141087002
Agent 1 loss: 0.07183947476247947
Agent 2 loss: 0.07168800383806229
A1 reward / A2 reward: 1.164
Win Rates -> Agent 1: 0.5380
       Agent 2: 0.4620

[stephane-caron]

It is unlikely you get pertinent feedback here, as this forum is a place for discussions around Upkie robots, while your question is about reinforcement learning in a different context. Maybe try asking your question on /r/reinforcementlearning, or on a Discord channel about reinforcement learning such as this one?

Hoping this helps and all the best for your project.