Issues training in 1.0.1

[sk2-fnc]

Hello all, I have upgraded from a prior version of Yawning titan using the 18-node and a custom 10 node network to train on a default game mode that had been edited to 100 steps instead of 1000. The other change made was to fix the "rewards_are_muiltipled_by_end_state". Some of my training results are shown below.

Upon upgrading to the v1.0.1 release and attempting to run the same scenarios I the below results occurred. As you can see the training is not going well and getting stuck fairly consistently at the same place.

As far as I can tell all of the settings in the game/agent/network are the same. Adjusted to 100 steps, same network, reward multiplied by end state and running with in a conda environment with the required packages.

I'm not sure what I might have missed that's making the training perform poorly and would appreciate any advice you might have. Are there any settings that might have changed that I have missed?

[czar-ec-envitia]

Could you provide a minimum reproducible example or console outputs/logs, etc?

This would help us reproduce the issue

[sk2-fnc]

Yeah sure thank you, this is the basic code I'm running in v1.0.1 of YT.

The only changes I have made is to hardcode the link to the GameModeConfig file and to fix issue 10 "Incorrect reward for blue agent reaching max_steps".

This is the same setup I had for v0.1.0 which produced reasonable looking results.

import time
from stable_baselines3 import PPO
from stable_baselines3.ppo import MlpPolicy as PPOMlp
from stable_baselines3.common.env_checker import check_env
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.callbacks import EvalCallback
from stable_baselines3.common.monitor import Monitor

from yawning_titan.envs.generic.core.blue_interface import BlueInterface
from yawning_titan.envs.generic.core.red_interface import RedInterface
from yawning_titan.envs.generic.generic_env import GenericNetworkEnv
from yawning_titan.envs.generic.helpers import network_creator
from yawning_titan.envs.generic.core.action_loops import ActionLoop
from yawning_titan.envs.generic.core.network_interface import NetworkInterface
from yawning_titan.config.network_config.network_config import NetworkConfig
from yawning_titan.config.game_modes import default_game_mode_path
from yawning_titan.config.game_config.game_mode_config import GameModeConfig

from datetime import datetime

#yawning_titan.config.game_modes
game_mode_config = GameModeConfig.create_from_yaml("C:/Users/XXX/Desktop/git/YAWNING-TITAN-v1.0.1/yawning_titan/config/_package_data/game_modes/default_game_mode.yaml")

matrix, node_positions = network_creator.create_18_node_network()
size = "18 node"

#network
network_config = NetworkConfig.create_from_args(matrix=matrix, positions=node_positions)
network_interface = NetworkInterface(game_mode=game_mode_config, network=network_config)

#create red and blue agents
red = RedInterface(network_interface)
blue = BlueInterface(network_interface)

env = GenericNetworkEnv(red, blue, network_interface)
check_env(env, warn=True)
# reset anything changed during the check
_ = env.reset()

#Evaluation Callback
eval_callback = EvalCallback(Monitor(env), eval_freq=100000, deterministic=False, render=False)
#create RL agent
agent = PPO(PPOMlp, env, learning_rate=3e-4,verbose=1, tensorboard_log="models/PPO")

#train the agent
timesteps = 2000000
agent.learn(total_timesteps=timesteps, n_eval_episodes=1, callback=eval_callback, tb_log_name="PPO")

evaluate_policy(agent, env, n_eval_episodes=50, render=True)

#save the agent
now = datetime.now()
date_string = now.strftime("%Y-%m-%dT%H%M%S")
agent.save(date_string + f" {timesteps}steps -{size}")

Appreciate any help.

[czar-ec-envitia]

You don't happen to have the code block you used for the Incorrect reward for blue agent reaching max_steps, do you?

Also, if you weren't aware already, there is also a new v1.1.1 release that has a fix for that bug

[czar-ec-envitia]

Hello, we have not heard from you in a while.
I will be marking this discussion as resolved, however, if you still have the problem, feel free to reopen the discussion and we will look into it further