Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sb3 sac onnx export #198

Merged
merged 1 commit into from
Aug 27, 2024
Merged

Add sb3 sac onnx export #198

merged 1 commit into from
Aug 27, 2024

Conversation

Ivan-267
Copy link
Collaborator

@Ivan-267 Ivan-267 commented Aug 22, 2024

This adds support for exporting SAC models with SB3.
Support is added to the onnx export code.

To use, the model in SB3 example needs to be changed from PPO to SAC (for load as well if resume functionality is used), and single obs env should be used with MlpPolicy, e.g. as below:

from godot_rl.wrappers.sbg_single_obs_wrapper import SBGSingleObsEnv
def handle_onnx_export():
    if args.onnx_export_path is not None:
        path_onnx = pathlib.Path(args.onnx_export_path).with_suffix(".onnx")
        print("Exporting onnx to: " + os.path.abspath(path_onnx))
        export_model_as_onnx(model, str(path_onnx), True) # Set to True when using single obs env with MlpPolicy (for both PPO and SAC)
env = SBGSingleObsEnv(
    env_path=args.env_path, show_window=args.viz, seed=args.seed, n_parallel=args.n_parallel, speedup=args.speedup
)
    model: SAC = SAC(
        "MlpPolicy",
        env,
        tensorboard_log=args.experiment_dir,
    )

Example video of running a trained onnx model on BallChase env, trained using SAC for ~1.7 hrs with 2 parallel envs and settings as below (the reward was also slightly changed):

    model: SAC = SAC(
        "MlpPolicy",
        env,
        gradient_steps=32,
        train_freq=32,
        verbose=2,
        learning_starts=250_000,
        tensorboard_log=args.experiment_dir,
    )
ball_chase_sac_onnx.mp4

@Ivan-267 Ivan-267 marked this pull request as draft August 22, 2024 18:50
@Ivan-267 Ivan-267 marked this pull request as ready for review August 22, 2024 19:54
@Ivan-267 Ivan-267 marked this pull request as draft August 26, 2024 03:22
@Ivan-267
Copy link
Collaborator Author

There is still some code polishing to do, will push an update once it's done.

@Ivan-267 Ivan-267 force-pushed the add_sb3_sac_onnx_export branch from 947cb6e to 14bfefe Compare August 26, 2024 04:00
@Ivan-267 Ivan-267 marked this pull request as ready for review August 26, 2024 14:22
@Ivan-267 Ivan-267 requested a review from edbeeching August 26, 2024 14:49
Copy link
Owner

@edbeeching edbeeching left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding SAC model export, LGTM

@Ivan-267 Ivan-267 merged commit 5ccfa8e into main Aug 27, 2024
13 checks passed
@Ivan-267 Ivan-267 deleted the add_sb3_sac_onnx_export branch August 27, 2024 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants