Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Add support for multi-agent off-policy algorithms in the new API stack. #45182

Merged
merged 41 commits into from
May 24, 2024
Merged

[RLlib] Add support for multi-agent off-policy algorithms in the new API stack. #45182

merged 41 commits into from
May 24, 2024

Commits on Apr 29, 2024

  1. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    baa1398 View commit details
    Browse the repository at this point in the history

  2. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    a1eb1f9 View commit details
    Browse the repository at this point in the history

  3. fixes 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    6538b58 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Merge branch 'master' of https://github.com/ray-project/ray into chan… 

    …ge_episode_buffers_to_return_episode_lists_from_sample
    @sven1977
    sven1977 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    683f515 View commit details
    Browse the repository at this point in the history

  2. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    366a4b9 View commit details
    Browse the repository at this point in the history

  3. wip 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    a8b2d0c View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. merge 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed May 3, 2024
    Configuration menu
    Copy the full SHA
    f76628a View commit details
    Browse the repository at this point in the history

  2. Fixed a bug with 'TERMINATEDS/TRUNCATEDS' in replay buffer sampling t… 

    …hat held DQN off from learning. In addition fixed some minor bugs.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 3, 2024
    Configuration menu
    Copy the full SHA
    81421d9 View commit details
    Browse the repository at this point in the history

  3. LINTER. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 3, 2024
    Configuration menu
    Copy the full SHA
    bd54d5a View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Added docs to new 'sample' method and removed old sample methods. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    6ee006f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into change_episode_buffers_to_return_episode_l… 

    …ists_from_sample

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    a345d09 View commit details
    Browse the repository at this point in the history

  3. Replaced 'td_error' by 'TD_ERROR_KEY'. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    b77fd5a View commit details
    Browse the repository at this point in the history

  4. Needed to define 'TD_ERROR_KEY' in 'replay_buffer.utils' b/c import e… 

    …rror occurred in CI tests.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    6e11ff6 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. Fixed a small bug in test code. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b39b9a8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into change_episode_buffers_to_return_episode_l… 

    …ists_from_sample

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    e6cf4f7 View commit details
    Browse the repository at this point in the history

  3. Interchanged 'new_obs' with our constant 'Columns.NEXT_OBS' for bette… 

    …r readability of the test code for users (we describe the connector to add the 'NEXT_OBS' to the batch).

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    eebc04d View commit details
    Browse the repository at this point in the history

  4. Added new sampling method in 'MultiAgentEpisodeReplayBuffer' for 'ind… 

    …ependent'-mode sampling. Added multi-agent example for SAC and modified 'compute_gradients' in 'SACTorchLearner' to deal with MARLModules. Commented 2 assertions in connectors that avoided multi-agent setups with 'SingleAgentEpisode's.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    d12f16f View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. Changed 'truncated/terminated' logic in 'MultiEnv' and 'MultiAgentEpi… 

    …sode'

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    2247c02 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Switched back to 'pid'. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    827adda View commit details
    Browse the repository at this point in the history

  2. Commented out NaN metrics b/c they produced hindreds of warnings. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    1e67ccf View commit details
    Browse the repository at this point in the history

  3. Changed comment. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    c748df8 View commit details
    Browse the repository at this point in the history

  4. Little changes here and there and to clean-up sample logic and multi-… 

    …agent off-policy algorithms.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    fc35faa View commit details
    Browse the repository at this point in the history

  5. Added suggestions from @sven1977's review. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    c336ac8 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Merge branch 'master' of https://github.com/ray-project/ray 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    6409007 View commit details
    Browse the repository at this point in the history

  2. Merged master 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    81c3893 View commit details
    Browse the repository at this point in the history

  3. Modified multi-agent buffer tests to correspond to the changes in '_s… 

    …ample_independent'. Added 'multi_gpu' to the BUILD test for multi-agent SAC.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    c522597 View commit details
    Browse the repository at this point in the history

  4. CHanged 'MultiAGentEpisode' and 'MultiEnv' back to master. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    b8fbe19 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Apply suggestions from code review 

    Signed-off-by: Sven Mika <sven@anyscale.io>
    @sven1977
    sven1977 authored May 14, 2024
    Configuration menu
    Copy the full SHA
    feafb6b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ray-project/ray 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    d2f9030 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. Added slots to 'MultiAgentEpisode' which should help reducing memory … 

    …footprint of the class. Changes to 'MultiAgentEpisodeReplayBuffer' to reduce memory usage and increase performance.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    2fd7717 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ray-project/ray 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    a3416a8 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Changed multi-agent SAC example such that at a minimum 2 agents are u… 

    …sed.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    2296cfc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ray-project/ray 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    8582ad9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into change_ma_buffer_to_use_list_of_episodes 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    c8d72fa View commit details
    Browse the repository at this point in the history

  4. Multiple performance tunings that bring the multi-agent buffer into d… 

    …irection single-agent buffer. Memory leak should be fixed with this commit.

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    ffbf3de View commit details
    Browse the repository at this point in the history

  5. LINTER. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    47888a4 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'change_ma_buffer_to_use_list_of_episodes' of github.com… 

    …:simonsays1980/ray into change_ma_buffer_to_use_list_of_episodes

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    7d6497e View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. Merge branch 'master' of https://github.com/ray-project/ray into chan… 

    …ge_ma_buffer_to_use_list_of_episodes
    @sven1977
    sven1977 committed May 17, 2024
    Configuration menu
    Copy the full SHA
    cccd48d View commit details
    Browse the repository at this point in the history

  2. test BAZEL printout 

    Signed-off-by: sven1977 <svenmika1977@gmail.com>
    @sven1977
    sven1977 committed May 17, 2024
    Configuration menu
    Copy the full SHA
    e96b9ce View commit details
    Browse the repository at this point in the history

  3. Commented out off-policy multi-agent examples that were not learning. 

    Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 17, 2024
    Configuration menu
    Copy the full SHA
    9d409dd View commit details
    Browse the repository at this point in the history

  4. Merge branch 'change_ma_buffer_to_use_list_of_episodes' of github.com… 

    …:simonsays1980/ray into change_ma_buffer_to_use_list_of_episodes

Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>
    @simonsays1980
    simonsays1980 committed May 17, 2024
    Configuration menu
    Copy the full SHA
    41d0b18 View commit details
    Browse the repository at this point in the history