Ray Launcher Plugin support for Python 3.11

[Pedrexus]

Motivation

Ray Launcher Plugin is now being supported in python 3.11 and Ray 2.7

Have you read the Contributing Guidelines on pull requests?

Yes

Test Notes

The tests are now passing. Updating the libraries to the newer version solved most issues.

AWS Ray Launcher is being deprecated as we could not execute tests within AWS infrastructure.

Related Issues and PRs

#2489

[facebook-github-bot]

Hi @Pedrexus!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[Jasha10]

Looks some of the test failures are due to a deprecation warning.
For example, on the test_linux-3.10 ci job:

FAILED tests/test_submitit_launcher.py::TestSubmititLauncher::test_sweep_1_job_strict[submitit_local-overrides0] - DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html

We are already ignoring this deprecation warning for the core tests; see line 511 of the noxfile.
I think it should be possible to do the same thing for plugin tests by modifying line 581 of the noxfile:

diff --git a/noxfile.py b/noxfile.py
index 693717dfe5..7a62ed1026 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -578,7 +578,11 @@ def test_selected_plugins(session: Session, selected_plugins: List[Plugin]) -> N
     # Run tests for all installed plugins
     for plugin in selected_plugins:
         session.chdir(plugin.abspath)
-        run_pytest(session)
+        run_pytest(
+            session,
+            ".",
+            "-W ignore:pkg_resources is deprecated as an API:DeprecationWarning",
+        )
 
 
 @nox.session(python="3.8")  # type: ignore

[Pedrexus]

it seems that adding "-W ignore:pkg_resources is deprecated as an API:DeprecationWarning" did not work. Other libraries do not have their tests passing.

[Jasha10]

Looks like there's one more linter issue:

nox > [2023-12-25 05:04:39,096] Command isort . --check --diff failed with exit code 1:
ERROR: /home/circleci/project/plugins/hydra_ray_launcher/tests/test_ray_aws_launcher.py Imports are incorrectly sorted and/or formatted.
--- /home/circleci/project/plugins/hydra_ray_launcher/tests/test_ray_aws_launcher.py:before     2023-12-25 04:52:13.966308
+++ /home/circleci/project/plugins/hydra_ray_launcher/tests/test_ray_aws_launcher.py:after      2023-12-25 05:04:39.064854
@@ -6,11 +6,11 @@
 import subprocess
 import sys
 import tempfile
+from importlib.metadata import version
 from pathlib import Path
 from typing import Any, Dict, Generator, Optional
 
 import boto3  # type: ignore
-from importlib.metadata import version
 from botocore.exceptions import NoCredentialsError, NoRegionError  # type: ignore
 from hydra.core.plugins import Plugins
 from hydra.plugins.launcher import Launcher
Skipped 1 files
nox > [2023-12-25 05:04:39,096] Session lint_plugins-3.11(hydra-ray-launcher) failed.

[Jasha10]

it seems that adding "-W ignore:pkg_resources is deprecated as an API:DeprecationWarning" did not work. Other libraries do not have their tests passing.

I'll submit a PR to address this shortly.

[Pedrexus]

I was able to fix the lint issue via pre-commit run --files ./plugins/**/*

[bryant1410]

Hey, I just wanted to share that some of these changes overlap with those in #2713, which I believe introduces a simpler change that can be merged already (I've been using that branch fine for a while), and then move to those in this PR. What do you think?

[bryant1410]

boto3 is constantly updating minor versions, and this constraint may be too harsh. What about the following?

Suggested change

	"boto3>=1.28,<1.29",
	"boto3>=1.28,<2",

[Pedrexus]

@Jasha10 should we close this PR given #2713 has been merged?

[Jasha10]

Yes. Thank you @Pedrexus.