chore: bump pyspark version

[project-defiant]

✨ Context

This PR closes #3189 and links to opentargets/orchestration#94

🛠 What does this PR implement

This PR allows pyspark>=3.5.0, <3.6 along with newest hail version, that supports it.

🙈 Missing

🚦 Before submitting

• Do these changes cover one single feature (one change at a time)?
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes?
• Did you make sure there is no commented out code in this PR?
• Did you follow conventional commits standards in PR title and commit messages?
• Did you make sure the branch is up-to-date with the dev branch?
• Did you write any new necessary tests?
• Did you make sure the changes pass local tests (make test)?
• Did you make sure the changes pass pre-commit rules (e.g poetry run pre-commit run --all-files)?

[ireneisdoomed]

Thank you!
2 questions:

1. Should we also give support to pyspark versions lower than 3.5? Afaik, there is nothing in the API that is not compatible. I know we have custom code to compare dataframes, something that Pyspark 3.5 supports
2. We need to update the Dataproc version too. Should we make a pipeline run to test nothing breaks?

[project-defiant]

Thank you! 2 questions:

1. Should we also give support to pyspark versions lower than 3.5? Afaik, there is nothing in the API that is not compatible. I know we have custom code to compare dataframes, something that Pyspark 3.5 supports
2. We need to update the Dataproc version too. Should we make a pipeline run to test nothing breaks?

These are very nice points!

1. I am aftraid, that poetry will not allow us to supoport more pyspark versions, latest hail requires version between 3.5 to 3.6, otherwise we could do it.
2. Yes I am planning to do that, just right after I finish the harmonisation :). Let's wait for this check before we merge.