[CR][ENG-5681] Great Big Python Upgrade #10648

felliott · 2024-06-25T12:42:24Z

Purpose

Double our pythons, Lacoön-style, by bumping from py3.6 to py3.12. Much dependency upgrades included!

Changes

Bump base python version from py3.6 to py3.12.
Switch to using poetry for dependency management
Bump most (not all) dependencies to their maximum version as of mid-March.
Significantly update Dockerfile
Upgrade Django to v4.2
Generate test summary reports in CI
1240 files changed, 23012 insertions(+), 22362 deletions(-)
- bulk of changes include: updates to f-strings, switching to basic asserts, tidying constructors, removing future

QA Notes

Full QA regression required, alongside selenium runs

Documentation

Invoke v0.20.0 started converting underscores in task names to hyphens. invoke test_addons_api is now invoke test-addons-api when called on the command line.

Side Effects

None whatsoever.

Ticket

This ticket: https://openscience.atlassian.net/browse/ENG-5681
Epic link: https://openscience.atlassian.net/browse/ENG-4145

[ENG-5336] Closes #10566

- bump version for the transitions module; - add new modules: pillow, geoip2;

- added a new parameter to specify the encoding of strings in constants for nameparser module;

ENG-5340 Closes #10567

[ENG-5456] Closes #10573

* update github deps to pull from COS repos

…rade [ENG-5332] Closes #10560

* Merge develop into feature/python-upgrade * Update module usage changes * Run pyupgrade over new code

[ENG-5364] Closes #10574

- replace ssl_cert_reqs with tlsAllowInvalidCertificates - remove dep from reqs.txt; update in addons/wiki/reqs.txt

mfraezz

This GitHub page is becoming unresponsive due to all the open files, so I'll be chunking the review. This chunk covers .dotfiles through /addons, the next chunk will begin with /admin. No need to respond in whole or in part to the CR until all the chunks are through.

So far, generally looks good. Mostly nit-picking about f-string consistency, but a couple open questions, and a couple potential concerns.

mfraezz · 2024-06-25T14:44:00Z

.travis.yml

@@ -3,7 +3,7 @@
 language: python

 python:
- - "3.6"
+ - "3.10"


3.12?

Also, since we're using GHA in favor of TCI, this file could probably be deleted.

Will be covered by Phase X ticket: https://openscience.atlassian.net/browse/ENG-5753

mfraezz · 2024-06-25T14:53:17Z

README-docker-compose.md

@@ -334,7 +334,7 @@ resetting docker. To back up your database, follow the following sequence of com

 1. Install Postgres on your local machine, outside of docker. (eg `brew install postgres`) To avoid migrations, the
 version you install must match the one used by the docker container.
- ([as of this writing](https://github.com/CenterForOpenScience/osf.io/blob/ce1702cbc95eb7777e5aaf650658a9966f0e6b0c/docker-compose.yml#L53), Postgres 9.6)
+ ([as of this writing](https://github.com/CenterForOpenScience/osf.io/blob/ce1702cbc95eb7777e5aaf650658a9966f0e6b0c/docker-compose.yml#L53), Postgres 15)


This commit SHA still points to 9.6, it should probably be updated, but perhaps not until squashing this PR.

Will be covered by Phase X ticket: https://openscience.atlassian.net/browse/ENG-5890

mfraezz · 2024-06-25T15:20:37Z

addons/base/generic_views.py

+ 'name': path.replace('All Files', '') if path != '/' else '/ (Full {})'.format(
 addon_full_name
 ),


In keeping with other f-string's, maybe something like

'name': path.replace('All Files', '') if path != '/' else f'/ (Full {addon_full_name})',

mfraezz · 2024-06-25T15:29:07Z

addons/base/models.py

+ 'Because you have not configured the {addon} add-on, your authentication will not be '
+ 'transferred to the forked {category}. You may authorize and configure the {addon} add-on '
+ 'in the new fork on the settings page.'
 ).format(
 addon=self.config.full_name,
 category=node.project_or_component,
 )


Should this (and other similar messages, both in this file and in specific <addon>/models.py files) use f-strings as well, rather than .format calls?

mfraezz · 2024-06-25T16:00:44Z

addons/base/tests/models.py

- assert_false(fork.get_addon(self.short_name).has_auth)
- assert_false(self.user_settings.verify_oauth_access(fork, self.external_account))
+ assert not fork.get_addon(self.short_name).has_auth
+ assert not self.user_settings.verify_oauth_access(fork, self.external_account)


I was going to leave a nit here about how False is not None, which could lead to expectation discrepancies, but I checked the source for assert_false and saw that it didn't care about that either. Carry on.

addons/owncloud/views.py

mfraezz · 2024-06-26T19:20:15Z

addons/s3/tests/test_view.py


 @mock.patch('addons.s3.views.utils.create_bucket')
 def test_create_bucket_fail(self, mock_make):
- error = S3ResponseError(418, 'because Im a test')
+ error = NoCredentialsError(operation_name='create_bucket')
 error.message = 'This should work'


Suggested change

error.message = 'This should work'

addons/twofactor/models.py

addons/twofactor/tests/test_models.py

mfraezz · 2024-06-26T22:21:23Z

addons/wiki/models.py

+ 'Page already exists with name {}'.format(
 new_name,
 )


Suggested change

'Page already exists with name {}'.format(

new_name,

)

f'Page already exists with name {new_name}'

[ENG-5772] Closes #10651

mfraezz

Page is slowing down again, so this second chunk covers through api_tests/base.
Will pick back up with api_tests/brands 📃

mfraezz · 2024-06-27T14:26:55Z

admin/base/settings/defaults.py

@@ -127,13 +125,6 @@
 # Custom user model (extends AbstractBaseUser)
 AUTH_USER_MODEL = 'osf.OSFUser'

-# TODO: Are there more granular ways to configure reporting specifically related to the API?
-RAVEN_CONFIG = {


Is there a new way to wire admin up to Sentry?

mfraezz · 2024-07-01T14:56:32Z

api/base/middleware.py

+enabled = (not settings.DEV_MODE) and settings.SENTRY_DSN
+if enabled:
+ init(
+ dsn=settings.SENTRY_DSN,
+ integrations=[CeleryIntegration(), DjangoIntegration(), FlaskIntegration()],
+ )
+


This seems like it'd be better located in api/base/wsgi.py

mfraezz · 2024-07-01T16:03:33Z

api/base/utils.py

+ if isinstance(value, bool) or value is None:
+ return value
+ return str(value).lower() in TRUTHY
+

 def is_falsy(value):
- return value in FALSY
+ if isinstance(value, bool) or value is None:
+ return not value
+ return str(value).lower() in FALSY


I'm not sure how much it matters, but the behavior here isn't quite the same -- None can now be returned by is_truthy.

Additionally, the conditionals could be removed because, e.g. str(True).lower() == 'true', which is in TRUTHY already.

Previously, is_truthy(None) == is_falsy(None) == False, because None was not in either of the source field values, so I'm not sure if we need to handle it. If we do, could be accomplished by defining FALSY = fields.BooleanField.FALSE_VALUES | {'none'}

mfraezz · 2024-07-01T16:19:55Z

api/caching/tasks.py

@@ -72,6 +72,7 @@ def get_bannable_urls(instance):
 return bannable_urls, parsed_absolute_url.hostname


+# this task is not runnable with celery as instance is not json serializable


Should this be resolved in some way? It may not matter, with Varnish disabled

mfraezz · 2024-07-01T16:36:25Z

api/files/serializers.py

@@ -260,7 +262,9 @@ class BaseFileSerializer(JSONAPISerializer):

 def absolute_url(self, obj):
 if obj.is_file:
- url = furl.furl(settings.DOMAIN).set(
+ # NOTE: furl encoding to be verified later


I think I commented on this previously, but anything to do about this note?

mfraezz · 2024-07-01T19:38:26Z

api_tests/base/test_auth.py

- assert_not_in(self.user.username, res.json)
+ assert res.status_code == 200
+ assert 'email' not in res.json['data']['attributes']
+ assert self.user.username not in res.json


This is correctly "translated," but it seems like this test should actually be checking username not in json.dumps(res.json) to verify it's not in there at all.

mfraezz · 2024-07-01T19:39:08Z

api_tests/base/test_auth.py

- assert_not_in(self.user2.username, res.json)
+ assert res.status_code == 200
+ assert 'email' not in res.json['data']['attributes']
+ assert self.user2.username not in res.json


Ditto above

mfraezz · 2024-07-01T20:03:49Z

api_tests/base/test_filters.py

@@ -263,7 +245,7 @@ def test_InvalidFilterOperator_parameterizes_valid_operators(self):
 r'one of (?P<ops>.+)\.$',
 err.detail
 ).groupdict()['ops']
- assert_equal(ops, 'contains, icontains, eq, ne')
+ assert ops == 'contains, icontains, eq, ne'

 def test_parse_query_params_supports_multiple_filters(self):


This should probably be using pytest.mark.skip rather than commenting the whole thing out. I'm also surprised the lack of a pass didn't anger Flake

mfraezz · 2024-07-01T21:19:04Z

api_tests/base/test_utils.py

- 'push_status_message() should have re-raised the '
- 'original RuntimeError with the original message.')
+ assert str(e) == exception_message, (
+ 'push_status_message() should have re-raised the ' 'original RuntimeError with the original message.'


Suggested change

'push_status_message() should have re-raised the ' 'original RuntimeError with the original message.'

'push_status_message() should have re-raised the original RuntimeError with the original message.'

mfraezz · 2024-07-01T21:19:40Z

api_tests/base/test_utils.py

- False, 'Unexpected Exception from push_status_message when called '
- 'from the v2 API with type "error"')
+ assert False, (
+ 'Unexpected Exception from push_status_message when called ' 'from the v2 API with type "error"'


Suggested change

'Unexpected Exception from push_status_message when called ' 'from the v2 API with type "error"'

'Unexpected Exception from push_status_message when called from the v2 API with type "error"'

mfraezz

The api_tests/ chunk is done, next chunk will resume at conftest.py

mfraezz · 2024-07-02T17:52:02Z

api_tests/draft_registrations/views/test_draft_registration_relationship_institutions.py

+ return '/{}draft_registrations/{}/relationships/institutions/'.format(
 API_BASE, node._id)


Suggested change

return '/{}draft_registrations/{}/relationships/institutions/'.format(

API_BASE, node._id)

return f'/{API_BASE}draft_registrations/{node._id}/relationships/institutions/'

mfraezz · 2024-07-02T17:54:26Z

api_tests/files/serializers/test_file_serializer.py

@@ -58,6 +59,7 @@ def primary_file(self, preprint):
 return preprint.primary_file

 def test_file_serializer(self, file_one, node, node_folder):
+ # TODO: check in qa url encoding


TODO? Several more reading the same in this file

mfraezz · 2024-07-02T18:13:52Z

api_tests/institutions/views/test_institution_auth.py

@@ -67,7 +67,7 @@ def make_payload(
 },
 settings.JWT_SECRET,
 algorithm='HS256'
- ),
+ ).encode(),


I asked about .encode additions in an earlier chunk, but failed to mention that /osf/utils/fields has an ensure_bytes helper. It may or may not be preferable in some situations

mfraezz · 2024-07-02T18:31:48Z

api_tests/nodes/views/test_node_addons.py

- 'owner':
- {'login': 'test name'}
- }))
+ mock_repo = Repository.from_json(


Another case where mock GH data could be DRY'd up a bit

mfraezz · 2024-07-02T18:49:49Z

api_tests/nodes/views/test_node_contributors_list.py

+ ), 'Permissions incorrect for {}. Should not have {} permission.'.format(
+ user_id, api_perm
+ )


Suggested change

), 'Permissions incorrect for {}. Should not have {} permission.'.format(

user_id, api_perm

)

), f'Permissions incorrect for {user_id}. Should not have {api_perm} permission.'

mfraezz · 2024-07-03T01:08:50Z

api_tests/providers/test_reindex_provider.py


 def test_reindex_provider_registration(self, mock_update_share, registration_provider, registration):
 call_command('reindex_provider', f'--providers={registration_provider._id}')
- assert mock_update_share.called_once_with(registration)
+ mock_update_share.assert_called_with(registration)


Suggested change

mock_update_share.assert_called_with(registration)

mock_update_share.assert_called_once_with(registration)

mfraezz · 2024-07-03T01:35:29Z

api_tests/share/test_share_node.py

@@ -184,6 +184,7 @@ def test_skips_no_settings(self, node, user):
 on_node_updated(node._id, user._id, False, {'is_public'})
 assert len(responses.calls) == 0

+ @mark.skip('Synchronous retries not supported if celery >=5.0')


Is there any wat to continue testing this behavior? If not, we might just want to remove these

mfraezz · 2024-07-03T14:04:32Z

api_tests/view_only_links/views/test_view_only_link_nodes.py

+ assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(
 project_two._id)


Unsure if this will lint

Suggested change

assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(

project_two._id)

assert (

res.json['errors'][0]['detail'] ==

f"The node {project_two._id} cannot be affiliated with this View Only Link because the node you're trying to affiliate is not descended from the node that the View Only Link is attached to."

)

mfraezz · 2024-07-03T14:11:25Z

api_tests/view_only_links/views/test_view_only_link_nodes.py

+ assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(
 component_one._id)


Unsure if this will lint

Suggested change

assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(

component_one._id)

assert (

res.json['errors'][0]['detail'] ==

f"The node {component_one._id} cannot be affiliated with this View Only Link because the node you're trying to affiliate is not descended from the node that the View Only Link is attached to."

)

mfraezz · 2024-07-03T14:13:22Z

api_tests/view_only_links/views/test_view_only_link_nodes.py

+ assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(
 project_two._id)


Unsure if this will lint

Suggested change

assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(

project_two._id)

assert (

res.json['errors'][0]['detail'] ==

f"The node {project_two._id} cannot be affiliated with this View Only Link because the node you're trying to affiliate is not descended from the node that the View Only Link is attached to."

)

* Update admin's django-webpack-loader (djwl) dep to point to a forked version. Recent versions of djwl expect a newer format for webpack-stats.json than is generated by the pinned version of the js package webpack-bundle-loader (webbt). The forked version of djwl supports both the older and the newer formats for webpack-stats.json.

[ENG-5818] [ENG-5872]

mfraezz

New chunk is in, this covers everything through osf_tests/. Next chunk will start at poetry.lock and also cover any updates to already-viewed files.

mfraezz · 2024-07-03T14:36:32Z

docker-compose.yml

@@ -77,13 +77,13 @@ services:
 stdin_open: true

 postgres:
- image: postgres:9.6
+ image: postgres:15


Suggested change

image: postgres:15

image: postgres:15.4

mfraezz · 2024-07-03T14:37:25Z

docker-compose.yml

@@ -77,13 +77,13 @@ services:
 stdin_open: true

 postgres:
- image: postgres:9.6
+ image: postgres:15
 command:


The whole command block should now be unnecessary, using a stock image

mfraezz · 2024-07-03T14:40:27Z

docker-compose.yml

- rm -Rf /python3.6/* &&
- cp -Rf -p /usr/lib/python3.6 /
+ - python -m venv /tmp/venv
+ && /tmp/venv/bin/pip install poetry==1.8.0 &&


Could be a me thing, but I can't seem to find poetry==1.8

~/D/c/osf.io ❯❯❯ dlog requirements (osf312) remotes/origin/pr/10648 ✭ ✱ ◼ osfio-requirements-1 | Collecting poetry==1.8.0 osfio-requirements-1 | Could not find a version that satisfies the requirement poetry==1.8.0 (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.4.0.post1, 0.4.1, 0.4.2, 0.5.0b1, 0.5.0b2, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.6.3b1, 0.6.3b2, 0.6.3b3, 0.6.3b4, 0.6.3b5, 0.6.3b6, 0.6.3b7, 0.6.3, 0.6.4b1, 0.6.4, 0.6.5, 0.7.0b1, 0.7.0b2, 0.7.0b3, 0.7.0b4, 0.7.0, 0.7.1, 0.8.0a0, 0.8.0a1, 0.8.0a2, 0.8.0a3, 0.8.0a4, 0.8.0, 0.8.1a0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5a0, 0.8.5, 0.8.6, 0.9.0a0, 0.9.0a1, 0.9.0a2, 0.9.0a3, 0.9.0, 0.9.1, 0.10.0a0, 0.10.0a1, 0.10.0a2, 0.10.0a3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.11.0a0, 0.11.0a1, 0.11.0a2, 0.11.0a3, 0.11.0a4, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.11.5, 0.12.0a0, 0.12.0a1, 0.12.0a2, 0.12.0a3, 0.12.0a4, 0.12.0a5, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.12.7, 0.12.8, 0.12.9, 0.12.10, 0.12.11, 0.12.12, 0.12.13, 0.12.14, 0.12.15, 0.12.16, 0.12.17, 1.0.0a0, 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0a4, 1.0.0a5, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0b4, 1.0.0b5, 1.0.0b6, 1.0.0b7, 1.0.0b8, 1.0.0b9, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.0.9, 1.0.10, 1.1.0a1, 1.1.0a2, 1.1.0a3, 1.1.0b1, 1.1.0b2, 1.1.0b3, 1.1.0b4, 1.1.0rc1, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.1.8, 1.1.9, 1.1.10, 1.1.11, 1.1.12, 1.1.13, 1.1.14, 1.1.15, 1.2.0a1, 1.2.0a2) osfio-requirements-1 | No matching distribution found for poetry==1.8.0

mfraezz · 2024-07-03T16:29:51Z

framework/auth/core.py

+ 'initials': lambda user: '{surname}, {initial}.'.format(
 surname=user.family_name,
 initial=user.given_name_initial,
 ),


Suggested change

'initials': lambda user: '{surname}, {initial}.'.format(

surname=user.family_name,

initial=user.given_name_initial,

),

'initials': lambda user: f'{user.family_name}, {user.given_name_initial}.',

mfraezz · 2024-07-03T16:35:26Z

framework/auth/views.py

+ status_message = ('If there is an OSF account associated with this unconfirmed email address {}, '
 'a confirmation email has been resent to it. If you do not receive an email and believe '
 'you should have, please contact OSF Support.').format(clean_email)


Suggested change

status_message = ('If there is an OSF account associated with this unconfirmed email address {}, '

'a confirmation email has been resent to it. If you do not receive an email and believe '

'you should have, please contact OSF Support.').format(clean_email)

status_message = (f'If there is an OSF account associated with this unconfirmed email address {clean_email}, '

'a confirmation email has been resent to it. If you do not receive an email and believe '

'you should have, please contact OSF Support.'

mfraezz · 2024-07-08T21:46:16Z

osf_tests/test_user.py

@@ -592,7 +591,7 @@ def test_get_confirmation_url_when_token_is_expired_force(self, random_string):
 random_string.return_value = '54321'

 url = u.get_confirmation_url('foo@bar.com', force=True)
- expected = '{0}confirm/{1}/{2}/'.format(settings.DOMAIN, u._id, '54321')
+ expected = '{}confirm/{}/{}/'.format(settings.DOMAIN, u._id, '54321')


Suggested change

expected = '{}confirm/{}/{}/'.format(settings.DOMAIN, u._id, '54321')

expected = f'{settings.DOMAIN}confirm/{u._id}/54321/'

mfraezz · 2024-07-08T21:47:23Z

osf_tests/test_user.py

+ assert (
 summary['user_display_name'] ==
 'Johnson'
 )


Suggested change

assert (

summary['user_display_name'] ==

'Johnson'

)

assert summary['user_display_name'] == 'Johnson'

mfraezz · 2024-07-08T21:48:27Z

osf_tests/test_user.py

+ assert (
 summary['user_display_name'] ==
 'Rock'
 )


Suggested change

assert (

summary['user_display_name'] ==

'Rock'

)

assert summary['user_display_name'] == 'Rock'

mfraezz · 2024-07-08T21:55:22Z

osf_tests/test_user.py

+ assert (
 user.get_activity_points() == get_total_activity_count(user._primary_key)
 )


Suggested change

assert (

user.get_activity_points() == get_total_activity_count(user._primary_key)

)

assert user.get_activity_points() == get_total_activity_count(user._primary_key)

mfraezz · 2024-07-08T21:59:48Z

osf_tests/test_user.py

+ assert (
 UserSessionMap.objects.filter(user=user).count() == 0
 )


Suggested change

assert (

UserSessionMap.objects.filter(user=user).count() == 0

)

assert UserSessionMap.objects.filter(user=user).count() == 0

mfraezz

The final chunk is complete

Overall, mostly very minor issues, with a couple noted exceptions. I also tried to keep a running list of places where things changed just enough that they might require special attention during QA:

- OwnCloud urls with scheme (e.g. http://place.cloud)
- S3, broadly
- 2FA, ensure existing configs don't break across upgrade
- Admin config for sentry
- Django-Sendgrid emails working
- Old webtest forms -- resend confirmation, forgot password, etc

With that, pass complete 😩

mfraezz · 2024-07-08T22:06:43Z

pyproject.toml

+elasticsearch2 = "2.5.1"
+elasticsearch = "6.8.2" # max version to support elasticsearch6
+elasticsearch-dsl = "6.4.0" # max version to support elasticsearch6
+elastic-transport = "8.13.0" # hopefully, this won't break things, more thorough is needed


Is any more thorough still needed?

mfraezz · 2024-07-08T22:07:47Z

pyproject.toml

+django-celery-results = "2.5.1"
+pyjwe = "1.0.0"
+# Required by pyjwe and ndg-httpsclient
+# Building wheel for cryptography >= 3.4.0 requires a Rust version incompatible with Docker base image.


This seems to be no longer true

mfraezz · 2024-07-08T22:29:30Z

pyproject.toml

+jsonschema = "4.21.1"
+django-guardian = "2.4.0"
+# Admin requirements
+django-webpack-loader = {git = "https://github.com/felliott/django-webpack-loader.git", rev = "af8438c2da909ec9f2188a6c07c9d2caad0f7e93"}


Should we move this to COS-forks?

mfraezz · 2024-07-08T22:47:16Z

requirements.txt

+hashids==1.3.1
+pyjwt==2.8.0
+django-celery-beat==2.6.0
+django-celery-results==2.5.1
 pyjwe==1.0.0
 # Required by pyjwe and ndg-httpsclient
 # Building wheel for cryptography >= 3.4.0 requires a Rust version incompatible with Docker base image.


Ditto, guessing this comment is now incorrect

mfraezz · 2024-07-08T22:48:16Z

requirements.txt

-django-webpack-loader==0.5.0
-sendgrid-django==2.0.0
+# django-webpack-loader==3.1.0
+git+https://github.com/felliott/django-webpack-loader@feature/old-style-peanut-butter


Ditto, should this be in COS-Forks?

mfraezz · 2024-07-10T18:54:49Z

tests/test_views.py


 def test_external_login_confirm_email_get_with_another_user_logged_in(self):
+ # TODO: check in qa url encoding


Is this TODO still relevant?

mfraezz · 2024-07-10T19:00:38Z

tests/test_views.py

- self.auth = Auth(self.user)
+ self.auth = (self.user.username, password)


I'm not sure why this change was necessary, it seems like it may change test behavior

mfraezz · 2024-07-10T19:01:29Z

tests/test_views.py


 @mock.patch('website.mails.send_mail')
 def test_external_login_confirm_email_get_create(self, mock_welcome):
- assert_false(self.user.is_registered)
+ # TODO: check in qa url encoding


Ditto re. TODO

mfraezz · 2024-07-10T20:38:39Z

tests/test_views.py


 # successfully reset password
 @pytest.mark.enable_enqueue_task
 @mock.patch('framework.auth.cas.CasClient.service_validate')
 def test_can_reset_password_if_form_success(self, mock_service_validate):
+ # TODO: check in qa url encoding


Ditto re. TODO

mfraezz · 2024-07-10T20:41:52Z

tests/test_views.py

@@ -5097,12 +5043,11 @@ def test_reset_password_get_returns_403(self):
 )
 res = self.app.get(
 reset_password_get_url,
- expect_errors=True,
- headers={
+ headers={


Suggested change

headers={

headers={

* many f-string updates * remove/update outdated comments * remove unnecessary kwargs * fix furl usage + use ensure_str

[ENG-5681]

* resolve conflicts in docker-compose * update furl.furl in addons.base.views

[ENG-5754]

* Convert categories from a list of strings to a list of `Category` objects * `mail.send` now returns a Response object rather than a tuple of status code and message. * Update tests to reflect above [ENG-5945]

[ENG-5945]

* This was old code that was intended to configure message signing, but was actually a no-op. It still doesn't do anything but now throws a configuration error, so give it the boot.

…upgrade

* newrelic-admin is failing to start on py3.11+, so fall back to manually init-ing it in the wsgi code. See: https://stackoverflow.com/a/75665547

* it's now kebabed rather than squished

into feature/preprints-affiliations * 'develop' of https://github.com/CenterForOpenScience/osf.io: Update CHANGELOG, bump version [CR][ENG-5681] Great Big Python Upgrade (CenterForOpenScience#10648) # Conflicts: # api/nodes/serializers.py # osf/models/mixins.py

into fix-preprint-state-machince2 * 'develop' of https://github.com/CenterForOpenScience/osf.io: update dataverse dep revision to get changes Update CHANGELOG, bump version [CR][ENG-5681] Great Big Python Upgrade (CenterForOpenScience#10648) # Conflicts: # api/actions/serializers.py # api/providers/workflows.py # api_tests/identifiers/views/test_identifier_detail.py # api_tests/preprints/views/test_preprint_list.py # api_tests/reviews/mixins/filter_mixins.py # osf/utils/machines.py # osf/utils/workflows.py # osf_tests/test_schema_responses.py

into remove-quickfiles-code * 'develop' of https://github.com/CenterForOpenScience/osf.io: (166 commits) update dataverse dep revision to get changes Update CHANGELOG, bump version [CR][ENG-5681] Great Big Python Upgrade (CenterForOpenScience#10648) Revert "[ENG-3685] Add permissions for withdrawn registration files (CenterForOpenScience#10650)" (CenterForOpenScience#10666) Check Registration READ perms on the Registration - Do not record download metrics for renders Fix signature Allow DOI metadata updates to be queued [ENG-3685] Add permissions for withdrawn registration files (CenterForOpenScience#10650) Update CHANGELOG, bump version [ENG-5030] Preprints Phase 2 - BE (CenterForOpenScience#10617) Update CHANGELOG, bump version Ensure Assumed-HAM users do not get autobanned [ENG-5762] Get GV set up in osf docker configs (CenterForOpenScience#10643) [ENG-5718] Use `make_auth` to avoid assumptions about `auth.user` (CenterForOpenScience#10647) [ENG-5699] Framework for getting Addon Info from GV (CenterForOpenScience#10641) [ENG-5178] Allow unauthenticated users to see public files (CenterForOpenScience#10645) Fix RelationshipPostMakesNoChanges exception in project creation (CenterForOpenScience#10644) [ENG - 5008] Support Unicode and special characters in file names during archiving (CenterForOpenScience#10627) Set Default Resource Type for Registrations to "Study Registration" (CenterForOpenScience#10636) Configurable GV Mock + HMAC Auth (CenterForOpenScience#10623) ... # Conflicts: # addons/base/views.py # api/caching/tasks.py # api_tests/files/views/test_file_detail.py # api_tests/wb/views/test_wb_hooks.py # osf/management/commands/data_storage_usage.py # osf/management/commands/reindex_quickfiles.py # osf/management/commands/transfer_quickfiles_to_projects.py # osf/models/__init__.py # osf/models/private_link.py # osf/models/quickfiles.py # osf/models/user.py # scripts/fix_merged_user_quickfiles.py # tests/test_views.py # website/mails/mails.py # website/search/elastic_search.py # website/settings/defaults.py

- Bump base python version from py3.6 to py3.12. - Switch to using poetry for dependency management - Bump most (not all) dependencies to their maximum version as of mid-March. - Significantly update Dockerfile - Upgrade Django to v4.2 - Generate test summary reports in CI --------- Co-authored-by: Oleh Paduchak <opaduchak@exoft.net> Co-authored-by: Mariia Lychko <lychko.mariia@gmail.com> Co-authored-by: Longze Chen <cslzchen@gmail.com>

opaduchak and others added 30 commits March 20, 2024 05:17

fixed flake8 warnings

473cd97

Merge branch 'feature/dep-group-dev' into feature/python-upgrade

3dc4b04

[ENG-5336] Closes #10566

bump version of gevent

1c47f33

update dep versions for transitions, pillow, geoip2

72b1aae

- bump version for the transitions module; - add new modules: pillow, geoip2;

specify str encoding in nameparser

a4afa04

- added a new parameter to specify the encoding of strings in constants for nameparser module;

remove unused Pystache module

5b4539c

replace maxminddb-geolite2 to more modern geoip2;

faa1f23

remove python-magic

e7f1084

downgrade geoip2 to avoid setuptools build failure

64a8561

Merge branch 'feature/dep-group-catchall' into feature/python-upgrade

78379ea

ENG-5340 Closes #10567

removed _app_ctx_stack and _request_ctx_stack

e46aa12

fied uritemplate.py -> uritemplate

5b28d61

fixed future version of ipdb

39d7dc2

Merge branch 'feature/fix-flask-context' into feature/python-upgrade

57f62fc

[ENG-5456] Closes #10573

remove egap assets missed from prior purge

61e16db

repoint dv-client-pin pin to COS repo

2f97bea

repoint mendeley-python-sdk pin to COS repo

33aecfd

repoint djelme pin to COS repo

d8ed3a9

replace mdx_del_ins fork with markdown-del-ins from pypi

054ba6b

Merge branch 'feature/misc-updates' into feature/python-upgrade

5391352

* update github deps to pull from COS repos

updated furl usage

0e9eb50

added comments in locations that may be affected by furl upgrade

c7864aa

added reasoning behind urrlib3 version

8a4ac00

replace furl / op with .add(path=); minor typo fix

e656848

Merge branch 'feature/dep-group-url-encoding' into feature/python-upg…

74e03e1

…rade [ENG-5332] Closes #10560

Merge remote-tracking branch 'cos/develop' into feature/python-upgrade

db6e960

* Merge develop into feature/python-upgrade * Update module usage changes * Run pyupgrade over new code

removed future usages

0946d6f

removed future from mendeley

7a3bdb0

Merge branch 'feature/dep-group-purge' into feature/python-upgrade

c3a437e

[ENG-5364] Closes #10574

update pymongo

1a9535f

- replace ssl_cert_reqs with tlsAllowInvalidCertificates - remove dep from reqs.txt; update in addons/wiki/reqs.txt

mfraezz reviewed Jun 27, 2024

View reviewed changes

Merge branch 'feature/fix-metrics' into feature/python-upgrade

f47ed16

[ENG-5772] Closes #10651

mfraezz reviewed Jul 1, 2024

View reviewed changes

mfraezz reviewed Jul 3, 2024

View reviewed changes

felliott added 2 commits July 4, 2024 00:34

Merge branch 'feature/fix-admin-pin-djwl' into feature/python-upgrade

da49c86

[ENG-5818] [ENG-5872]

mfraezz reviewed Jul 8, 2024

View reviewed changes

mfraezz reviewed Jul 10, 2024

View reviewed changes

felliott added 14 commits July 11, 2024 01:50

responding to python-upgrade cr

bcbffde

* many f-string updates * remove/update outdated comments * remove unnecessary kwargs * fix furl usage + use ensure_str

Merge branch 'feature/python-upgrade-cr' into feature/python-upgrade

eb2badf

[ENG-5681]

Merge remote-tracking branch 'cos/develop' into feature/python-upgrade

c4b6463

* resolve conflicts in docker-compose * update furl.furl in addons.base.views

New django 4.2 migrations

efed3c9

Merge branch 'feature/dj4.2-migrations' into feature/python-upgrade

a1d6de5

[ENG-5754]

fix sendgrid module usage

3863fa9

* Convert categories from a list of strings to a list of `Category` objects * `mail.send` now returns a Response object rather than a tuple of status code and message. * Update tests to reflect above [ENG-5945]

Merge branch 'feature/unbreak-sendgrid' into feature/python-upgrade

6b32642

[ENG-5945]

remove celery setup_security()

12d1db4

* This was old code that was intended to configure message signing, but was actually a no-op. It still doesn't do anything but now throws a configuration error, so give it the boot.

Merge branch 'feature/celery-config' into feature/python-upgrade

f9c4dae

remove obsolete gevent monkeypatches

12e28b1

Merge branch 'feature/remove-gevent-monkeypatch' into feature/python-…

635dff4

…upgrade

manually init newrelic, since newrelic-admin is bork

11b62d4

* newrelic-admin is failing to start on py3.11+, so fall back to manually init-ing it in the wsgi code. See: https://stackoverflow.com/a/75665547

update celery's max-tasks-per-child arg

f8f1e1e

* it's now kebabed rather than squished

Merge branch 'feature/no-newrelic-admin' into feature/python-upgrade

c285ed4

mfraezz merged commit 8cb3723 into develop Jul 22, 2024
17 checks passed

		@@ -72,6 +72,7 @@ def get_bannable_urls(instance):
		return bannable_urls, parsed_absolute_url.hostname


		# this task is not runnable with celery as instance is not json serializable

	'push_status_message() should have re-raised the ' 'original RuntimeError with the original message.'
	'push_status_message() should have re-raised the original RuntimeError with the original message.'

	'Unexpected Exception from push_status_message when called ' 'from the v2 API with type "error"'
	'Unexpected Exception from push_status_message when called from the v2 API with type "error"'

		return '/{}draft_registrations/{}/relationships/institutions/'.format(
		API_BASE, node._id)

	return '/{}draft_registrations/{}/relationships/institutions/'.format(
	API_BASE, node._id)
	return f'/{API_BASE}draft_registrations/{node._id}/relationships/institutions/'

	mock_update_share.assert_called_with(registration)
	mock_update_share.assert_called_once_with(registration)

		assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(
		project_two._id)

		assert res.json['errors'][0]['detail'] == 'The node {} cannot be affiliated with this View Only Link because the node you\'re trying to affiliate is not descended from the node that the View Only Link is attached to.'.format(
		component_one._id)

	expected = '{}confirm/{}/{}/'.format(settings.DOMAIN, u._id, '54321')
	expected = f'{settings.DOMAIN}confirm/{u._id}/54321/'


		def test_external_login_confirm_email_get_with_another_user_logged_in(self):
		# TODO: check in qa url encoding

		self.auth = Auth(self.user)
		self.auth = (self.user.username, password)

[CR][ENG-5681] Great Big Python Upgrade #10648

[CR][ENG-5681] Great Big Python Upgrade #10648

Conversation

felliott commented Jun 25, 2024

Purpose

Changes

QA Notes

Documentation

Side Effects

Ticket

mfraezz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfraezz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfraezz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfraezz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfraezz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfraezz left a comment •

edited

Loading