Sentry randomly crash application

[mihalikv]

How do you use Sentry?

Self-hosted/on-premise

Version

2.2.0

Steps to Reproduce

requirements.txt

settings.py:

def profiles_sampler(sampling_context):
    return 1


def traces_sampler(sampling_context):
    return 1


if SENTRY_DSN:
    sentry_sdk.init(
        dsn=SENTRY_DSN,
        integrations=[
            DjangoIntegration(
                transaction_style="function_name",
                middleware_spans=True,
                signals_spans=True,
                cache_spans=False,
            ),
            CeleryIntegration(),
            RedisIntegration(),
        ],
        traces_sampler=traces_sampler,
        profiles_sampler=profiles_sampler,
        send_default_pii=True,
    )

We have Django application with following middleware, that I suspect is a problem:

class SimpleMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        request.user.my_custom_attribute = {} #here is dict that is filled from Elasticsearch 
        response = self.get_response(request)
        return response

Then anywhere in the code when I access request.user.my_custom_attribute, I get the following exception

My first thought was that problem is in my application code so I put logging message to every line to be able to debug "where my_custom_attrbibute goes". Result from logging was that attribute magically disappear on different places.

Error occured totally random -> with more users -> more request -> more random occurences.

When I disabled sentry, app works as expected.

My tought is that sentry somehow patch/copy User object and custom attribute is not included.

I also tried to do minimal setup but in minimal setup application it worked as expected.

Expected Result

Working application code.
I realize that it is not the best idea to set attributes on user but I still think that sentry should not break application.

Actual Result

[sentrivana]

Hey @mihalikv, thanks for writing in.

To clarify, this my_custom_attribute that you're putting on the user in your middleware example is the user_doc from your Sentry screenshot I assume? And you are randomly getting an AttributeError when trying to access it?

From the SDK point of view our only interaction with request.user as far as I can see is here, where we copy some attributes from it over to the Sentry event. We shouldn't be modifying the user. We are accessing it though, which might potentially be relevant because of the whole Django SimpleLazyObject business.

Could you post the full Python stacktrace for the error from your app's logs?

Do you use any additional auth middleware or any middleware that does something user related?

[mihalikv]

Hi @sentrivana ,

yes you are right my_custom_attribute in example is user_doc in real application.

Full stack trace is(I am not sure, how can I provide more details):

AttributeError: 'User' object has no attribute 'user_doc'
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "project/search/views.py", line 501, in get_rich_data
    result = self.get_rich_data_from_es()
  File "project/search/views.py", line 227, in get_rich_data_from_es
    search = Catalog.search(index=self.index_name).protect_query(Catalog.get_query_filter(self.request.user))
  File "project/search/mappings/default.py", line 134, in get_query_filter
    document_filter = tenant.get_permissions().PERMISSIONS[user_group][cls.__name__]['handler'](licence, user, child_groups=child_groups)
  File "project/search/permissions/tenant.py", line 268, in get_my_products
    country = "sk" if user.user_doc["state"] == "Slovensko" else "cz"
  File "django/utils/functional.py", line 259, in inner
    return func(self._wrapped, *args)

Middleware definition is following:

MIDDLEWARE = [
    'django_tenants.middleware.main.TenantMainMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django_middleware_global_request.middleware.GlobalRequestMiddleware',
    'project.basket.middleware.BasketMiddleware',
    'project.account.middleware.AccountMiddleware',
    'project.account.middleware.SwitchLanguage',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'project.account.middleware.CustomRedirectMiddleware',
]

It's probablly not very helpful, but I can say that in our custom middlewares we are also just accesing user_doc attribute and it's not replaced.

Another notable think is that we are using DRF and DRF process normal request to own custom request and even user is something special code here.

It was our first point of interest while trying to fix it. We even try to monkeypatch it in this way:

OriginalRequest = rest_framework.request.Request

class CustomRequest(OriginalRequest):

    @property
    def user(self):
        user = self._request.user
        user.user_doc = self._request.user_doc
        return user

    @user.setter
    def user(self, value):
        OriginalRequest.user.fset(self, value)

if OriginalRequest.__name__ != "CustomRequest":
    rest_framework.request.Request = CustomRequest

but result was the same.
Fact is that error occures even on normal views and WSGI request so it's probably not a problem of DRF.

[szokeasaurusrex]

Ok, thanks for the information @mihalikv, we will continue investigating

[szokeasaurusrex]

Hey, this error might be the same as #3459. We are planning to release a fix for that issue this week; once the release is out, could you try it out and let us know if the problem is fixed?

[sentrivana]

The release should be out today (2.18.0) -- please try and let us know!