Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

martinhaintz · 2024-09-16T15:10:16Z

📜 Description

The request body is now also available via the hint in beforeSend so the user can do further steps.
This is available for failed requests and if tracing is enabled.

options.beforeSend = (event, hint) async {
  final response = hint.get(TypeCheckHint.httpResponse);
  if (response is StreamedResponse) {
    final body = await response.stream.bytesToString();
    // user can now use it
  }
  return event;
};

💡 Motivation and Context

close #2220

💚 How did you test it?

📝 Checklist

I reviewed submitted code
I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPii is enabled
I updated the docs if needed
All tests passing
No breaking changes

🔮 Next steps

github-actions · 2024-09-16T15:11:00Z

	Messages
📖	Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 52ecb5e

codecov · 2024-09-16T15:11:42Z

Codecov Report

Attention: Patch coverage is 98.52941% with 1 line in your changes missing coverage. Please review.

Project coverage is 85.12%. Comparing base (dd16d3e) to head (52ecb5e).

Files with missing lines	Patch %	Lines
dart/lib/src/http_client/sentry_http_client.dart	96.42%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2293      +/-   ##
==========================================
+ Coverage   85.06%   85.12%   +0.06%     
==========================================
  Files         257      258       +1     
  Lines        9191     9225      +34     
==========================================
+ Hits         7818     7853      +35     
+ Misses       1373     1372       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-09-16T15:19:50Z

iOS Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1245.40 ms	1274.94 ms	29.54 ms
Size	8.38 MiB	9.75 MiB	1.37 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`d4d0807`	1246.94 ms	1260.69 ms	13.75 ms
`f9d18f3`	1240.20 ms	1242.78 ms	2.57 ms
`04bd9e6`	1230.78 ms	1250.71 ms	19.94 ms
`051e97a`	1245.94 ms	1249.51 ms	3.57 ms
`b8562d0`	1249.92 ms	1267.56 ms	17.64 ms
`affcf07`	1240.61 ms	1266.49 ms	25.88 ms
`2e8b1e1`	1247.45 ms	1263.67 ms	16.22 ms
`0f067d3`	1245.71 ms	1269.59 ms	23.88 ms
`f1314d5`	1246.46 ms	1270.92 ms	24.46 ms
`e239c83`	1248.40 ms	1269.28 ms	20.89 ms

App size

Revision	Plain	With Sentry	Diff
`d4d0807`	8.33 MiB	9.64 MiB	1.31 MiB
`f9d18f3`	8.29 MiB	9.36 MiB	1.07 MiB
`04bd9e6`	8.33 MiB	9.61 MiB	1.27 MiB
`051e97a`	8.28 MiB	9.34 MiB	1.06 MiB
`b8562d0`	8.33 MiB	9.54 MiB	1.22 MiB
`affcf07`	8.38 MiB	9.70 MiB	1.33 MiB
`2e8b1e1`	8.33 MiB	9.64 MiB	1.31 MiB
`0f067d3`	8.32 MiB	9.52 MiB	1.20 MiB
`f1314d5`	8.10 MiB	9.08 MiB	1004.30 KiB
`e239c83`	8.38 MiB	9.74 MiB	1.36 MiB

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

Revision	Plain	With Sentry	Diff
`2308122`	1231.69 ms	1256.85 ms	25.16 ms
`bc3d263`	1240.98 ms	1259.69 ms	18.71 ms

App size

Revision	Plain	With Sentry	Diff
`2308122`	8.38 MiB	9.73 MiB	1.36 MiB
`bc3d263`	8.38 MiB	9.75 MiB	1.37 MiB

buenaflor · 2024-09-18T13:59:32Z

dart/lib/src/sentry_options.dart

+      "All Request and Responses are now logged, if `sendDefaultPii` is `true` and `maxRequestBodySize` and `maxResponseBodySize` conditions are met.")
  bool captureFailedRequests = true;


let's rethink this, I don't think deprecating this flag is the right way to go

we still want to provide this flag to disable capturing of failed network requests completely, so yes we shouldn't deprecate this

buenaflor · 2024-10-15T11:25:37Z

@kahest JS Replay uses captureNetworkBodies for both request and response

getsentry/sentry-javascript#7589

wdyt if we also adopt this flag?

kahest · 2024-10-15T17:54:09Z

@kahest JS Replay uses captureNetworkBodies for both request and response

getsentry/sentry-javascript#7589

wdyt if we also adopt this flag?

I think this has since been dropped in favor of networkDetailAllowUrls - see docs. I also don't see references to the option in code anymore.

I like the new option - it makes it more granular and explicit, though of course it's more work. It's still only available for JS SR I think, so not widely adopted. If there's no competing option for other HTTP integrations on other SDKs, I'm fine with adopting this

buenaflor · 2024-10-15T18:15:37Z

Ah I missed that, thx. I'll have a look at it

buenaflor · 2024-10-16T11:08:15Z

@kahest I haven't found anything specific in other SDKs that allow sending http response bodies other than replay.

However it's possible a user can do this:

beforeSend: (event, hint) async {
    final response = hint.get(TypeCheckHint.httpResponse);
    if (response is StreamedResponse) {
        final body = getResponseBody(response)
        // user can now use it
    }
}

this also aligns with what the js http integration would like to add: getsentry/sentry-javascript#12544

kahest · 2024-10-16T17:00:58Z

@buenaflor IIUC this would work for http+dio? I like the idea, it's very flexible and allows users to decide based on endpoint etc. if they want to add, but of course it's more complex to use than a switch

buenaflor · 2024-10-17T10:54:43Z

this would work for http+dio

yes but only errors so this won't work for transactions (like the user in the referenced issue wants) because we don't have a system in place to pair hints with transactions

I'll take a look for alternatives

…ttp-client

github-actions · 2024-11-04T09:36:00Z

Android Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	450.11 ms	474.63 ms	24.53 ms
Size	6.49 MiB	7.56 MiB	1.07 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`e5b744f`	302.70 ms	342.17 ms	39.47 ms
`0ac1eed`	370.60 ms	441.54 ms	70.94 ms
`90a08ea`	477.25 ms	534.10 ms	56.85 ms
`0db91cc`	327.85 ms	387.31 ms	59.46 ms
`061fed2`	434.11 ms	506.49 ms	72.38 ms
`4829ad3`	381.55 ms	455.45 ms	73.90 ms
`519423f`	357.00 ms	415.77 ms	58.77 ms
`ddc97ad`	331.45 ms	384.06 ms	52.61 ms
`e3ef570`	389.71 ms	459.16 ms	69.45 ms
`2d3b03d`	309.53 ms	353.40 ms	43.87 ms

App size

Revision	Plain	With Sentry	Diff
`e5b744f`	6.06 MiB	7.09 MiB	1.03 MiB
`0ac1eed`	6.06 MiB	7.03 MiB	990.44 KiB
`90a08ea`	6.49 MiB	7.55 MiB	1.06 MiB
`0db91cc`	5.94 MiB	6.95 MiB	1.01 MiB
`061fed2`	6.52 MiB	7.59 MiB	1.06 MiB
`4829ad3`	6.33 MiB	7.26 MiB	943.11 KiB
`519423f`	6.06 MiB	7.03 MiB	989.24 KiB
`ddc97ad`	6.16 MiB	7.14 MiB	1003.75 KiB
`e3ef570`	6.33 MiB	7.26 MiB	950.38 KiB
`2d3b03d`	6.06 MiB	7.09 MiB	1.03 MiB

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

Revision	Plain	With Sentry	Diff
`bc3d263`	454.46 ms	498.00 ms	43.54 ms
`f9abe16`	456.26 ms	480.49 ms	24.23 ms
`2308122`	422.54 ms	491.65 ms	69.11 ms

App size

Revision	Plain	With Sentry	Diff
`bc3d263`	6.49 MiB	7.57 MiB	1.08 MiB
`f9abe16`	6.49 MiB	7.57 MiB	1.08 MiB
`2308122`	6.49 MiB	7.55 MiB	1.07 MiB

…ttp-client

buenaflor · 2024-11-11T12:16:45Z

before I look at the PR, does this also work for transactions? because afaik beforeSend is not triggered for transactions @martinhaintz

buenaflor · 2024-11-12T12:14:02Z

In order to make this fully compatible with tracing we should add the hint to the beforeSendTransaction API

It is available in our sdks so imo it's safe to implement it:

martinhaintz · 2024-11-13T16:19:58Z

before I look at the PR, does this also work for transactions? because afaik beforeSend is not triggered for transactions @martinhaintz

it wasn't done previously, but I implemented it.

martinhaintz · 2024-11-13T16:20:33Z

In order to make this fully compatible with tracing we should add the hint to the beforeSendTransaction API

It is available in our sdks so imo it's safe to implement it:

.net

javascript

this is by far, the better solution. I will change it tomorrow.

…ttp-client

vaind

This can break existing applications and cause OOM errors in some edge cases. Are there other implementation options that wouldn't have this issue?

For example, only do the capture of the response body with limits on response size (read the response header first)

vaind · 2024-11-14T10:54:01Z

dart/lib/src/utils/http_deep_copy_streamed_response.dart

+  await for (final List<int> chunk in originalResponse.stream) {
+    bufferedData.addAll(chunk);
+  }


This will cause out-of-memory on large responses that are meant to be streamed and not copied to memory.

capture response body for failed requests and if tracing is enabled.

2308122

unified event capturing for request and response

da865b2

buenaflor reviewed Sep 18, 2024

View reviewed changes

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

bc3d263

…ttp-client

martinhaintz added 3 commits November 5, 2024 16:40

make response body accessible via hint

710ab1a

fix warning

f9abe16

add changelog entry

a309f16

martinhaintz marked this pull request as ready for review November 5, 2024 16:57

martinhaintz requested review from krystofwoldrich and stefanosiano as code owners November 5, 2024 16:57

martinhaintz requested a review from buenaflor November 5, 2024 16:58

martinhaintz changed the title ~~capture response body for failed requests and if tracing is enabled in SentryHttpClient~~ Make response body accessible via hint in beforSend callback for SentryHttpClient Nov 5, 2024

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

715ab7d

…ttp-client

Merge branch 'main' into feat/capture-http-response-body-for-sentry-h…

52ecb5e

…ttp-client

vaind requested changes Nov 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

martinhaintz commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor Sep 18, 2024

kahest Oct 15, 2024

buenaflor commented Oct 15, 2024

kahest commented Oct 15, 2024 •

edited

Loading

buenaflor commented Oct 15, 2024

buenaflor commented Oct 16, 2024 •

edited

Loading

kahest commented Oct 16, 2024

buenaflor commented Oct 17, 2024

github-actions bot commented Nov 4, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor commented Nov 11, 2024 •

edited

Loading

buenaflor commented Nov 12, 2024

martinhaintz commented Nov 13, 2024

martinhaintz commented Nov 13, 2024

vaind left a comment •

edited

Loading

vaind Nov 14, 2024

		"All Request and Responses are now logged, if `sendDefaultPii` is `true` and `maxRequestBodySize` and `maxResponseBodySize` conditions are met.")
		bool captureFailedRequests = true;

Make response body accessible via hint in beforSend callback for SentryHttpClient #2293

Are you sure you want to change the base?

Make response body accessible via hint in beforSend callback for SentryHttpClient #2293

Conversation

martinhaintz commented Sep 16, 2024 • edited Loading

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

github-actions bot commented Sep 16, 2024 • edited Loading

codecov bot commented Sep 16, 2024 • edited Loading

Codecov Report

github-actions bot commented Sep 16, 2024 • edited Loading

iOS Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor Sep 18, 2024

Choose a reason for hiding this comment

kahest Oct 15, 2024

Choose a reason for hiding this comment

buenaflor commented Oct 15, 2024

kahest commented Oct 15, 2024 • edited Loading

buenaflor commented Oct 15, 2024

buenaflor commented Oct 16, 2024 • edited Loading

kahest commented Oct 16, 2024

buenaflor commented Oct 17, 2024

github-actions bot commented Nov 4, 2024 • edited Loading

Android Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Previous results on branch: feat/capture-http-response-body-for-sentry-http-client

Startup times

App size

buenaflor commented Nov 11, 2024 • edited Loading

buenaflor commented Nov 12, 2024

martinhaintz commented Nov 13, 2024

martinhaintz commented Nov 13, 2024

vaind left a comment • edited Loading

Choose a reason for hiding this comment

vaind Nov 14, 2024

Choose a reason for hiding this comment

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

Make response body accessible via hint in `beforSend` callback for SentryHttpClient #2293

martinhaintz commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

kahest commented Oct 15, 2024 •

edited

Loading

buenaflor commented Oct 16, 2024 •

edited

Loading

github-actions bot commented Nov 4, 2024 •

edited

Loading

buenaflor commented Nov 11, 2024 •

edited

Loading

vaind left a comment •

edited

Loading