Line Numbers Not Showing for Mobile Exceptions (Android & some iOS)

[cstavitsky]

Opening this issue on behalf of a trialing Sentry customer. Bug Report may not be quite accurate but we aren't clear as to why line numbers aren't showing for Unity exceptions captured by Sentry on Android (and some on iOS), and are looking for help troubleshooting.

Environment

How do you use Sentry?
Sentry SaaS (sentry.io)

Which version of the SDK?

sentry.dotnet.unity
--
Version | 2.0.1+318ab00a2245007aba0d09b24cbf39dc6daabafd

How did you install the package? (Git-URL, Assetstore)
Git-URL

Which version of Unity?
2021.3.19f1

Is this happening in Unity (editor) or on a player like Android, iOS, Windows?
Android/iOS

Steps to Reproduce

Triggered errors in Unity mobile game in iOS and Android.

• They confirmed that they are using IL2CPP, which should make this IL2CPP Line Numbers documentation relevant.- The customer has uploaded debug symbols for their Android project
  • (Per the docs on IL2CPP Line Numbers, “The support for line numbers in C# Exceptions with IL2CPP relies on the debug symbol upload. With the automated debug symbol upload enabled, the Sentry SDK will upload the line mapping automatically.”)
  • Link to Debug symbols, project, etc included in internal customer case, also attached in associated Jira ticket -- reach out to @cstavitsky on Slack if you have trouble finding these
• (Per the docs on IL2CPP Line Numbers, line numbers should be enabled by default: “This feature is enabled by default. You can opt out of it through the editor configuration window by selecting Tools -> Sentry -> Advanced -> IL2CPP line numbers or programmatically.”)
  • Based on this, I asked the team to confirm their editor configuration and they provided this screenshot. My understanding is that IL2CPP line numbers are enabled correctly since the box is checked:

• Here is their Sentry config, which looks normal to me:

Expected Result

Line numbers would be associated for exceptions triggered in both Android and iOS.

Actual Result

Line numbers are not associated.

Additional Context

we tried to rebuild several times and test with the apk, there was line numbers. We also tested the APK that come with current Live build and it worked, however it didn't work with the play store version (that we assume it come from the .aab build)

We just try to submit the new build (tested with apk version) and will test it when it available on internal beta play store to confirm it.

• our live build will output both apk file and aab file.
• we submit the aab file to google play store
• the current live build download from doesn't show the line numbers
• BUT when we test with the apk file from first step, it has the line numbers displayed.

So we assume that there is something different with apk and aab build. We just submitted a new beta build to confirm it again tomorrow.

after they tried that, they wrote:

On the Sentry Dashboard, the number of lines of code is always shown when we use an APK file to install for testing. However, no number of lines of code is displayed when we install our game from the Google Play Store. The number of lines of code is normally shown on iOS. [settings are shown in the images below. under Steps to Reproduce]

• Full context and event links, project, etc included in internal customer case, also attached in associated Jira ticket -- reach out to @cstavitsky on Slack if you have trouble finding these

Screenshots

Android Example with no line numbers

iOS Example with no line numbers

┆Issue is synchronized with this Jira Improvement by Unito

[bitsandfoxes]

Hey,
the configuration looks fine. So this is nothing that

I'm having trouble summarizing this as there seem to be a couple of issues in one.
On Android:

1. Line numbers work in regular builds/dev-builds when outputting an APK and running that on a device
2. Line numbers do not work when uploading to the google playstore as an aab

On iOS - it never works? It doesn't work in a released build out of the app store?

Just to make sure this isn't it, we have a know limitation regarding line numbers documented here.
But based on the stacktrace in the screenshots this ain't the issue here.

One thing I noticed on the stacktraces of these issues: Both of them seem to originate from plugins. Do those plugins provide debug symbols as part of their packages? Are they just .cs scripts?

Some links to those issues would be greatly appreciated.

[cstavitsky]

I'm having trouble summarizing this as there seem to be a couple of issues in one. On Android:

1. Line numbers work in regular builds/dev-builds when outputting an APK and running that on a device
2. Line numbers do not work when uploading to the google playstore as an aab

@bitsandfoxes thanks for taking a look! Today our customer sent this additional context:

We have more context about the issue:

• We use the following command to distribute the AAB file:

gradle bundleRelease -b build.gradle

• We have tried installing the AAB file on a device and got the following results:

Line of code is shown in the case of:

1. Installing using the adb tool from a MacOS machine with bundleTool and mode=universal:

java -jar ~/bundletool/bundletool.jar build-apks --bundle=/path/to/your_app.aab --output=/path/to/output.apks --mode=universal

2. Uploading the AAB file to Microsoft AppCenter, then downloading and installing it. Note that in AppCenter’s documentation it is mentioned:
   "When you distribute Android Application Bundle (AAB), App Center generates a universal APK, signs it with a generated signing key, and distributes it to a device." Source: link

The line of code is NOT shown in the case of:

• Installing from Google beta/Google Play Store

There is a difference between installing the app from Google and manually installing using Mode Universal. When installing from Google, the AAB itself will be split into smaller APKs as defined in the build.gradle, for example: base-master.apk, base-arm64_v8a.apk, base-armeabi_v7a.apk.
For some reason, the symbol files uploaded to Sentry do not work fully with this method. Below is the list of symbol files uploaded with the build.

From the early days of implementing Sentry, we found that if we only use the “Upload Dev Symbols” and “Upload Sources” options in the Symbols tab of Sentry Unity Tools, the line of code is not displayed even with the APK file. Therefore, we used this command to upload the Symbols after building the app:

export SENTRY_PROPERTIES=./sentry.properties && ./sentry-cli-Darwin-universal debug-files upload --include-sources --il2cpp-mapping [project-path]

We noticed that using the above command results in a more complete upload of symbol files, and the line of code is displayed for the APK file. So, we are currently continuing to use this method of uploading.

That concludes their additional context. To address your other questions:

On iOS - it never works? It doesn't work in a released build out of the app store?

This I will need to clarify with the customer. They came asking about the problem primarily with Android (and while poking around I noticed that some line numbers are also missing). So from their perspective, the original problem is primarily the Android one, I just surfaced what I thought might be a relevant problem.

Just to make sure this isn't it, we have a know limitation regarding line numbers documented here. But based on the stacktrace in the screenshots this ain't the issue here.

I will also need to confirm this with the customer. I'll also encourage them to reply directly here to you to save time!

One thing I noticed on the stacktraces of these issues: Both of them seem to originate from plugins. Do those plugins provide debug symbols as part of their packages? Are they just .cs scripts?

I will encourage the customer to follow up about this as well.

Some links to those issues would be greatly appreciated.

Links to these issues are here in the internal customer case.

[phuongdt88]

@bitsandfoxes
As we can see currently, line numbers are displayed for the iOS production build, except for some cases:

• Async methods
• IEnumerator
• UniRx.Async.UniTask.Run

We will retry these cases with StackTraceMode.Enhanced.

Currently, we are focusing on the issue of line numbers not displaying for the Android production build (installed with the aab file). And this issue simply comes from the .cs files.

[bitsandfoxes]

Thanks for the additional feedback.
We'd definitely expect those to work for async methods and coroutines. I think we even have samples for those. We'll check up on that.

Issues with UniRx seem familiar to Cysharp's UniTask which is a known limitation.
A bit of context for those: At the time of an exception, the SDK requests the instruction address from Unity's IL2CPP backend. When originating from UniRx and UniTask Unity always returns addresses as 0x0.

[hunglvgeargames]

Hi @bitsandfoxes, just wanted to let you know that we still need support on the Android production build with aab file downloaded from google play store that cannot show the line numbers on Sentry UI dashboard.

[bitsandfoxes]

@hunglvgearinc, definitely!

[bitsandfoxes]

As an update and since we're rolling this up from the Android side:
I'm testing this with Unity 2021.3.38f1 and checking for error events with their line numbers intact:

• APK: works
• AAB: works

So this is working locally, which leads me to believe that something is going on on the Google Play Store side of things.

[smeubank]

I am re-opening this issue for the time being to correctly reflect feedback for at least one customer, and we need to budget time to investigate

[bitsandfoxes]

I've submitted and setup our own sample via the Google Play Console to be able to fully test this. I need to wait for us to pass the review to be able to download the sample via the playstore. Directly downloading the apk from the dashboard seems to be no different than a local dev-build.
I'll update here once I know more.

[bitsandfoxes]

I'm failing to reproduce the missing line numbers in builds downloaded from the Play Store.