Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Version 3.0.9 Instantly crashing (Mono assertion in garbage collector) #251

Closed
1 task done
kristianvld opened this issue Feb 15, 2023 · 11 comments
Closed
1 task done

Comments

@kristianvld
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Using the new 3.0.9 version that was just release instantly crashes and is not able to run.

Host: Intel NUC
OS: NixOS 21.11 (Porcupine)
Docker: Docker version 20.10.9, build v20.10.9
CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz

System resource usage is low, all other containers and services are running fine.

Running docker run linuxserver/sonarr yields the following error:

[migrations] started
[migrations] no migrations found
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support the app dev(s) visit:
Sonarr: https://sonarr.tv/donate

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[custom-init] No custom files found, skipping...
* Assertion at gc.c:1005, condition `is_ok (error)' not met, function:mono_gc_init_finalizer_thread, Couldn't create thread. Error 0x0 assembly:<unknown assembly> type:<unknown type> member:(null)


=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x556ccc876ffb - mono : 
	0x556ccc87738d - mono : 
	0x556ccc824087 - mono : 
	0x556ccc87657c - mono : 
	0x7fc5e5e80520 - /lib/x86_64-linux-gnu/libc.so.6 : 
	0x7fc5e5ed4a7c - /lib/x86_64-linux-gnu/libc.so.6 : pthread_kill
	0x7fc5e5e80476 - /lib/x86_64-linux-gnu/libc.so.6 : raise
	0x7fc5e5e667f3 - /lib/x86_64-linux-gnu/libc.so.6 : abort
	0x556ccc7e63c4 - mono : 
	0x556cccacb135 - mono : 
	0x556cccae85be - mono : 
	0x556cccae8c73 - mono : monoeg_assertion_message
	0x556ccca65f08 - mono : 
	0x556ccc9539f6 - mono : 
	0x556ccc7e9047 - mono : 
	0x556ccc7f77b6 - mono : mono_main
	0x556ccc7e64df - mono : 
	0x7fc5e5e67d90 - /lib/x86_64-linux-gnu/libc.so.6 : 
	0x7fc5e5e67e40 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main
	0x556ccc7e6b1e - mono : _start

=================================================================
	Telemetry Dumper:
=================================================================
Entering thread summarizer pause from 0x140487942197376x
Finished thread summarizer pause from 0x140487942197376x.
Failed to create breadcrumb file (null)/crash_hash_0x0

Waiting for dumping threads to resume

=================================================================
	External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fc5e5ed4a7c):0x7fc5e5ed4a6c  05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05  ..D.............
0x7fc5e5ed4a7c  41 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00  A..A..=.........
0x7fc5e5ed4a8c  44 0f 46 e8 e9 6d ff ff ff 0f 1f 00 48 89 ef e8  D.F..m......H...
0x7fc5e5ed4a9c  10 a8 ff ff e9 29 ff ff ff 0f 1f 00 48 89 ef e8  .....)......H...

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
* Assertion at gc.c:1005, condition `is_ok (error)' not met, function:mono_gc_init_finalizer_thread, Couldn't create thread. Error 0x0 assembly:<unknown assembly> type:<unknown type> member:(null)


=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

And then the "Native Crash Reporting" keeps repeating about once every second for as long as the container is running. The web server never starts, when mounting config volumes for logs and such they are also empty (I believe it crashes before it even opens any log files).
Running a test with 3.0.8 works just fine as expected. Pinning this version in my current docker-compose file also fixes my current stack.
Example output of working docker run linuxserver/sonarr:3.0.8

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service 00-legacy: starting
s6-rc: info: service 00-legacy successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/01-envfile
cont-init: info: /etc/cont-init.d/01-envfile exited 0
cont-init: info: running /etc/cont-init.d/01-migrations
[migrations] started
[migrations] no migrations found
cont-init: info: /etc/cont-init.d/01-migrations exited 0
cont-init: info: running /etc/cont-init.d/02-tamper-check
cont-init: info: /etc/cont-init.d/02-tamper-check exited 0
cont-init: info: running /etc/cont-init.d/10-adduser
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support the app dev(s) visit:
Sonarr: https://sonarr.tv/donate

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

cont-init: info: /etc/cont-init.d/10-adduser exited 0
cont-init: info: running /etc/cont-init.d/30-config
cont-init: info: /etc/cont-init.d/30-config exited 0
cont-init: info: running /etc/cont-init.d/90-custom-folders
cont-init: info: /etc/cont-init.d/90-custom-folders exited 0
cont-init: info: running /etc/cont-init.d/99-custom-scripts
[custom-init] no custom files found exiting...
cont-init: info: /etc/cont-init.d/99-custom-scripts exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-mods: starting
s6-rc: info: service init-mods successfully started
s6-rc: info: service init-mods-package-install: starting
s6-rc: info: service init-mods-package-install successfully started
s6-rc: info: service init-mods-end: starting
s6-rc: info: service init-mods-end successfully started
s6-rc: info: service init-services: starting
s6-rc: info: service init-services successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun sonarr (no readiness notification)
s6-rc: info: service legacy-services successfully started
s6-rc: info: service 99-ci-service-check: starting
[ls.io-init] done.
... (continues normally) ...

I also tried starting the 3.0.9 version on my mac and there it started fine, so I assume there might be some hardware comparability problem.

I am not super familiar with development in Mono so not entirely sure what info might be useful for such an issue, but I can provide more system details or logs if necessary.

Expected Behavior

I expect Sonarr to not instantly crash on startup.

Steps To Reproduce

Run docker run linuxserver/sonarr:3.0.9 and Sonarr instantly crashes.

Environment

- OS: 21.11 (Porcupine)
- How docker service was installed:
`/etc/nixos/configuration.nix`:

...
environment.systemPackages = with pkgs; [
  ...
  docker-compose
  ...
];
...

CPU architecture

x86-64

Docker creation

Normally using docker compose:

version: "3.4"
services:
  sonarr:
    image: linuxserver/sonarr:latest
    restart: unless-stopped
    environment:
      - PUID=${PUID} 
      - PGID=${PGID} 
      - TZ=${TZ} 
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config/sonarr:/config
      - ./media:/media
    ports:
      - 8989:8989

But also simply running `docker run --rm linuxserver/sonarr:3.0.9` causes the same results.

Container logs

[migrations] started
[migrations] no migrations found
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support the app dev(s) visit:
Sonarr: https://sonarr.tv/donate

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[custom-init] No custom files found, skipping...
* Assertion at gc.c:1005, condition `is_ok (error)' not met, function:mono_gc_init_finalizer_thread, Couldn't create thread. Error 0x0 assembly:<unknown assembly> type:<unknown type> member:(null)


=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x556ccc876ffb - mono : 
	0x556ccc87738d - mono : 
	0x556ccc824087 - mono : 
	0x556ccc87657c - mono : 
	0x7fc5e5e80520 - /lib/x86_64-linux-gnu/libc.so.6 : 
	0x7fc5e5ed4a7c - /lib/x86_64-linux-gnu/libc.so.6 : pthread_kill
	0x7fc5e5e80476 - /lib/x86_64-linux-gnu/libc.so.6 : raise
	0x7fc5e5e667f3 - /lib/x86_64-linux-gnu/libc.so.6 : abort
	0x556ccc7e63c4 - mono : 
	0x556cccacb135 - mono : 
	0x556cccae85be - mono : 
	0x556cccae8c73 - mono : monoeg_assertion_message
	0x556ccca65f08 - mono : 
	0x556ccc9539f6 - mono : 
	0x556ccc7e9047 - mono : 
	0x556ccc7f77b6 - mono : mono_main
	0x556ccc7e64df - mono : 
	0x7fc5e5e67d90 - /lib/x86_64-linux-gnu/libc.so.6 : 
	0x7fc5e5e67e40 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main
	0x556ccc7e6b1e - mono : _start

=================================================================
	Telemetry Dumper:
=================================================================
Entering thread summarizer pause from 0x140487942197376x
Finished thread summarizer pause from 0x140487942197376x.
Failed to create breadcrumb file (null)/crash_hash_0x0

Waiting for dumping threads to resume

=================================================================
	External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fc5e5ed4a7c):0x7fc5e5ed4a6c  05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05  ..D.............
0x7fc5e5ed4a7c  41 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00  A..A..=.........
0x7fc5e5ed4a8c  44 0f 46 e8 e9 6d ff ff ff 0f 1f 00 48 89 ef e8  D.F..m......H...
0x7fc5e5ed4a9c  10 a8 ff ff e9 29 ff ff ff 0f 1f 00 48 89 ef e8  .....)......H...

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
* Assertion at gc.c:1005, condition `is_ok (error)' not met, function:mono_gc_init_finalizer_thread, Couldn't create thread. Error 0x0 assembly:<unknown assembly> type:<unknown type> member:(null)


=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x5622d501fffb - mono : 
	0x5622d502038d - mono : 
	0x5622d4fcd087 - mono : 
	0x5622d501f57c - mono : 
	0x7ff784ac5520 - /lib/x86_64-linux-gnu/libc.so.6 : 
	0x7ff784b19a7c - /lib/x86_64-linux-gnu/libc.so.6 : pthread_kill
	0x7ff784ac5476 - /lib/x86_64-linux-gnu/libc.so.6 : raise
	0x7ff784aab7f3 - /lib/x86_64-linux-gnu/libc.so.6 : abort
	0x5622d4f8f3c4 - mono : 
	0x5622d5274135 - mono : 
	0x5622d52915be - mono : 
	0x5622d5291c73 - mono : monoeg_assertion_message
	0x5622d520ef08 - mono : 
	0x5622d50fc9f6 - mono : 
	0x5622d4f92047 - mono : 
	0x5622d4fa07b6 - mono : mono_main
	0x5622d4f8f4df - mono : 
	0x7ff784aacd90 - /lib/x86_64-linux-gnu/libc.so.6 : 
	0x7ff784aace40 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main
	0x5622d4f8fb1e - mono : _start

=================================================================
	Telemetry Dumper:
=================================================================
Entering thread summarizer pause from 0x140701059260544x
Finished thread summarizer pause from 0x140701059260544x.
Failed to create breadcrumb file (null)/crash_hash_0x0

Waiting for dumping threads to resume

=================================================================
	External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7ff784b19a7c):0x7ff784b19a6c  05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05  ..D.............
0x7ff784b19a7c  41 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00  A..A..=.........
0x7ff784b19a8c  44 0f 46 e8 e9 6d ff ff ff 0f 1f 00 48 89 ef e8  D.F..m......H...
0x7ff784b19a9c  10 a8 ff ff e9 29 ff ff ff 0f 1f 00 48 89 ef e8  .....)......H...

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
* Assertion at gc.c:1005, condition `is_ok (error)' not met, function:mono_gc_init_finalizer_thread, Couldn't create thread. Error 0x0 assembly:<unknown assembly> type:<unknown type> member:(null)


=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
... infinity repeating about once every second ...
@github-actions
Copy link

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

@thespad
Copy link
Member

thespad commented Feb 15, 2023

Please try https://docs.linuxserver.io/faq#jammy

@kristianvld
Copy link
Author

Upgrading docker to a newer version past 20.10.10 seems to have fixed the issue.
Cheers!

@thespad thespad pinned this issue Feb 20, 2023
@thewikki
Copy link

thewikki commented Mar 5, 2023

I am having this same issue. I only found these same logs in portainer though when i was looking to revert the image to a previous image(which I have and am now up and working.) However, docker version for me I am currently on Docker version 23.0.1, build a5ee5b1... so upgrading docker to a newer version past 20.10.10 isnt an option currently. Anything else I can do to resolve this with the :latest tag?

@mikecardwell
Copy link

mikecardwell commented Mar 5, 2023

I have this same issue. I'm on Debian 10 using the distro version of Docker.

I find it bizarre that upgrading docker would fix this. Why? There must be some way of running the image on the distro version of Docker on Debian 10

@thespad
Copy link
Member

thespad commented Mar 5, 2023

It's not bizarre at all, Docker uses seccomp to limit which syscalls a container can make and has a default profile it ships with releases that whitelists a bunch of necessary ones. If a new syscall gets introduced, but Docker doesn't have it included in the profile then your containers can't use it and if, as in this case, the syscall replaces an older one, new distros using it will break previously functional containers.

As per the FAQ if you can't/won't upgrade Docker or you have some other combination of factors (ancient kernel, podman, selinux, etc.) which are causing the issue to manifest even with the latest Docker engine, you can run the container with --security-opt seccomp=unconfined to have it bypass the seccomp profile restrictions.

@mikecardwell
Copy link

Adding seccomp=unconfined to my docker-compose file fixed the issue. I'm sure that recommending people use the latest image whilst rolling out a change which breaks on an extremely commonly used distro made sense to somebody.

@thespad
Copy link
Member

thespad commented Mar 5, 2023

Docker engine 20.10.10, which added support for the syscall in question, was released 2021-10-25. We generally assume that 18 months is a sufficient buffer for most people to have updated, and where they can't/won't, we put together an FAQ entry with alternative options.

Docker's official position, and our recommendation to anyone who uses our images, is that you should use their repositories for installing where possible, because the Distro repos typically vary from extremely out of date to laughably out of date.

@aptalca
Copy link
Member

aptalca commented Mar 5, 2023

I'm sure that recommending people use the latest image whilst rolling out a change which breaks on an extremely commonly used distro made sense to somebody.

No need for passive aggressive comments. We didn't roll out a change. We simply updated the baseimages to the latest LTS/stable builds. If your distro is behind on their updates, you should take it up with them.

You could have just as easily said thanks.

@thewikki
Copy link

thewikki commented Mar 5, 2023

so using ubuntu... on docker version 23.0.1... however... on ubuntu 20.04.5 LTS... Do I just need to upgrade my distro? or is there other stuff that I also need to update to get this working on the latest for Sonarr?

@thespad
Copy link
Member

thespad commented Mar 5, 2023

so using ubuntu... on docker version 23.0.1... however... on ubuntu 20.04.5 LTS... Do I just need to upgrade my distro? or is there other stuff that I also need to update to get this working on the latest for Sonarr?

We'd need to establish why you're having problems despite having an up-to-date Docker install. Please open a new issue, fill in the template with as much detail as you can and we'll try and work out where the issue lies. 20.04 shouldn't be old enough that it's a kernel problem, unless you've upgraded from a previous LTS release and somehow never updated the kernel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants