getting a core dump (on linux)

When testing the new wallet you may encounter various problems. Any problem may be one that has already been reported, or something new and not previously seen. The three most useful pieces of diagnostic information in either case are:

• a clear, detailed description of the problem, with steps to reproduce if possible,
• the debug.log,
• a core dump (if crownd (or crown-qt) crashed or is hung) The first two are pretty easy for anyone to provide. The core dump is a bit harder but still manageable. These instructions are primarily intended for masternodes/systemnodes running on the recommended Ubuntu 22 environment, but are adaptable to other linux platforms (or linux wallet daemons).

Years ago it used to be easy to collect a coredump on a linux system. Nowadays, in the pursuit of making things "simpler" for normal users, it's much harder than it needs to be. In Ubuntu 22 you have to battle apport and systemd. Fortunately there are only a few steps needed to disable these interferences. The result will be core dumps produced in the current working directory of the crownd (or crown-qt) program.

Prepare to create/collect coredumps

Disable and stop apport

$ sudo systemctl disable apport
$ sudo systemctl stop apport 

Create and check the name pattern for core dumps

$ sudo sysctl -w kernel.core_pattern=core.%u.%p.%t
kernel.core_pattern = core.%u.%p.%t

$ cat /proc/sys/kernel/core_pattern
core.%u.%p.%t

Note that this is just a simple name (including unique identifying information: userid, process id and time), not a pipeline (which directs the dump into some other process). If the name doesn't include a path component the dump will be created in the current working directory at the time the program was started.

Enable unlimited size core dumps

$ ulimit -c unlimited

Check the limit was removed

$ulimit -a

You should see something like:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
...

where the important bit is the first line, saying core file size is unlimited. This change lasts only until you logout. There are various ways of increasing the limit permanently if you wish to do so.

(Re-)start the daemon

Having enabled coredump collection you must (re-)start crownd (or crown-qt).

Retrieve coredump from crashed program

If crownd (or crown-qt) crashes it will create a coredump in whichever directory was current at the time the program started. If you make a habit of starting the program from the $HOME directory or the .crown/testnet3 data directory it will be easy to find your dumps. Alternatively, you can include a path component in the core_pattern and dumps will always end up in the same directory.

Generate a coredump for a hung program

If the symptoms of the problem you're reporting include the wallet hanging (rather than crashing) you can collect a coredump manually using the gcore program. gcore is not installed by default, but is part of the gdb package. On the recommended VPS installation this is also not installed by default. You will need to:

sudo apt update
sudo apt install -y gdb

As part of the installation it will ask which services to restart:

Hit TAB to get to the <Ok> button and hit ENTER to proceed. Now you can check gcore is installed by

$ which gcore

The result should be

/usr/bin/gcore

Now, if you have a hung wallet (program is still running but not doing anything useful) you need to find the processid using pidof crownd (or pidof crown-qt) and then

sudo gcore -a pid

The result will be a core.pid file in the current working directory (the core_pattern setting is ignored in this case).

Packaging the diagnostics

It is convenient to combine the debug.log and coredump in one package using the zip utility. If it's not already installed you can install it by:

sudo apt update
sudo apt install -y zip

Then, assuming the coredump is located in the testnet datadirectory you can:

cd
cd .crown/testnet3
zip descriptive-name-here debug.log coredump-name-here

Replacing descriptive-name-here and coredump-name-here with values which suit your circumstances. This will compress the files and combine them into a single archive, reducing the size by probably >90%, for example:

If the zip file is less than 25MB in size it can be directly uploaded to a Github issue. If it is any larger you will need to upload it to any convenient Cloud storage service and paste a link to it into a Github issue.