Print thread dump on specific signals #27

Bouncheck · 2024-10-21T13:22:27Z

Adds a signal handler that will print thread dump to standard output on
the following signals: SIGABRT, SIGTERM, SIGINT.
Does not work for SIGKILL or other abrupt shutdowns.

Addresses #28

dkropachev · 2024-10-21T13:46:43Z

tools/stress/src/org/apache/cassandra/stress/settings/SettingsLog.java

@@ -93,11 +95,12 @@ public static final class Options extends GroupedOptions
        final OptionSimple hdrOutputFile = new OptionSimple("hdrfile=", ".*", null, "Log to a file", false);
        final OptionSimple interval = new OptionSimple("interval=", "[0-9]+(ms|s|)", "1s", "Log progress every <value> seconds or milliseconds", false);
        final OptionSimple level = new OptionSimple("level=", "(minimal|normal|verbose)", "normal", "Logging level (minimal, normal or verbose)", false);
+        final OptionSimple printThreadDumpOnShutdown = new OptionSimple("print-thread-dump-on-shutdown", "", null, "Print local thread dump to the standard output on shutdown (normal, SIGTERM, SIGINT, SIGABRT).", false);


I will require additional code on dtest side to understand whethere given c-s support this option or not.
It would be better to just do that by default, but only if c-s was terminated.

Reworked to a version without new option.

tools/stress/src/org/apache/cassandra/stress/Stress.java

CodeLieutenant · 2024-10-21T14:00:08Z

tools/stress/src/org/apache/cassandra/stress/Stress.java

+import sun.misc.Signal;
+import sun.misc.SignalHandler;


Are the sun.misc.* inside the internal parts of JDK and shouldn't be accessed?

Yes, but outside of shutdown hooks I don't know of a way to do this. If it ever gets removed I assume there will be something else for signal handling and in that case this change should be easy to port or ditch altogether since it does not go too deep.

You are right, there is no other way to get to the Unix signals, and handle specific signal through Java Public APIs

dkropachev · 2024-10-21T14:18:06Z

tools/stress/src/org/apache/cassandra/stress/Stress.java

@@ -88,6 +94,14 @@ private static int run(String[] arguments)
                return 1;
            }

+            if (settings.log.printThreadDumpOnShutdown) {
+                Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+                    System.out.println("Shutdown thread dump:");


It should not print thread dump when it terminates in a regular way, only when it is terminated by signal

Adds a signal handler that will print thread dump to standard output on the following signals: SIGABRT, SIGTERM, SIGINT.

Bouncheck · 2024-10-21T14:29:04Z

Removed shutdown hook. Switched to solution with only signal handler. Now no config options are needed and thread dump will be printed only on three mentioned signals.

Co-authored-by: Dmitry Kropachev <dmitry.kropachev@gmail.com>

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <dusan.malusev@scylladb.com>

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <dusan.malusev@scylladb.com> (cherry picked from commit 02997a6) # Conflicts: # defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <dusan.malusev@scylladb.com> (cherry picked from commit 02997a6)

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <dusan.malusev@scylladb.com> (cherry picked from commit 02997a6) # Conflicts: # defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <dusan.malusev@scylladb.com> (cherry picked from commit 02997a6)

Main reason for version change: Using cassandra-stress 3.17 to mittigate - Switch to ReplicaOrdering.RANDOM for select LBPs [32](scylladb/cassandra-stress#32) Other Noticable Changes since the last version used in SCT: - Add support for hostname verification [31](scylladb/cassandra-stress#31) - Print thread dump on specific signals [27](scylladb/cassandra-stress#27) - Replace uninterruptible wait [26](scylladb/cassandra-stress#26) - Make it use DCAwareRoundRobinPolicy unless rack is provided [21](scylladb/cassandra-stress#21) - feature(docker): adding support for dependabot [19](scylladb/cassandra-stress#19) Signed-off-by: Dusan Malusev <dusan.malusev@scylladb.com> (cherry picked from commit 02997a6) # Conflicts: # defaults/docker_images/cassandra-stress/values_cassandra-stress.yaml

Bouncheck force-pushed the thread-dump-on-shutdown branch from 5d1911d to 5fe921d Compare October 21, 2024 13:23

dkropachev requested changes Oct 21, 2024

View reviewed changes

CodeLieutenant reviewed Oct 21, 2024

View reviewed changes

CodeLieutenant approved these changes Oct 21, 2024

View reviewed changes

dkropachev reviewed Oct 21, 2024

View reviewed changes

Add signal handler that prints thread dump

67f8796

Adds a signal handler that will print thread dump to standard output on the following signals: SIGABRT, SIGTERM, SIGINT.

Bouncheck force-pushed the thread-dump-on-shutdown branch from 5fe921d to 67f8796 Compare October 21, 2024 14:28

Bouncheck changed the title ~~Add thread dump on shutdown~~ Print thread dump on specific signals Oct 21, 2024

CodeLieutenant approved these changes Oct 21, 2024

View reviewed changes

CodeLieutenant requested a review from dkropachev October 21, 2024 15:00

CodeLieutenant assigned Bouncheck Oct 21, 2024

CodeLieutenant added the enhancement New feature or request label Oct 21, 2024

CodeLieutenant and others added 2 commits October 21, 2024 17:02

Update tools/stress/src/org/apache/cassandra/stress/Stress.java

59432cb

Co-authored-by: Dmitry Kropachev <dmitry.kropachev@gmail.com>

Stress.java

211f2d4

dkropachev approved these changes Oct 21, 2024

View reviewed changes

CodeLieutenant merged commit 1f432b8 into scylladb:master Oct 21, 2024
2 checks passed

dkropachev mentioned this pull request Oct 28, 2024

Make c-s to print stacktrace of all threads when it is getting forcefully killed, say getting SIGKILL and SIGABRT. #28

Closed

CodeLieutenant mentioned this pull request Nov 21, 2024

fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9296

Merged

2 tasks

mergify bot mentioned this pull request Nov 26, 2024

[Backport 6.0] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9376

Merged

2 tasks

This was referenced Nov 26, 2024

[Backport 6.1] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9377

Merged

[Backport 6.2] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9378

Merged

mergify bot mentioned this pull request Nov 26, 2024

[Backport perf-v16] fix(cassandra-stress): bump c-s version to 3.17.0 scylladb/scylla-cluster-tests#9382

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Print thread dump on specific signals #27

Print thread dump on specific signals #27

Bouncheck commented Oct 21, 2024 •

edited

Loading

dkropachev Oct 21, 2024

Bouncheck Oct 21, 2024

CodeLieutenant Oct 21, 2024

Bouncheck Oct 21, 2024

CodeLieutenant Oct 21, 2024

dkropachev Oct 21, 2024

Bouncheck Oct 21, 2024

Bouncheck commented Oct 21, 2024

		import sun.misc.Signal;
		import sun.misc.SignalHandler;

Print thread dump on specific signals #27

Print thread dump on specific signals #27

Conversation

Bouncheck commented Oct 21, 2024 • edited Loading

dkropachev Oct 21, 2024

Choose a reason for hiding this comment

Bouncheck Oct 21, 2024

Choose a reason for hiding this comment

CodeLieutenant Oct 21, 2024

Choose a reason for hiding this comment

Bouncheck Oct 21, 2024

Choose a reason for hiding this comment

CodeLieutenant Oct 21, 2024

Choose a reason for hiding this comment

dkropachev Oct 21, 2024

Choose a reason for hiding this comment

Bouncheck Oct 21, 2024

Choose a reason for hiding this comment

Bouncheck commented Oct 21, 2024

Bouncheck commented Oct 21, 2024 •

edited

Loading