Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scylla-bench v0.1.8 got fatal error: runtime: out of memory #89

Open
yarongilor opened this issue Feb 7, 2022 · 0 comments
Open

scylla-bench v0.1.8 got fatal error: runtime: out of memory #89

yarongilor opened this issue Feb 7, 2022 · 0 comments

Comments

@yarongilor
Copy link

yarongilor commented Feb 7, 2022 •
edited
Loading

Installation details
Kernel version: 5.11.0-1022-aws
Scylla version (or git commit hash): 4.6.rc5-0.20220203.5694ec189 with build-id f5d85bf5abe6d2f9fd3487e2469ce1c34304cc14
Cluster size: 3 3 3 nodes (i3.2xlarge)
Scylla running with shards number (live nodes):
longevity-counters-multidc-4-6-db-node-ec7ffc0b-1 (34.254.228.100 | 10.0.0.60): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-2 (52.19.209.202 | 10.0.3.34): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-3 (54.75.1.178 | 10.0.2.127): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-4 (52.34.214.217 | 10.0.3.8): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-5 (35.88.112.46 | 10.0.2.163): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-6 (35.88.221.142 | 10.0.2.18): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-7 (44.200.34.43 | 10.0.3.238): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-8 (44.195.21.232 | 10.0.0.201): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-9 (3.228.17.3 | 10.0.3.20): 8 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-06df116ea1c70171c ami-0d2bccabf736b9dd0 ami-097f7b2efb7723718 (aws: eu-west-1 us-west-2 us-east-1)

Test: longevity-counters-multidc-test
Test name: longevity_test.LongevityTest.test_custom_time
Test config file(s):

  • longevity-counters-multidc.yaml

Issue description

====================================

scylla-bench got the following OOM with version 0.1.8:
from the loader-set log of scylla-bench-l0-48d0dafb-2e83-46b1-8add-6e7dc9fc431c.log:
(./longevity-counters-multidc-4-6-loader-node-ec7ffc0b-1/scylla-bench-l0-48d0dafb-2e83-46b1-8add-6e7dc9fc431c.log)

2022/02/06 16:51:00 writev tcp 10.0.2.255:39009->35.88.221.142:9042: i/o timeout
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x6fbe3b, 0x400000})
	/usr/local/go/src/runtime/panic.go:1198 +0x71
runtime.sysMap(0xc5cf800000, 0x1, 0xc000080800)
	/usr/local/go/src/runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x93c080, 0x43bdef)
	/usr/local/go/src/runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x93c080, 0x1, 0x0, 0x12)
	/usr/local/go/src/runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:913 +0x69
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:383 +0x49

goroutine 671 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc00c367820 sp=0xc00c367818 pc=0x462220
runtime.(*mheap).alloc(0x761f50, 0xc0000180b0, 0x68, 0x1)
	/usr/local/go/src/runtime/mheap.go:907 +0x73 fp=0xc00c367870 sp=0xc00c367820 pc=0x426ed3
runtime.(*mcentral).grow(0x2000)
	/usr/local/go/src/runtime/mcentral.go:241 +0x65 fp=0xc00c3678b8 sp=0xc00c367870 pc=0x418865
runtime.(*mcentral).cacheSpan(0x94d048)
	/usr/local/go/src/runtime/mcentral.go:161 +0x69e fp=0xc00c367930 sp=0xc00c3678b8 pc=0x41869e
runtime.(*mcache).refill(0x7febc181d5b8, 0x12)
	/usr/local/go/src/runtime/mcache.go:162 +0xaf fp=0xc00c367980 sp=0xc00c367930 pc=0x4178cf
runtime.(*mcache).nextFree(0x7febc181d5b8, 0x12)
	/usr/local/go/src/runtime/malloc.go:880 +0x85 fp=0xc00c3679c8 sp=0xc00c367980 pc=0x40da85
runtime.mallocgc(0x70, 0x6a3580, 0x1)
	/usr/local/go/src/runtime/malloc.go:1071 +0x4e8 fp=0xc00c367a48 sp=0xc00c3679c8 pc=0x40e108
runtime.newobject(0xc5cf771b90)
	/usr/local/go/src/runtime/malloc.go:1228 +0x27 fp=0xc00c367a70 sp=0xc00c367a48 pc=0x40e627
main.DoReadsFromTable.func1(0xc0003bc990)
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:479 +0x78e fp=0xc00c367c78 sp=0xc00c367a70 pc=0x680bee
main.RunTest(0xc0003bc990, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80}, 0xc00c367e40)
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:126 +0x1b0 fp=0xc00c367db0 sp=0xc00c367c78 pc=0x67d5d0
main.DoReadsFromTable({0x6f8ac5, 0xd}, 0x0, 0x0, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80})
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:427 +0x16e fp=0xc00c367ea0 sp=0xc00c367db0 pc=0x6802ce
main.DoCounterReads(0x7ffe23e16d11, 0x0, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80})
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:384 +0x53 fp=0xc00c367ef0 sp=0xc00c367ea0 pc=0x67f9f3
main.main.func4(0x0, 0x0, {0x75f8a0, 0x952d80})
	/tmp/sb_install/scylla-bench-0.1.8/main.go:527 +0x105 fp=0xc00c367f88 sp=0xc00c367ef0 pc=0x67c7e5
main.RunConcurrently.func1(0x5c)
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:78 +0x66 fp=0xc00c367fc8 sp=0xc00c367f88 pc=0x67d326
main.RunConcurrently·dwrap·2()
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:79 +0x2d fp=0xc00c367fe0 sp=0xc00c367fc8 pc=0x67d28d
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00c367fe8 sp=0xc00c367fe0 pc=0x464441
created by main.RunConcurrently
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:76 +0x10c

goroutine 1 [runnable]:
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).getCountAtIndex(...)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:595
github.com/HdrHistogram/hdrhistogram-go.(*iterator).nextCountAtIdx(0x4e714e7c6b18b182, 0x630d02662c804285)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:662 +0x65
github.com/HdrHistogram/hdrhistogram-go.(*iterator).next(0xc000107530)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:670 +0x25
github.com/HdrHistogram/hdrhistogram-go.(*rIterator).next(...)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:683
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).Merge(0xf00000009, 0x4000000000a)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:177 +0x98
github.com/scylladb/scylla-bench/pkg/results.(*MergedResult).AddResult(0xc07adacba0, {0x0, 0x0, 0x6, 0x6, 0x0, {0x0, 0x0, 0x0}, 0xc09f0c2600, ...})
	/tmp/sb_install/scylla-bench-0.1.8/pkg/results/merged_result.go:53 +0x1b0
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetResultsFromThreadsAndMerge(0xc0002e6600)
	/tmp/sb_install/scylla-bench-0.1.8/pkg/results/thread_results.go:60 +0x89
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetTotalResults(0xc0002e6600)
	/tmp/sb_install/scylla-bench-0.1.8/pkg/results/thread_results.go:82 +0xcd
main.main()
	/tmp/sb_install/scylla-bench-0.1.8/main.go:530 +0x31f7

goroutine 80 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020ebc0})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7feb9848b038, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c400, 0xc000144000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c400, {0xc000144000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c400, {0xc000144000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010038, {0xc000144000, 0x123, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c300, {0xc000146048, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c300}, {0xc000146048, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c300}, {0xc000146048, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146000, {0x761f18, 0xc00005a240})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a240})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 79 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a558, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009cc00, 0xc000250000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009cc00, {0xc000250000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009cc00, {0xc000250000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0c8, {0xc000250000, 0x40, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002018c0, {0xc00023a4c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc0002018c0}, {0xc00023a4c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc0002018c0}, {0xc00023a4c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a480, {0x761f18, 0xc00020ebc0})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0xc0000f00e0, {0x761f18, 0xc00020ebc0})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 51 [select, 2 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc00019a190)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:39 +0x67
created by github.com/gocql/gocql.newEventDebouncer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:27 +0x11f

goroutine 52 [select, 54 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc00019a230)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:39 +0x67
created by github.com/gocql/gocql.newEventDebouncer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:27 +0x11f

goroutine 72 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a380})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 37 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00009ab40})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 26 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a240})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 35 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002008a0, 0xc000187480)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:874 +0x1b9
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 36 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848af50, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c580, 0xc0000f6000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c580, {0xc0000f6000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c580, {0xc0000f6000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0000960f8, {0xc0000f6000, 0x41, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00008eba0, {0xc0000f8048, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00008eba0}, {0xc0000f8048, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00008eba0}, {0xc0000f8048, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc0000f8000, {0x761f18, 0xc00009ab40})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00009ab40})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 39 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/control.go:70 +0x10d
created by github.com/gocql/gocql.(*controlConn).connect
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/control.go:259 +0x13b

goroutine 75 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a280})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 529 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005be80})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 86 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a400})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 69 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e6c0})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 73 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000201800, 0xc00005a380)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:874 +0x1b9
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 48 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002014a0, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 98 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00005d140, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 84 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00008f560, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 67 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002015c0, 0xc00020e700)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 70 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002016e0, 0xc00020e6c0)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 33 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00008f440, 0xc00002df88)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 74 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ae68, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c680, 0xc00014a000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c680, {0xc00014a000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c680, {0xc00014a000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010048, {0xc00014a000, 0x3f, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c480, {0xc0001461c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c480}, {0xc0001461c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c480}, {0xc0001461c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146180, {0x761f18, 0xc00005a280})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a280})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 99 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ac98, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000212380, 0xc000238000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000212380, {0xc000238000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000212380, {0xc000238000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0a8, {0xc000238000, 0x42, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000200cc0, {0xc00023a048, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc000200cc0}, {0xc00023a048, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc000200cc0}, {0xc00023a048, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a000, {0x761f18, 0xc00020e640})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e640})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 49 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ad80, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00018c700, 0xc00014e000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00018c700, {0xc00014e000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00018c700, {0xc00014e000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010050, {0xc00014e000, 0x3e, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c720, {0xc000146348, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c720}, {0xc000146348, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c720}, {0xc000146348, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146300, {0x761f18, 0xc00005a300})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a300})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 68 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848aac8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00018c800, 0xc00023e000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00018c800, {0xc00023e000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00018c800, {0xc00023e000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0b0, {0xc00023e000, 0x3f, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000200f60, {0xc00023a1c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc000200f60}, {0xc00023a1c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc000200f60}, {0xc00023a1c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a180, {0x761f18, 0xc00020e6c0})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e6c0})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 82 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a8f8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c900, 0xc000242000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c900, {0xc000242000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c900, {0xc000242000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0b8, {0xc000242000, 0x3e, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002010e0, {0xc00023a348, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc0002010e0}, {0xc00023a348, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc0002010e0}, {0xc00023a348, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a300, {0x761f18, 0xc00020e700})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e700})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 71 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a9e0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000212480, 0xc000158000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000212480, {0xc000158000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000212480, {0xc000158000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010058, {0xc000158000, 0x41, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c9c0, {0xc0001464c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c9c0}, {0xc0001464c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c9c0}, {0xc0001464c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146480, {0x761f18, 0xc00005a380})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a380})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 85 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848abb0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c780, 0xc00015a000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c780, {0xc00015a000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c780, {0xc00015a000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010060, {0xc00015a000, 0x40, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005cb40, {0xc000146648, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005cb40}, {0xc000146648, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005cb40}, {0xc000146648, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146600, {0x761f18, 0xc00005a400})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a400})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 66 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a300})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 83 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e700})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 100 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e640})
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 332 [select]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc0001ba380, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/session.go:321 +0x106
created by github.com/gocql/gocql.(*Session).init

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor ec7ffc0b-b344-43db-9777-6db183568e53
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs ec7ffc0b-b344-43db-9777-6db183568e53

Test id: ec7ffc0b-b344-43db-9777-6db183568e53

Logs:
grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png)&source=gmail-html&ust=1644321554025000&usg=AOvVaw3HDEgKbq2l99B4HNUgaoLU)
grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png)&source=gmail-html&ust=1644321554025000&usg=AOvVaw1NGtRRtTSjjUYhAvvI-V5Y)
db-cluster - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554025000&usg=AOvVaw2gBqC6DDm6Lrbp1S0efTuN)
loader-set - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554025000&usg=AOvVaw2LBMsL-uiQIHVZLMrXuV0L)
monitor-set - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554025000&usg=AOvVaw0by3rWi6sHGCB5cMkwLy-b)
sct - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554026000&usg=AOvVaw2Ae2VgboaRnnRSgWtjEQEK)

Jenkins job URL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yarongilor