Fatal Validator Crash - libp2p_peerbook:random

[anthonyra]

libp2p_group_gossip_server:handle_cast:{307,41} gossip handler libp2p_peerbook failed to init with error {{case_clause,{ok,<<>>,<<10,219,4,18,99,164,3,38,0,0,43,155,224,125,26,169,2,167,181,35,89,179,84,151,59,241,99,96,206,103,223,67,185,156,154,228,164,86,103,137,235,104,158,74,126,66,162,2,55,112,50,112,47,49,49,65,111,104,112,67,51,97,71,88,69,105,103,65,83,82,121,54,78,113,90,107,120,54,88,74,98,66,120,49,49,50,49,84,99,82,72,114,97,107,56,115,56,69,81,89,113,88,109,103,26,33,0,27,62,104,203,106,211,151,246,223,89,26,32,222,144,205,17,111,167,104,243,74,214,168,79,6,186,170,211,119,202,114,194,26,33,0,142,77,212,61,185,96,37,245,158,247,217,101,183,97,172,24,246,184,29,90,238,235,171,64,112,153,131,83,191,25,203,9,26,33,0,208,21,41,93,215,209,84,11,49,5,238,21,206,62,159,233,41,167,53,250,37,66,229,175,112,38,200,93,124,118,221,158,26,33,0,213,52,63,159,73,141,65,171,2,127,204,113,170,189,140,125,78,7,22,209,193,217,206,173,108,42,189,71,183,79,104,178,26,33,0,160,244,7,63,182,108,68,135,253,200,130,33,205,162,149,82,237,199,49,229,201,206,198,86,245,83,30,207,195,241,180,127,26,33,0,42,249,31,150,117,6,181,188,33,205,104,133,217,157,170,22,132,190,130,200,197,85,80,205,224,170,18,208,249,175,203,49,26,33,0,172,146,245,216,71,92,73,192,249,169,249,88,164,246,61,236,25,128,48,5,40,33,110,103,43,95,205,152,100,58,54,102,26,33,0,155,172,9,43,156,94,67,136,98,196,200,244,229,43,100,80,35,72,147,215,199,199,247,106,248,11,177,135,96,195,248,54,26,33,0,138,103,47,165,80,105,83,57,178,27,24,164,144,116,192,237,7,83,168,106,111,221,237,235,218,222,34,80,186,31,231,174,26,33,0,120,151,218,61,153,123,56,60,89,104,140,62,145,132,5,162,91,12,187,98,163,105,181,187,234,211,31,252,116,223,89,186,26,33,0,62,150,154,159,86,67,91,142,242,235,13,204,165,73,114,3,87,84,16,56,39,228,230,181,111,88,23,31,66,232,199,11,32,3,40,163,237,188,200,128,48,58,32,45,174,143,185,95,52,62,207,106,76,253,55,223,211,240,155,174,42,223,247,94,49,57,47,232,147,168,215,110,117,16,146,66,29,10,19,108,97,115,116,95,98,108,111,99,107,95,97,100,100,95,116,105,109,101,18,6,8,141,201,192,146,6,66,25,10,12,114,101,108,101,97,115,101,95,105,110,102,111,18,9,26,7,117,110,107,110,111,119,110,66,14,10,6,104,101,105,103,104,116,18,4,8,146,195,79,18,71,48,69,2,32,31,199,167,18,174,4,124,227,95,6,248,213,114,252,31,33,169,105,226,192,24,207,232,155,207,218,65,79,154,211,144,0,2,33,0,246,81,89,238,195,155,13,52,185,83,65,165,134,207,177,166,175,168,159,190,13,139,116,36,27,30,19,121,94,101,22,28>>}},[{libp2p_peerbook,random,4,[{file,"libp2p_peerbook.erl"},{line,206}]},{libp2p_peerbook,init_gossip_data,1,[{file,"libp2p_peerbook.erl"},{line,400}]},{libp2p_group_gossip_server,'-handle_cast/2-fun-8-',4,[{file,"libp2p_group_gossip_server.erl"},{line,305}]},{maps,fold_1,3,[{file,"maps.erl"},{line,410}]},{libp2p_group_gossip_server,handle_cast,2,[{file,"libp2p_group_gossip_server.erl"},{line,304}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}

I noticed today that my validator testing 1.8.4 crashed after it tried to pick a random peer from it's peerbook. It ended up being the first entry to my peerbook and then failed a pattern match.. After some digging it appears that this peer was added to my peerbook without having an associated pubkey. When decoding that peer data you can see the following;

miner eval 'libp2p_peer:pubkey_bin(libp2p_peer:decode_unsafe(<<10,219,4,18,99,164,3,38,0,0,43,155,224,125,26,169,2,167,181,35,89,179,84,151,59,241,99,96,206,103,223,67,185,156,154,228,164,86,103,137,235,104,158,74,126,66,162,2,55,112,50,112,47,49,49,65,111,104,112,67,51,97,71,88,69,105,103,65,83,82,121,54,78,113,90,107,120,54,88,74,98,66,120,49,49,50,49,84,99,82,72,114,97,107,56,115,56,69,81,89,113,88,109,103,26,33,0,27,62,104,203,106,211,151,246,223,89,26,32,222,144,205,17,111,167,104,243,74,214,168,79,6,186,170,211,119,202,114,194,26,33,0,142,77,212,61,185,96,37,245,158,247,217,101,183,97,172,24,246,184,29,90,238,235,171,64,112,153,131,83,191,25,203,9,26,33,0,208,21,41,93,215,209,84,11,49,5,238,21,206,62,159,233,41,167,53,250,37,66,229,175,112,38,200,93,124,118,221,158,26,33,0,213,52,63,159,73,141,65,171,2,127,204,113,170,189,140,125,78,7,22,209,193,217,206,173,108,42,189,71,183,79,104,178,26,33,0,160,244,7,63,182,108,68,135,253,200,130,33,205,162,149,82,237,199,49,229,201,206,198,86,245,83,30,207,195,241,180,127,26,33,0,42,249,31,150,117,6,181,188,33,205,104,133,217,157,170,22,132,190,130,200,197,85,80,205,224,170,18,208,249,175,203,49,26,33,0,172,146,245,216,71,92,73,192,249,169,249,88,164,246,61,236,25,128,48,5,40,33,110,103,43,95,205,152,100,58,54,102,26,33,0,155,172,9,43,156,94,67,136,98,196,200,244,229,43,100,80,35,72,147,215,199,199,247,106,248,11,177,135,96,195,248,54,26,33,0,138,103,47,165,80,105,83,57,178,27,24,164,144,116,192,237,7,83,168,106,111,221,237,235,218,222,34,80,186,31,231,174,26,33,0,120,151,218,61,153,123,56,60,89,104,140,62,145,132,5,162,91,12,187,98,163,105,181,187,234,211,31,252,116,223,89,186,26,33,0,62,150,154,159,86,67,91,142,242,235,13,204,165,73,114,3,87,84,16,56,39,228,230,181,111,88,23,31,66,232,199,11,32,3,40,163,237,188,200,128,48,58,32,45,174,143,185,95,52,62,207,106,76,253,55,223,211,240,155,174,42,223,247,94,49,57,47,232,147,168,215,110,117,16,146,66,29,10,19,108,97,115,116,95,98,108,111,99,107,95,97,100,100,95,116,105,109,101,18,6,8,141,201,192,146,6,66,25,10,12,114,101,108,101,97,115,101,95,105,110,102,111,18,9,26,7,117,110,107,110,111,119,110,66,14,10,6,104,101,105,103,104,116,18,4,8,146,195,79,18,71,48,69,2,32,31,199,167,18,174,4,124,227,95,6,248,213,114,252,31,33,169,105,226,192,24,207,232,155,207,218,65,79,154,211,144,0,2,33,0,246,81,89,238,195,155,13,52,185,83,65,165,134,207,177,166,175,168,159,190,13,139,116,36,27,30,19,121,94,101,22,28>>)).'
#Bin<>
miner eval 'libp2p_peer:listen_addrs(libp2p_peer:decode_unsafe(<<10,219,4,18,99,164,3,38,0,0,43,155,224,125,26,169,2,167,181,35,89,179,84,151,59,241,99,96,206,103,223,67,185,156,154,228,164,86,103,137,235,104,158,74,126,66,162,2,55,112,50,112,47,49,49,65,111,104,112,67,51,97,71,88,69,105,103,65,83,82,121,54,78,113,90,107,120,54,88,74,98,66,120,49,49,50,49,84,99,82,72,114,97,107,56,115,56,69,81,89,113,88,109,103,26,33,0,27,62,104,203,106,211,151,246,223,89,26,32,222,144,205,17,111,167,104,243,74,214,168,79,6,186,170,211,119,202,114,194,26,33,0,142,77,212,61,185,96,37,245,158,247,217,101,183,97,172,24,246,184,29,90,238,235,171,64,112,153,131,83,191,25,203,9,26,33,0,208,21,41,93,215,209,84,11,49,5,238,21,206,62,159,233,41,167,53,250,37,66,229,175,112,38,200,93,124,118,221,158,26,33,0,213,52,63,159,73,141,65,171,2,127,204,113,170,189,140,125,78,7,22,209,193,217,206,173,108,42,189,71,183,79,104,178,26,33,0,160,244,7,63,182,108,68,135,253,200,130,33,205,162,149,82,237,199,49,229,201,206,198,86,245,83,30,207,195,241,180,127,26,33,0,42,249,31,150,117,6,181,188,33,205,104,133,217,157,170,22,132,190,130,200,197,85,80,205,224,170,18,208,249,175,203,49,26,33,0,172,146,245,216,71,92,73,192,249,169,249,88,164,246,61,236,25,128,48,5,40,33,110,103,43,95,205,152,100,58,54,102,26,33,0,155,172,9,43,156,94,67,136,98,196,200,244,229,43,100,80,35,72,147,215,199,199,247,106,248,11,177,135,96,195,248,54,26,33,0,138,103,47,165,80,105,83,57,178,27,24,164,144,116,192,237,7,83,168,106,111,221,237,235,218,222,34,80,186,31,231,174,26,33,0,120,151,218,61,153,123,56,60,89,104,140,62,145,132,5,162,91,12,187,98,163,105,181,187,234,211,31,252,116,223,89,186,26,33,0,62,150,154,159,86,67,91,142,242,235,13,204,165,73,114,3,87,84,16,56,39,228,230,181,111,88,23,31,66,232,199,11,32,3,40,163,237,188,200,128,48,58,32,45,174,143,185,95,52,62,207,106,76,253,55,223,211,240,155,174,42,223,247,94,49,57,47,232,147,168,215,110,117,16,146,66,29,10,19,108,97,115,116,95,98,108,111,99,107,95,97,100,100,95,116,105,109,101,18,6,8,141,201,192,146,6,66,25,10,12,114,101,108,101,97,115,101,95,105,110,102,111,18,9,26,7,117,110,107,110,111,119,110,66,14,10,6,104,101,105,103,104,116,18,4,8,146,195,79,18,71,48,69,2,32,31,199,167,18,174,4,124,227,95,6,248,213,114,252,31,33,169,105,226,192,24,207,232,155,207,218,65,79,154,211,144,0,2,33,0,246,81,89,238,195,155,13,52,185,83,65,165,134,207,177,166,175,168,159,190,13,139,116,36,27,30,19,121,94,101,22,28>>)).'
["/p2p/11LCvynMMYJmoXmZhXwqoaaGyS6qeJHssUzrJQYeym59PUinSty/p2p-circuit/p2p/11AohpC3aGXEigASRy6NqZkx6XJbBx1121TcRHrak8s8EQYqXmg"]

The function libp2p_peerbook:random crashes with a missing pattern match since the above entry results in {ok,<<>>, Peer} instead of the expected {ok, FirstAddr = <<Start:(33*8)/integer-unsigned-big>>, FirstPeer}... not sure how this peer ended up without a pubkey in the first place...

[anthonyra]

After further digging, a peer can be added to the peerbook while having a <<>> key for its pubkey. If there's more then one hotspot responding with these results via gossip... there could be a lot of peers missing from the peerbook.