Possible undefined behavior with bots connected

[TestingPlant]

The tag program aborts on debug mode and panics on release mode after using rust-mc-bot and testing with 500 bots.

Debug:

2024-12-19T13:12:42.207209Z  INFO crates/hyperion/src/simulation/skin.rs:60: player skin cache miss for a730781f-f613-0a8b-8ba5-ef6309ead7bc
2024-12-19T13:12:42.207386Z  INFO crates/hyperion/src/ingress/mod.rs:127: Starting login: Bot_410 553abe3e-dc6c-26cf-de5c-05c477767348
2024-12-19T13:12:42.207517Z DEBUG /home/remote-dev/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.10/src/client/legacy/pool.rs:270: reuse idle connection for ("https", mowojang.matdoes.dev)
target/debug/tag(+0x1a41b8c) [0xae714ad91b8c]
target/debug/tag(+0x1a58f00) [0xae714ada8f00]
target/debug/tag(+0x1a59038) [0xae714ada9038]
target/debug/tag(+0x1a597e0) [0xae714ada97e0]
target/debug/tag(+0x1a44754) [0xae714ad94754]
target/debug/tag(+0x1a45484) [0xae714ad95484]
target/debug/tag(+0x1a29928) [0xae714ad79928]
target/debug/tag(+0x78adf4) [0xae7149adadf4]
target/debug/tag(+0x6f0ca0) [0xae7149a40ca0]
target/debug/tag(+0x99b35c) [0xae7149ceb35c]
target/debug/tag(+0x6ec010) [0xae7149a3c010]
target/debug/tag(+0x85e298) [0xae7149bae298]
target/debug/tag(+0x810e18) [0xae7149b60e18]
target/debug/tag(+0x99f8bc) [0xae7149cef8bc]
target/debug/tag(+0x9f1c04) [0xae7149d41c04]
target/debug/tag(+0x1ad9344) [0xae714ae29344]
target/debug/tag(+0x1ab02e0) [0xae714ae002e0]
target/debug/tag(+0x1ab18a8) [0xae714ae018a8]
target/debug/tag(+0x1ab482c) [0xae714ae0482c]
target/debug/tag(+0x1ab1e84) [0xae714ae01e84]
target/debug/tag(+0x1a51b84) [0xae714ada1b84]
target/debug/tag(+0x1a51d30) [0xae714ada1d30]
target/debug/tag(+0x1a51b24) [0xae714ada1b24]
target/debug/tag(+0x1a51cd8) [0xae714ada1cd8]
target/debug/tag(+0x19da018) [0xae714ad2a018]
target/debug/tag(+0x518384) [0xae7149868384]
target/debug/tag(+0x1bc310) [0xae714950c310]
target/debug/tag(+0x1b6780) [0xae7149506780]
target/debug/tag(+0x1b79a4) [0xae71495079a4]
target/debug/tag(+0x1b5378) [0xae7149505378]
target/debug/tag(+0x1bfd558) [0xae714af4d558]
target/debug/tag(+0x1b5340) [0xae7149505340]
target/debug/tag(+0x1bcf0c) [0xae714950cf0c]
/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xf238d68773fc]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xf238d68774cc]
target/debug/tag(+0x1b08b0) [0xae71495008b0]
Aborted (core dumped)

Release:

2024-12-19T13:16:33.400346Z  INFO player_join_world{name="Bot_65"}: crates/hyperion/src/egress/player_join/mod.rs:367: Bot_65 joined the world
2024-12-19T13:16:33.399757Z  INFO player_join_world{name="Bot_66"}: crates/hyperion/src/egress/player_join/mod.rs:367: Bot_66 joined the world
2024-12-19T13:16:33.400207Z  INFO player_join_world{name="Bot_55"}: crates/hyperion/src/egress/player_join/mod.rs:179: sending skins for 47 players
2024-12-19T13:16:33.400971Z  INFO player_join_world{name="Bot_55"}: crates/hyperion/src/egress/player_join/mod.rs:367: Bot_55 joined the world
2024-12-19T13:16:33.402026Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.402451Z  INFO crates/hyperion/src/ingress/mod.rs:127: Starting login: Bot_77 ee925f6b-4c61-eedb-b03b-2bf9341085ec
2024-12-19T13:16:33.403465Z  INFO crates/hyperion/src/ingress/mod.rs:127: Starting login: Bot_79 f82e45d7-b5ac-58ca-dd6b-37a06fbc969e
2024-12-19T13:16:33.403515Z  INFO crates/hyperion/src/ingress/mod.rs:127: Starting login: Bot_76 963f5be3-fafe-9c74-338d-3a84999fec96
2024-12-19T13:16:33.403719Z  INFO crates/hyperion/src/simulation/skin.rs:60: player skin cache miss for ee925f6b-4c61-eedb-b03b-2bf9341085ec
2024-12-19T13:16:33.404235Z  INFO crates/hyperion/src/ingress/mod.rs:127: Starting login: Bot_78 3fb4580b-c2fa-ff5d-1de9-150816fdd002
2024-12-19T13:16:33.404733Z  INFO crates/hyperion/src/simulation/skin.rs:60: player skin cache miss for f82e45d7-b5ac-58ca-dd6b-37a06fbc969e
2024-12-19T13:16:33.404807Z  INFO crates/hyperion/src/simulation/skin.rs:60: player skin cache miss for 963f5be3-fafe-9c74-338d-3a84999fec96
2024-12-19T13:16:33.404840Z  INFO crates/hyperion/src/simulation/skin.rs:60: player skin cache miss for 3fb4580b-c2fa-ff5d-1de9-150816fdd002
2024-12-19T13:16:33.406650Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.407077Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.408920Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.420370Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.437485Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.437522Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.443026Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.445178Z ERROR crates/hyperion/src/ingress/mod.rs:136: failed to get skin failed to parse json from response: "". Using empty skin
2024-12-19T13:16:33.446019Z  INFO generate_ingress_events: crates/hyperion/src/ingress/mod.rs:316: player_connect
2024-12-19T13:16:33.446216Z  INFO generate_ingress_events: crates/hyperion/src/ingress/mod.rs:316: player_connect
2024-12-19T13:16:33.446224Z  INFO generate_ingress_events: crates/hyperion/src/ingress/mod.rs:316: player_connect
2024-12-19T13:16:33.446227Z  INFO generate_ingress_events: crates/hyperion/src/ingress/mod.rs:316: player_connect
thread '2024-12-19T13:16:33.449438Z  INFO player_join_world{name="Bot_57"}: crates/hyperion/src/egress/player_join/mod.rs:179: sending skins for 57 players
2024-12-19T13:16:33.449665Z  INFO player_join_world{name="Bot_57"}: crates/hyperion/src/egress/player_join/mod.rs:367: Bot_57 joined the world
2024-12-19T13:16:33.450355Z  INFO player_join_world{name="Bot_31"}: crates/hyperion/src/egress/player_join/mod.rs:179: sending skins for 57 players
2024-12-19T13:16:33.450474Z  INFO player_join_world{name="Bot_31"}: crates/hyperion/src/egress/player_join/mod.rs:367: Bot_31 joined the world
2024-12-19T13:16:33.451081Z  INFO player_join_world{name="Bot_70"}: crates/hyperion/src/egress/player_join/mod.rs:179: sending skins for 57 players
2024-12-19T13:16:33.451222Z  INFO player_join_world{name="Bot_70"}: crates/hyperion/src/egress/player_join/mod.rs:367: Bot_70 joined the world
<unnamed>' panicked at /usr/local/cargo/git/checkouts/flecs-rust-ff75a97755951fdf/db1b212/flecs_ecs/src/core/get_tuple.rs:414:1:
Component `hyperion::simulation::Name` not found on `EntityView::get`operation
with parameters: `(&hyperion::simulation::Uuid, &hyperion::simulation::Name, &hyperion::simulation::Position, &hyperion::simulation::Yaw, &hyperion::simulation::Pitch, &hyperion::net::ConnectionId)`.
Use `try_get` variant to avoid assert/panicking if you want to handle the error
or use `Option<&hyperion::simulation::Name> instead to handle individual cases.
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: rayon::iter::plumbing::bridge_producer_consumer::helper
   3: rayon_core::join::join_context::{{closure}}
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon_core::join::join_context::{{closure}}
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   8: rayon_core::registry::WorkerThread::wait_until_cold
   9: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Since there's a different behavior on release mode, I suspect some UB is occurring and that the panic/abort is just a symptom. In addition, on commit d358e2, release mode sometimes caused a segfault and sometimes caused a panic, although on the latest main (ef00b8) I only seem to get a panic on release mode.

A git bisect shows that the abort started in commit 22d628. However, that commit doesn't seem to have any obviously unsound code, so I believe the UB was introduced earlier than that, and 22d628 is the first commit to show a symptom of UB.

[TestingPlant]

I'm currently working on this branch which replaces unsafe code with safe code to try to find which unsafe block is causing issues: https://github.com/TestingPlant/hyperion/tree/reduce-unsafe

[andrewgazelka]

btw we have https://github.com/hyperion-mc/hyperion/tree/main/tools/antithesis-bot

all machines have 48 cores so they should be able to run hundreds of bots. https://youtu.be/m3HwXlQPCEU?si=ZULB7Be7BVF58sK3