Session networking handling rewrite testing #3021
Replies: 10 comments 27 replies
-
|
When trying to update my mod for the prerelease, I see that things like Where some of those metrics deprecated with the change? |
Beta Was this translation helpful? Give feedback.
-
|
It's under I didn't remove any stats, but they're in different places now. |
Beta Was this translation helpful? Give feedback.
-
|
I see, thanks! Also tested with a small session (4-5 users max) and only one issue: for some reason someone couldn't join the world at the first try. I'll try to create a setup to reproduce that. Otherwise, no apparent issues. |
Beta Was this translation helpful? Give feedback.
-
|
This is a great update! The sync of values still seems to be paused, is this unavoidable? 2024-09-29.02-40-56_cut.mp4This is an item that visualizes the delay by having each user update their line position every frame, but we found that the updates stopped for a while after the user joined the world. I have tried several games and items for about an hour and so far have not found any glitches. |
Beta Was this translation helpful? Give feedback.
-
|
@orange3134 I've pushed a new version 2024.9.28.1132 The host will now log the time it takes in the synchronous loop to capture the data model. Can you replicate this issue with this build please? Once replicated, check the log on the host for following lines: |
Beta Was this translation helpful? Give feedback.
-
|
I will try again with the new version. All my friends have already gone to bed tonight, so I will try again tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
No worries, there's no rush! Even if this turns out to be fixable with this new system, I might not implement this optimization a build until after this part is released anyway. Thank you for testing! |
Beta Was this translation helpful? Give feedback.
-
|
@Frooxius From the host's perspective, everything appears to stop at that moment. (It appears the game drawing frames are not being updated) From the non-host's perspective, it appeared that the synchronization of values other than my own temporarily stopped. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the testing. Since the hitch seems to be caused by the initial capture, that is unfortunatelly not solvable as part of this specific optimization. It might be worthwhile to test similar scenario with a .NET 8 headless, since the increased runtime performance could make this much faster. We'll have to evaluate approaches to speed up this initial capture in the future - there are some possibilities, like optimizing the capture itself or maintaining an incrementally updated full state batch. However with this optimization, the async pose and voice streams are no longer interrupted, which should make the experience much smoother. The only thing that hitches will be the data model updates. |
Beta Was this translation helpful? Give feedback.
-
|
I've updated the pre-release with new diagnostic 2024.9.28.1132
|
Beta Was this translation helpful? Give feedback.
-
|
Audio stream had almost constant 3% Packet Loss. |
Beta Was this translation helpful? Give feedback.
-
|
Does this not happen on the normal build? This can be normal behavior depending on the network used - the changes don't really change networking that much, though there can be more packets being thrown out than normal in some cases. |
Beta Was this translation helpful? Give feedback.
-
|
The Grand Oasis conducted a test on the pre-release. We didn't notice anything particularly out of the ordinary. Everything we tested seemed to work fine outside of already known issues with the main build. Thing's we've tested:
Issues we had:
Bellow is the log from our headless. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thank you so much for the thorough testing!
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, I believe you are right. The headless cannot use Steam Sockets in this instance (Dunno why I didn't think of that lol) The TCP I will test later via LAN. The denied message I can test later tonight when the Oasis has their event to verify its part of issue 3000. |
Beta Was this translation helpful? Give feedback.
-
Tested with Steam sockets on a Windows Headless, they're working as expected. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the info and testing! |
Beta Was this translation helpful? Give feedback.
-
Retested the denied message on the main build with the same user and the same issue was exhibited. So I'm confident the issue is not related to the prerelease. |
Beta Was this translation helpful? Give feedback.
-
|
I only did some slight testing and all of it was in Jack's headless but I did a temp ban on myself (via the Resonite bot) and successfully was booted from the session and was denied joining til I unbanned myself. I did not notice any issue with streams and they remained uninterrupted throughout the session even with people joining. Honestly pretty impressed how there seems to be no apparent issues despite being the first prerelease of an entire join revamp! Of course, perhaps having people do self hosted sessions compared to headless might be a good test. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for testing and sharing! |
Beta Was this translation helpful? Give feedback.
-
|
I've tested my headless and graphical client on the prerelease build yesterday, and everything seemed to work mostly fine. We played Cloudscape Harvest with others, later more people joined and everything from editing to sharing assets and streaming audio was tested - all apparently normal. Earlier in the session EpicEaston encountered pretty severe packet loss, that was however corrected by adjusting the window size and re-joining the session. It's also not the first time she had issues on my headless, so I'm not sure I would put that on the networking changes. It did promt me to look at my own packet loss though, and I've noticed a slight discrepency. My headless is in a data center and I'm running on residential fibreoptics, no wireless is used, and yet I noticed that my client was reporting slight packet loss in the prerelease. I've never encountered packet loss on my own headless, and sure enough after switching back to the main build without changing any networking setups, the packet loss is gone again. Prerelease session: Session on main build: So something seems to be off here, even though this amount of packet loss likely won't affect anything. |
Beta Was this translation helpful? Give feedback.
-
|
I did some more testing, the packet loss seems completely unrelated to the prerelease version, so this is a non-issue! |
Beta Was this translation helpful? Give feedback.
-
|
Small question. Is this supposed to be greyed out? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you did not set a VerifyJoinRequest node to handle it so you can not turn it on til you do. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks everyone for testing! Since this is pretty much without any major issues, I'm going to merge this in and publish a build today! |
Beta Was this translation helpful? Give feedback.
-
|
This might be an edge case, but I ran into an Issue with audio crackling way more. Ive been fine in other sessions and even with over 40 People on an US headless, but in this case, I got major audio crackling and from the looks of it also updating lag. Is there anything I can do to analyze this futher? |
Beta Was this translation helpful? Give feedback.
-
|
Its just me so far, with other people Voices. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, can you check what % packet loss you see on their OpusStreams? |
Beta Was this translation helpful? Give feedback.
-
|
Oohh yes... 7% Packet loss only to the voice for some reason & only for me, other people have none. PS: It -must- have been way more earlier, now the voices getting better to understand again with that 7%.
|
Beta Was this translation helpful? Give feedback.
-
|
Does it happen with only this specific host? It sounds more like a networking issue to me. But yeah 7 % is quite a bit. |
Beta Was this translation helpful? Give feedback.
-
|
I havent expirienced it anywhere else -voice- wise. It might have been way higher than 7%, when I took the video probably.... |
Beta Was this translation helpful? Give feedback.
-
Hello everyone!
I've got another thing that needs some testing before it gets released to public, since it can cause some stuff to blow up.
Build Version
2024.9.28.1132
What is being tested
The session networking handling system has been completely restructured and rewritten as part of performance optimizations, switching over to the Task Parallel Library for a lot of the heavy duty processing instead of manual thread management. More of the processing is now parallelized as a result and also hoisted out of the world update loop. There are some small optimizations and improvements sprinkled along as well.
The performance goal of this change is to eliminate the server hitches when new users join, especially populated sessions (IMPORTANT: This doesn't affect hitches from loading assets, that's a separate future set of optimizations), which is documented in this issue: #1477
Another goal of this is to make the code and its flow much more readable and manageable, which will make a lot of things easier in the future: fixing bugs, improving/expanding security and privacy and implementing new features like mesh networking.
The system that was rewritten is responsible for a number of things:
Since this is a critical system for Resonite and being able to join, host and exist in sessions, I'd like this to be tested before it's fully deployed, since some stuff can break. I did some preliminary smoke testing, but to fully test the system it needs a number of users and networking scenarios.
How to test
prereleasebranch and make sure the build updates before you run itYou will need to test with multiple users since this affects session hosting and joining sessions.
Here's a list of things you should try out (or at least some of these) and focus on while testing:
What to test
What to watch out for
These are good things to watch out for when testing:
What NOT to test & report
Generally things that do not involve this system - any existing unrelated bugs with components and other systems.
-- IMPORTANT: If you can get it to happen a lot more frequently than it does usually, then DO report it.
If you find something that got broken
If you find that stuff works okay
Please report it too and include details on what you generally tested, so we know things are okay. You don't need to make a new issue for this - just a commend in this Discussion is fine.
This lets us know that a bunch of stuff was tested and get better confidence that stuff won't break with release.
How long will testing go for?
With the nature of these changes issues should become apparent pretty quickly - at least the frequent ones. If we get a bunch of reports of things working okay, we might release this on Monday already. If there's blocker issues cropping up, we'll hold up the release until things are resolved.
Beta Was this translation helpful? Give feedback.
All reactions