Skip to content

Implement room instancing in socketio-server.js#458

Merged
vincentfretin merged 1 commit intomasterfrom
room-instancing
Mar 22, 2024
Merged

Implement room instancing in socketio-server.js#458
vincentfretin merged 1 commit intomasterfrom
room-instancing

Conversation

@vincentfretin
Copy link
Member

@vincentfretin vincentfretin commented Mar 20, 2024

Use case: experience with 3000 users, using room instancing by batch of 50 users, no audio, no video, only websocket. We are interested only in avatar movements.

For 50 in a room, you probably want to reduce the update rate to 1 per second instead of 15 by setting NAF.options.updateRate = 1 (doc)

If you test it inside the networked-aframe repo, be sure to use npm run dev-socketio and setting networked-scene="adapter:socketio" in the basic.html or basic-chat.html examples.

Be aware there are pending issues with the socketio adapter:

That's a quick implementation I tested only with maxOccupantsInRoom = 3 instead of 50.
I didn't test the load. Someone has to create a script to test 3000 connections, creating the websocket, sending avatar positions every second, do that for a random duration about 30 seconds to 3 min for each connection and close the websocket connection to see how the node process supports the load.

If someone wants to go further, some things to be tested for cpu/memory/latency:

  • Use a more performance framework, replace express by fastify (nodejs only) or hono (on deno or bun), or h3 (on node or deno or bun)
  • Replace socketio by a more efficient websocket implementation (ws on nodejs?), modify the socketio adapter to use WebSocket directly instead of socketio.
  • Or rewrite the server part entirely in golang or rust/tokio or elixir/phoenix

@vincentfretin vincentfretin merged commit 1d0977d into master Mar 22, 2024
@vincentfretin vincentfretin deleted the room-instancing branch March 22, 2024 13:23
@Utopiah
Copy link
Contributor

Utopiah commented Apr 1, 2024

I'd be curious about metrics here, i.e how many users (10, 100, 1000...) managed on which hardware (RPi Zero, RPi5, i9...) and what is the bottleneck.

@vincentfretin
Copy link
Member Author

If you need perf, I'll probably rewriting server/socketio-server.js with
https://github.com/uNetworking/uWebSockets.js
You can still use socketio lib on the client side I guess.

@vincentfretin
Copy link
Member Author

vincentfretin commented Nov 30, 2024

test done on my machine, 32GB of memory, 8 cores
50 users on http://localhost:8080/basic.html

socketio adapter

a room of 50 users is about 30% cpu for networked-aframe-server, 0.3% memory, smooth movement, my 8 cores were at 78% for 50 chrome windows
https://x.com/vincentfretin/status/1863320216735764693
5 rooms of 10 users is 10% cpu (I changed the maxOccupantsInRoom variable to 10 in server/socketio-server.js)

wseasyrtc adapter

Tested with easyrtc.setOption("logLevel", "info") instead of debug in server/easyrtc-server.js

a room of 50 users is about 80% cpu for networked-aframe-server, 1.7% memory, some choppy movements, I don't know if it's server side or client side, probably both, because I was testing both clients and server on the same machine, my 8 cores were at 95%
20 users in a room is smooth
30 users smooth at the beginning but we lost a few during the test

easyrtc adapter (so WebRTC datachannel instead of WebSocket for data, no audio)

a room of 50 users, very choppy movement
networked-aframe-server 60%, increasing memory +0.1% every second, memory not freed at the end

janus adapter

20 users in a room with audio, cpu 84%, memory 0.3%, some choppy movements
transports over datachannel
EVENT_LOOPS=4 with adapter.setEventLoops(4)
MESSAGE_THREADS=1
MAX_ROOM_SIZE=20
ALLOW_LOOP_INDICATION=true

20 users in a room with audio, cpu 78%, memory 0.3%, some choppy movements, my 8 cores were at 98% for 20 chrome windows
transports over websocket
EVENT_LOOPS=4 without adapter.setEventLoops(4)
MESSAGE_THREADS=1
MAX_ROOM_SIZE=20
ALLOW_LOOP_INDICATION=false

20 users in a room with audio, cpu 63%, memory 0.3%, some choppy movements with networked-audio-source="positional: true", smooth movements without networked-audio-source or with networked-audio-source="positional: false", my 8 cores were at 68% for 20 chrome windows. With networked-audio-source="positional: true", my 8 cores were at 88%
transports over websocket
EVENT_LOOPS=4 with adapter.setEventLoops(4)
MESSAGE_THREADS=1
MAX_ROOM_SIZE=20
ALLOW_LOOP_INDICATION=true

Chrome profiling gives sometimes linearRampToValueAtTime 4.26 ms with positional audio, so hearing the positional audio is really the bottleneck on the browser.

@vincentfretin
Copy link
Member Author

@Utopiah for now maximum is 70 users in a room with socketio server with socketio 4.8.1 running with node or bun on a Raspberry Pi 400 (ARM Cortex-A72 Quad-core 1.8 GHz)
https://x.com/vincentfretin/status/1865678789906161967

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants