WEBRTC+BABYLONJS使用案例
1.说明
由于工作有webgl+云渲染的需求，因此本人写了一个主流实时通讯技术WEBRTC+开源web引擎Babylonjs实现的实时云渲染案例，可以给有做类似项目的同学们参考，有啥问题也欢迎私信。
代码自取：链接：https://pan.baidu.com/s/1Tcrpviae_5vr6AeI6SSdWQ?pwd=3ncn
提取码：3ncn
2.技术说明
2.1webrtc
WebRTC (Web Real-Time Communications) 是一项实时通讯技术，它允许网络应用或者站点，在不借助中间媒介的情况下，建立浏览器之间点对点（Peer-to-Peer）的连接，实现视频流和（或）音频流或者其他任意数据的传输。WebRTC 包含的这些标准使用户在无需安装任何插件或者第三方的软件的情况下，创建点对点（Peer-to-Peer）的数据分享和电话会议成为可能。
简单总结一下：WebRTC可以在不用流媒体服务器的情况下，让浏览器与浏览器（或者客户端与浏览器之间）完成点对点通讯，从而实现云渲染。其中重中之重就是一定要有信令服务器的支持。
2.2Babylonjs
目前市面上开源的webgl引擎，本案例就用它来做本地渲染以实现原始时评流的生成。
官网：Babylon.js: Powerful, Beautiful, Simple, Open - Web-Based 3D At Its Best
本案例用的demo：Babylon.js Playground
因为本案例主要说明云渲染流程，所以就不讲webgl引擎的相关技术原理了。
2.3信令服务器
简单的服务器，主要用来实现渲染端与显示交互端的点对点数据传输，主要实现登录、房间匹配、Webrtc的Peer对象的传递
2.4Nodejs/ES
用于搭建信令服务器和前端引擎的交互操作，开发工具就不再进行介绍。
3.具体实现
额外说明：本文将渲染端称为a端、交互显示端称为b端、信令服务器称为服务器。
3.1实现服务器逻辑
在开始用代码实现服务器逻辑之前，我们要搞清楚WebRTC的逻辑，我这里简单的总结一下，如果有不明白的同学可以进一步查看相关文档。
①服务器开始运行
②a端加入服务器：a端调用接口，在服务器内创建房间room1
③服务器接到创建房间指令，生成房间对象room1{用户：a端}
④b端加入服务器：b端输入房间号，调用接口加入房间
⑤服务器收到b端加入信息后，通知a端有用户b已经加入进来了
⑥a端接收到b端加入信息，调用自身CreateOffer方法并发送offer给服务器
⑦服务器接收到a发来的offer，将offer转交给同房间的b，同时a创建peerConnection对象。
⑧b收到来自a的offer，调用自身的CreateAnswer方法answer给服务器，同时b创建peerConnection对象并保存offer中关于a的信息。
⑨服务器收到来自b的answer，将answer转交给同房间的a
⑩a收到answer同时保存answer中关于b的信息。此时offer和answer已经完成，即已经建立链接，可以进行流媒体的传输了。
3.2在a端的引擎中完成图像渲染
a端的文件结构如下：
在html内完成对babylon的引用，js文件内实现渲染逻辑；实质上就是在html页面内放置一个canvas面板、然后调用图形引擎接口在画布内完成渲染。
html代码如下：
RenderStreamingDemo
This is RenderStreamingDemo
js的引擎初始化部分代码如下
//init babylon engine
var canvas = document.getElementById("renderCanvas");var startRenderLoop = function (engine, canvas) {engine.runRenderLoop(function () {if (sceneToRender && sceneToRender.activeCamera) {sceneToRender.render();}});
}var engine = null;
var scene = null;
window[engine] = engine;
var sceneToRender = null;
var createDefaultEngine = function () { return new BABYLON.Engine(canvas, true, { preserveDrawingBuffer: true, stencil: true, disableWebGL2Support: false }); };
var createScene = function () {// This creates a basic Babylon Scene object (non-mesh)var scene = new BABYLON.Scene(engine);// This creates and positions a free camera (non-mesh)var camera = new BABYLON.FreeCamera("camera1", new BABYLON.Vector3(0, 5, -10), scene);// This targets the camera to scene origincamera.setTarget(BABYLON.Vector3.Zero());// This attaches the camera to the canvascamera.attachControl(canvas, true);// This creates a light, aiming 0,1,0 - to the sky (non-mesh)var light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 1, 0), scene);// Default intensity is 1. Let's dim the light a small amountlight.intensity = 0.7;// Our built-in 'sphere' shape.var sphere = BABYLON.MeshBuilder.CreateSphere("sphere", { diameter: 2, segments: 32 }, scene);// Move the sphere upward 1/2 its heightsphere.position.y = 1;// Our built-in 'ground' shape.var ground = BABYLON.MeshBuilder.CreateGround("ground", { width: 6, height: 6 }, scene);return scene;
};
//注册初始化函数
window.initFunction = async function () {var asyncEngineCreation = async function () {try {return createDefaultEngine();} catch (e) {console.log("the available createEngine function failed. Creating the default engine instead");return createDefaultEngine();}}window.engine = await asyncEngineCreation();if (!engine) throw 'engine should not be null.';startRenderLoop(engine, canvas);window.scene = createScene();
};
initFunction().then(() => {sceneToRender = scene
});// Resize
window.addEventListener("resize", function () {engine.resize();
});这个时候打开html就可以看到渲染出来的三维场景了
3.3完成服务器的中转逻辑
服务器主要的中转逻辑如下：
1.创建仅能容纳两人的房间，并处理进入房间/退出房间的逻辑。
2.接收房间用户的offer和answer并传递给另外一个人。
3.offer，answer互通完毕后，再互通iceCandidate对象
代码结构及内容如下：
首先定义房间类
class RTCMap{//房间列表entries ;constructor(){this.entries = new Array();}//创建房间put=(key,value)=>{if(!key||!value){return;}var index = this.getIndex(key);if(index<0){let entry = new Object();entry.key = key;entry.value = value;this.entries.push(entry);}else{let entry = new Object();entry.key = key;entry.value = value;this.entries[index] = entry;}}get=(key)=>{var index = this.getIndex(key);return (index<0)?null:this.entries[index].value;}remove=(key)=>{var index = this.getIndex(key);if(index>-1){this.entries.splice(index,1);}}clear=(key)=>{this.entries = new Array();}contains=(key)=>{var index = this.getIndex(key);return index>-1;}size=()=>{return this.entries.length;}getEntries=()=>{return this.entries;}getIndex=(key)=>{if(!key){return -1;}for (let index = 0; index < this.entries.length; index++) {const entry = this.entries[index];if(entry.key === key){return index;}}return -1;}
}
exports.RTCMap = RTCMap再实现信令服务器逻辑
var ws = require('nodejs-websocket');
var os = require('os');
var { RTCMap } = require('./map.js')
var port = 8098;const SIGNAL_TYPE_JOIN = "join"
const SIGNAL_TYPE_RESP_JOIN = "resp-join"
const SIGNAL_TYPE_LEAVE = "leave"
const SIGNAL_TYPE_NEW_PEER = "new-peer"
const SIGNAL_TYPE_PEER_LEAVE = "peer-leave"
const SIGNAL_TYPE_OFFER = "offer"
const SIGNAL_TYPE_ANSWER = "answer"
const SIGNAL_TYPE_CANDIDATE = "candidate"
const SIGNAL_ERROR = "error"
const SIGNAL_INFO = "info"const SIGNAL_BTN_CLICK = 'btn-click';
const SIGNAL_CANVAS_INTERACT = 'canvas-interact';
//用户类
class Client {uid;conn;roomId;constructor(uid, conn, roomId) {this.uid = uid;this.conn = conn;this.roomId = roomId;}
}var server = ws.createServer(function (conn) {console.log("创建一个新的链接");conn.sendText("your connection is received");var client;conn.on("text", function (str) {var msgJson = JSON.parse(str);switch (msgJson.cmd) {case SIGNAL_TYPE_JOIN:handleJoin(msgJson, conn);break;case SIGNAL_TYPE_LEAVE:handleLeave(msgJson, conn);break;case SIGNAL_TYPE_OFFER:handleOffer(msgJson, conn);break;case SIGNAL_TYPE_ANSWER:handleAnswer(msgJson, conn);break;case SIGNAL_TYPE_CANDIDATE:handleCandidate(msgJson, conn);break;case SIGNAL_BTN_CLICK:handleBtnClick(msgJson, conn);case SIGNAL_CANVAS_INTERACT:handleCanvasInteract(msgJson, conn);break;default:break;}});conn.on("close", function(code, reason) {console.info("connection is closed, code:" + code + ",reason:" + reason);if (conn.client) {var client = conn.client;var roomId = client.roomId;var uid = client.uid;var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('can not find the room: ' + roomId);sendError(conn, 'can not finde the room');return;}console.info('user: ' + uid + " has left the room: " + roomId);roomMap.remove(uid);conn.client = null;if (roomMap.size() > 0) {var leaveJsonMsg = {cmd: SIGNAL_TYPE_LEAVE,remoteUid: uid,}var clients = roomMap.getEntries();clients.forEach(client => {client.value.conn?.sendText(JSON.stringify(leaveJsonMsg));})}}});conn.on("error", function (err) {console.info("error is found:" + err);});
}).listen(port);var roomTableMap = new RTCMap();function handleJoin (msg, conn) {var roomId = msg.roomId;var uid = msg.uid;console.info('user: ' + uid + " has joined the room: " + roomId);var roomMap = roomTableMap.get(roomId);if (!roomMap) {roomMap = new RTCMap();roomTableMap.put(roomId, roomMap);}client = roomMap.get(uid);if (client) {sendError(conn, 'you have already joined the room,roomid: ' + roomId)return;}if (roomMap.size() >= 2) {sendError(conn, 'roomid: ' + roomId + ' is out of capacity!')return;}client = new Client(uid, conn, roomId);conn.client = client;if (roomMap.size() == 1) {//通知对方var remoteClients = roomMap.getEntries().map(item => item.value);var remoteJsonMsg = {cmd: SIGNAL_TYPE_NEW_PEER,remoteUid: uid,}remoteClients.forEach(remoteClient => {remoteClient.conn.sendText(JSON.stringify(remoteJsonMsg));})//通知自己var localJsonMsg = {cmd: SIGNAL_TYPE_RESP_JOIN,remoteUids: remoteClients.map(item => item.uid),}conn.sendText(JSON.stringify(localJsonMsg));}roomMap.put(uid, client);
}handleLeave = (msg, conn) => {var roomId = msg.roomId;var uid = msg.uid;console.info('user: ' + uid + " has left the room: " + roomId);var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('can not find the room: ' + roomId);sendError(conn, 'can not finde the room');return;}roomMap.remove(uid);conn.client = null;if (roomMap.size() > 0) {var leaveJsonMsg = {cmd: SIGNAL_TYPE_LEAVE,remoteUid: uid,}var clients = roomMap.getEntries();clients.forEach(client => {client.value.conn?.sendText(JSON.stringify(leaveJsonMsg));})}
}handleOffer = (msg, conn) => {let { roomId, uid, remoteUid, message } = msg;console.info('an offer received:');var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('handle offer can not find the room: ' + roomId);sendError(conn, 'can not finde the room: ' + roomId);return;}if (!roomMap.get(uid)) {console.error('handle offer can not find the user: ' + uid);sendError(conn, 'you are not in the room: ' + roomId);return;}var remoteClient = roomMap.get(remoteUid);if (remoteClient) {remoteClient.conn?.sendText(JSON.stringify(msg));}else {console.error('handle offer can not find the remote user: ' + remoteUid);sendError(conn, 'handle offer can not find the remote user: ' + remoteUid);}}handleAnswer = (msg, conn) => {let { roomId, uid, remoteUid, message } = msg;console.info('an answer received:');var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('handle answer can not find the room: ' + roomId);sendError(conn, 'can not finde the room: ' + roomId);return;}if (!roomMap.get(uid)) {console.error('handle answer can not find the user: ' + uid);sendError(conn, 'you are not in the room: ' + roomId);return;}var remoteClient = roomMap.get(remoteUid);if (remoteClient) {remoteClient.conn?.sendText(JSON.stringify(msg));}else {console.error('handle answer can not find the remote user: ' + remoteUid);sendError(conn, 'handle answer can not find the remote user: ' + remoteUid);}
}handleCandidate = (msg, conn) => {let { roomId, uid, remoteUid, message } = msg;console.info('a candidate received:');var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('handle candidate can not find the room: ' + roomId);sendError(conn, 'can not finde the room: ' + roomId);return;}if (!roomMap.get(uid)) {console.error('handle candidate can not find the user: ' + uid);sendError(conn, 'you are not in the room: ' + roomId);return;}var remoteClient = roomMap.get(remoteUid);if (remoteClient) {remoteClient.conn?.sendText(JSON.stringify(msg));}else {console.error('handle candidate can not find the remote user: ' + remoteUid);sendError(conn, 'handle candidate can not find the remote user: ' + remoteUid);}
}handleBtnClick = (msg, conn) => {let { roomId, uid, eventcode, eventvalue } = msg;console.info('btn click event received:');console.info(msg);var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('handle candidate can not find the room: ' + roomId);sendError(conn, 'can not finde the room: ' + roomId);return;}if (!roomMap.get(uid)) {console.error('handle candidate can not find the user: ' + uid);sendError(conn, 'you are not in the room: ' + roomId);return;}// var remoteClient = roomMap.get(remoteUid);// if (remoteClient) {// remoteClient.conn?.sendText(JSON.stringify(msg));// }// else {// console.error('handle candidate can not find the remote user: ' + remoteUid);// sendError(conn, 'handle candidate can not find the remote user: ' + remoteUid);// }var clients = roomMap.getEntries().map(item => item.value);clients.forEach(client => {if (client.key != uid)client.conn.sendText(JSON.stringify(msg));})
}
handleCanvasInteract = (msg, conn) => {let { roomId, uid, eventcode, eventvalue } = msg;var roomMap = roomTableMap.get(roomId);if (!roomMap) {console.error('handle candidate can not find the room: ' + roomId);sendError(conn, 'can not finde the room: ' + roomId);return;}if (!roomMap.get(uid)) {console.error('handle candidate can not find the user: ' + uid);sendError(conn, 'you are not in the room: ' + roomId);return;}var clients = roomMap.getEntries().map(item => item.value);clients.forEach(client => {if (client.key != uid)client.conn.sendText(JSON.stringify(msg));})}
sendError = (conn, content) => {var remoteJsonMsg = {cmd: SIGNAL_ERROR,content: content,}console.error(content);conn.sendText(JSON.stringify(remoteJsonMsg));
}
//服务器运行时返回访问路径
getIPAddress = () => {const interfaces = os.networkInterfaces();const addresses = [];for (const k in interfaces) {for (const k2 in interfaces[k]) {const address = interfaces[k][k2];if (address.family === 'IPv4') {addresses.push(address.address);}}}return addresses;
}
const addresses = getIPAddress();
for (const address of addresses) {console.log(`ws://${address}:${port}`);
}
3.4完成a端的调用webrtc及服务器逻辑
a段之前我们已经完成了三维场景的渲染了，剩下的就是和服务器的交互逻辑，无非也就是我们之前说的offer和answer，这里需要特别注意的是，三维图像的流化。
要用到一下方法
//canvas内容流化
var stream = canvas.captureStream();
//--------------------
//
//--------------------
//把流对象给peerConnection
stream.getTracks().forEach(track => {pc.addTrack(track, localStream);})然后完整代码如下（包括了之前的三维渲染部分）
'use strict'
//define the signal request type
const SIGNAL_TYPE_JOIN = "join"
const SIGNAL_TYPE_RESP_JOIN = "resp-join"
const SIGNAL_TYPE_LEAVE = "leave"
const SIGNAL_TYPE_NEW_PEER = "new-peer"
const SIGNAL_TYPE_PEER_LEAVE = "peer-leave"
const SIGNAL_TYPE_OFFER = "offer"
const SIGNAL_TYPE_ANSWER = "answer"
const SIGNAL_TYPE_CANDIDATE = "candidate"
const SIGNAL_ERROR = "error"
const SIGNAL_INFO = "info"
//例如 eventcode 0为点击灯光强度按钮
const SIGNAL_BTN_CLICK = 'btn-click';
//例如eventcode：0为鼠标down 1为up 2为move 3为wheel 4为click ；botton: 0为左 1为中 2为右
const SIGNAL_CANVAS_INTERACT = 'canvas-interact';//var localUserId = Math.random().toString(36).substring(2);
var localUserId = 'engine-client';console.log('welcome,user: ' + localUserId)
var remoteUid = -1;var roomId = 0;var localStream;
var remoteStream;//init babylon enginevar canvas = document.getElementById("renderCanvas");var startRenderLoop = function (engine, canvas) {engine.runRenderLoop(function () {if (sceneToRender && sceneToRender.activeCamera) {sceneToRender.render();}});
}var engine = null;
var scene = null;
window[engine] = engine;
var sceneToRender = null;
var createDefaultEngine = function () { return new BABYLON.Engine(canvas, true, { preserveDrawingBuffer: true, stencil: true, disableWebGL2Support: false }); };
var createScene = function () {// This creates a basic Babylon Scene object (non-mesh)var scene = new BABYLON.Scene(engine);// This creates and positions a free camera (non-mesh)var camera = new BABYLON.FreeCamera("camera1", new BABYLON.Vector3(0, 5, -10), scene);// This targets the camera to scene origincamera.setTarget(BABYLON.Vector3.Zero());// This attaches the camera to the canvascamera.attachControl(canvas, true);// This creates a light, aiming 0,1,0 - to the sky (non-mesh)var light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 1, 0), scene);// Default intensity is 1. Let's dim the light a small amountlight.intensity = 0.7;// Our built-in 'sphere' shape.var sphere = BABYLON.MeshBuilder.CreateSphere("sphere", { diameter: 2, segments: 32 }, scene);// Move the sphere upward 1/2 its heightsphere.position.y = 1;// Our built-in 'ground' shape.var ground = BABYLON.MeshBuilder.CreateGround("ground", { width: 6, height: 6 }, scene);return scene;
};
window.initFunction = async function () {var asyncEngineCreation = async function () {try {return createDefaultEngine();} catch (e) {console.log("the available createEngine function failed. Creating the default engine instead");return createDefaultEngine();}}window.engine = await asyncEngineCreation();if (!engine) throw 'engine should not be null.';startRenderLoop(engine, canvas);window.scene = createScene();
};
initFunction().then(() => {sceneToRender = scene
});// Resize
window.addEventListener("resize", function () {engine.resize();
});var localVideo = document.querySelector("#localvideo");
var remoteVideo = document.querySelector("#remotevideo");var idText = document.getElementById('btn-roomid');
var joinBtn = document.getElementById("btn-join");
var leaveBtn = document.getElementById("btn-leave");var rtcEngine;class RtcEngine {wsurl;signaling;pc;constructor(wsurl) {this.init(wsurl);}init = (wsurl) => {this.wsurl = wsurl;this.signaling = null;}//----------------------------RtcEvent------------------------//#region sendMessage = (message) => {this.signaling.send(message);}handleNewPeer = (jsonMsg) => {remoteUid = jsonMsg.remoteUid;console.log('new peer: ' + remoteUid + ' has join');//Do Offerthis.offer();}handleRespJoin = (jsonMsg) => {console.log(jsonMsg);remoteUid = jsonMsg.remoteUids[0];console.log('a peer: ' + remoteUid + ' is in');}handlePeerLeave = (jsonMsg) => {remoteUid = jsonMsg.remoteUids[0];console.log('a peer:' + remoteUid + ' has left');localVideo.srcObject = null;if (pc) {this.pc.close();this.pc = null;}}handleOffer = (msg) => {console.info('handle offer:');console.info(msg);if (!this.pc) {this.pc = this.createPeerConnection();}var desc = JSON.parse(msg.msg);this.pc.setRemoteDescription(desc);this.answer();}handleAnswer = (msg) => {console.info('handle answer');console.info(msg);if (!this.pc) {this.pc = this.createPeerConnection();}var desc = JSON.parse(msg.msg);this.pc.setRemoteDescription(desc);}handleRemoteCandidate = (msg) => {console.info('handle remote candidate');console.info(msg);var candidate = JSON.parse(msg.msg);this.pc.addIceCandidate(candidate).catch((e) => {console.error('add ice candidate fail: ' + e)});}handleBtnClick = (msg) => {let { eventcode, eventvalue } = msg;switch (eventcode) {case 0:engine.scenes[0].lights[0].intensity = msg.eventvalue;break;default:break;}};handleCanvasInteract = (msg) => {console.log(msg);let { eventcode } = msg;switch (eventcode) {case 0://交互事件代码 TODObreak;default:break;}}onOpen = () => {console.log("websocket is open");}onMessage = (ev) => {try {var jsonMsg = JSON.parse(ev.data)} catch (error) {console.log(ev.data);return}switch (jsonMsg.cmd) {case SIGNAL_TYPE_NEW_PEER:this.handleNewPeer(jsonMsg);break;case SIGNAL_TYPE_RESP_JOIN:console.log(jsonMsg);this.handleRespJoin(jsonMsg);break;case SIGNAL_ERROR:alert("错误：" + jsonMsg.content)console.error(jsonMsg.content);break;case SIGNAL_TYPE_LEAVE:this.handlePeerLeave(jsonMsg);break;case SIGNAL_TYPE_OFFER:this.handleOffer(jsonMsg);break;case SIGNAL_TYPE_ANSWER:this.handleAnswer(jsonMsg);break;case SIGNAL_TYPE_CANDIDATE:this.handleRemoteCandidate(jsonMsg);case SIGNAL_BTN_CLICK:this.handleBtnClick(jsonMsg);break;case SIGNAL_CANVAS_INTERACT:this.handleCanvasInteract(jsonMsg);break;default:break;}}onError = function (ev) {console.log("websocket error:" + ev.data);}onClose = function (ev) {console.log("websocket is closed:" + ev.code + ",reason" + ev.reason);}createWebsocket = () => {rtcEngine.signaling = new WebSocket(this.wsurl);rtcEngine.signaling.onopen = this.onOpen;rtcEngine.signaling.onmessage = this.onMessage;rtcEngine.signaling.onerror = this.onError;rtcEngine.signaling.onclose = this.onClose;}//#endregion//----------------------------PC-------------------------------------------//#region offer = () => {if (!this.pc) {this.pc = this.createPeerConnection();}this.pc.createOffer().then(this.createOfferAndSendMessage).catch(this.handleCreateOfferError);}createPeerConnection = () => {var defaultConfiguration = {bundlePolicy: 'max-bundle',rctpMuxPolicy: 'require',iceTransportPolicy: 'all',iceServers: [{"urls": ["turn:192.168.194.128:3478?transport=udp","turn:192.168.194.128:3478?transport=tcp",],"username": "lqf","credential": "123456"},{"urls": ["stun:192.168.194.128:3478"]}]}let pc = new RTCPeerConnection(defaultConfiguration);pc.onicecandidate = this.handleIceCandidate;pc.ontrack = this.handleTrack;pc.onconnectionstatechange = this.handleConnectStateChange;pc.oniceconnectionstatechange = this.handdleIceConnectStateChange;localStream.getTracks().forEach(track => {pc.addTrack(track, localStream);})return pc;}handleIceCandidate = (event) => {console.log('handle ice candidate');console.log(event);if (event.candidate) {var jsonMsg = {cmd: SIGNAL_TYPE_CANDIDATE,roomId: roomId,uid: localUserId,remoteUid: remoteUid,msg: JSON.stringify(event.candidate)}rtcEngine.sendMessage(JSON.stringify(jsonMsg));console.info('handleCandidate message: ' + JSON.stringify(jsonMsg))}else {console.warn('end of candidtate')}}handleTrack = (event) => {console.log('handle track');remoteStream = event.streams[0];remoteVideo.srcObject = remoteStream;}handleConnectStateChange = () => {if (this.pc) {console.info("Connect State Change:" + this.pc.connectionState);}}handdleIceConnectStateChange = (event) => {if (this.pc) {console.info("Ice Connect State Change:" + this.pc.iceConnectionState);}}createOfferAndSendMessage = (session) => {this.pc.setLocalDescription(session).then(function () {var jsonMsg = {cmd: SIGNAL_TYPE_OFFER,roomId: roomId,uid: localUserId,remoteUid: remoteUid,msg: JSON.stringify(session)}rtcEngine.sendMessage(JSON.stringify(jsonMsg));console.info('create and send offer message: ')}).catch(function (error) {console.error('offer setlocaldescription fail: ' + error);});}handleCreateOfferError = () => {console.error('handle offer fail: ' + error);}answer = () => {this.pc.createAnswer().then(this.createAnswerAndSendMessage).catch(this.handleAnswerError)}createAnswerAndSendMessage = (session) => {this.pc.setLocalDescription(session).then(function () {var jsonMsg = {cmd: SIGNAL_TYPE_ANSWER,roomId: roomId,uid: localUserId,remoteUid: remoteUid,msg: JSON.stringify(session)}rtcEngine.sendMessage(JSON.stringify(jsonMsg));console.info('create and send answer message: ')}).catch(function (error) {console.error('answer setLocalDescription fail: ' + error);});}handleAnswerError = () => {console.error('handle answer fail: ' + error);}//#endregion}rtcEngine = new RtcEngine("ws://你的地址");rtcEngine.createWebsocket();
//--------------------------UI-------------------------------------
//#region
joinBtn.onclick = function () {roomId = idText.value;if (roomId)//初始化本地码流InitLocalStream();elsealert('please input roomId')
}leaveBtn.onclick = () => {roomId = idText.value;var leaveMsg = {cmd: SIGNAL_TYPE_LEAVE,roomId: roomId,uid: localUserId,}rtcEngine.sendMessage(JSON.stringify(leaveMsg));remoteStream.srcObject = null;CloseLocalStream();if (pc) {pc.close();pc = null;}
}
let CloseLocalStream = () => {if (localStream) {localStream.getTracks().forEach(track => {track.stop();})}
}function InitLocalStream() {// navigator.mediaDevices.getUserMedia({// audio:true,// vedio:true// }).then(OpenLocalStream).catch(function(e){// alert("getUserMedia() error: "+e.name)// })if (engine) {var stream = canvas.captureStream();OpenLocalStream(stream);}elsealert("error: babylon engine not found")}function OpenLocalStream(stream) {localVideo.srcObject = stream;localStream = stream;join(roomId);
}function join(roomId) {var joinMsg = {'cmd': SIGNAL_TYPE_JOIN,'roomId': roomId,'uid': localUserId,}var message = JSON.stringify(joinMsg);rtcEngine.sendMessage(message);
}function leave() {}
//#endregion3.5完成b端的调用webrtc及服务器逻辑
b端的内容和a端和相似，不同之处在于文件流的接收，如下：
//peerconnection注册事件
pc.ontrack = this.handleTrack;handleTrack = (event) => {console.log('handle track');remoteStream = event.streams[0];//remoteVideo就是一个html的video组件remoteVideo.srcObject = remoteStream;}完整代码如下
html：
RenderStreamingDemo
This is RenderStreamingDemo
js：
'use strict'
//define the signal request type
const SIGNAL_TYPE_JOIN = "join"
const SIGNAL_TYPE_RESP_JOIN = "resp-join"
const SIGNAL_TYPE_LEAVE = "leave"
const SIGNAL_TYPE_NEW_PEER = "new-peer"
const SIGNAL_TYPE_PEER_LEAVE = "peer-leave"
const SIGNAL_TYPE_OFFER = "offer"
const SIGNAL_TYPE_ANSWER = "answer"
const SIGNAL_TYPE_CANDIDATE = "candidate"
const SIGNAL_ERROR = "error"
const SIGNAL_INFO = "info"//例如 eventcode 0为点击灯光强度按钮
const SIGNAL_BTN_CLICK = 'btn-click';
//例如eventcode：0为鼠标down 1为up 2为move 3为wheel 4为click ；botton: 0为左 1为中 2为右
const SIGNAL_CANVAS_INTERACT = 'canvas-interact';//var localUserId = Math.random().toString(36).substring(2);
var localUserId = 'frondend-client';console.log('welcome,user: ' + localUserId)
var remoteUid = -1;var roomId = 0;var localStream;
var remoteStream;//init enginelet mainDiv = document.getElementById('mainDiv');
var remoteVideo = document.querySelector("#remotevideo");var idText = document.getElementById('btn-roomid');
var joinBtn = document.getElementById("btn-join");
var leaveBtn = document.getElementById("btn-leave");
var intensityBtn = document.getElementById("btn-intensity");var rtcEngine;class RtcEngine {wsurl;signaling;pc;constructor(wsurl) {this.init(wsurl);}init = (wsurl) => {this.wsurl = wsurl;this.signaling = null;}//----------------------------RtcEvent------------------------//#region sendMessage = (message) => {this.signaling.send(message);}handleNewPeer = (jsonMsg) => {remoteUid = jsonMsg.remoteUid;console.log('new peer: ' + remoteUid + ' has join');//Do Offerthis.offer();}handleRespJoin = (jsonMsg) => {console.log(jsonMsg);remoteUid = jsonMsg.remoteUids[0];console.log('a peer: ' + remoteUid + ' is in');}handlePeerLeave = (jsonMsg) => {remoteUid = jsonMsg.remoteUids[0];console.log('a peer:' + remoteUid + ' has left');}handleOffer=(msg)=>{console.info('handle offer:');console.info(msg);if(!this.pc){this.pc = this.createPeerConnection();}var desc = JSON.parse(msg.msg);this.pc.setRemoteDescription(desc);this.answer();}handleAnswer=(msg)=>{console.info('handle answer');console.info(msg);if(!this.pc){this.pc = this.createPeerConnection();}var desc = JSON.parse(msg.msg);this.pc.setRemoteDescription(desc);}handleRemoteCandidate=(msg)=>{ console.info('handle remote candidate');console.info(msg);var candidate = JSON.parse(msg.msg);this.pc.addIceCandidate(candidate).catch((e)=>{console.error('add ice candidate fail: '+e)});}onOpen = () => {console.log("websocket is open");}onMessage = (ev) => {try {var jsonMsg = JSON.parse(ev.data)} catch (error) {console.log(ev.data);return}switch (jsonMsg.cmd) {case SIGNAL_TYPE_NEW_PEER:this.handleNewPeer(jsonMsg);break;case SIGNAL_TYPE_RESP_JOIN:console.log(jsonMsg);this.handleRespJoin(jsonMsg);break;case SIGNAL_ERROR://alert("错误：" + jsonMsg.content)console.error(jsonMsg.content);break;case SIGNAL_TYPE_LEAVE:this.handlePeerLeave(jsonMsg);break;case SIGNAL_TYPE_OFFER:this.handleOffer(jsonMsg);break;case SIGNAL_TYPE_ANSWER:this.handleAnswer(jsonMsg);break;case SIGNAL_TYPE_CANDIDATE:this.handleRemoteCandidate(jsonMsg) break;default:break;}}onError = function (ev) {console.log("websocket error:" + ev.data);}onClose = function (ev) {console.log("websocket is closed:" + ev.code + ",reason" + ev.reason);}createWebsocket = () => {rtcEngine.signaling = new WebSocket(this.wsurl);rtcEngine.signaling.onopen = this.onOpen;rtcEngine.signaling.onmessage = this.onMessage;rtcEngine.signaling.onerror = this.onError;rtcEngine.signaling.onclose = this.onClose;}//#endregion//----------------------------PC-------------------------------------------//#region offer = () => {if (!this.pc) {this.pc = this.createPeerConnection();}this.pc.createOffer().then(this.createOfferAndSendMessage).catch(this.handleCreateOfferError);}createPeerConnection = () => {var defaultConfiguration = {bundlePolicy:'max-bundle',rctpMuxPolicy:'require',iceTransportPolicy:'all',//or "all"iceServers:[{"urls":["turn:192.168.194.128:3478?transport=udp","turn:192.168.194.128:3478?transport=tcp",],"username":"lqf","credential":"123456"},{"urls":["stun:192.168.194.128:3478"]}]}let pc = new RTCPeerConnection(defaultConfiguration);pc.onicecandidate = this.handleIceCandidate;pc.ontrack = this.handleTrack;pc.onconnectionstatechange = this.handleConnectStateChange;pc.oniceconnectionstatechange = this.handdleIceConnectStateChange;localStream?.getTracks().forEach(track => {pc.addTrack(track, localStream);})return pc;}handleIceCandidate = (event) => {console.log('handle ice candidate');console.log(event);if (event.candidate) {var jsonMsg = {cmd: SIGNAL_TYPE_CANDIDATE,roomId: roomId,uid: localUserId,remoteUid: remoteUid,msg: JSON.stringify(event.candidate)}rtcEngine.sendMessage(JSON.stringify(jsonMsg));console.info('handleCandidate message: ' + JSON.stringify(jsonMsg))}else {console.warn('end of candidtate')}}handleTrack = (event) => {console.log('handle track');remoteStream = event.streams[0];remoteVideo.srcObject = remoteStream;}handleConnectStateChange = ()=>{if(this.pc){console.info("Connect State Change:"+this.pc.connectionState);}}handdleIceConnectStateChange=(event)=>{if(this.pc){console.info("Ice Connect State Change:"+this.pc.iceConnectionState);}}createOfferAndSendMessage = (session) => {this.pc.setLocalDescription(session).then(function () {var jsonMsg = {cmd: SIGNAL_TYPE_OFFER,roomId: roomId,uid: localUserId,remoteUid: remoteUid,msg: JSON.stringify(session)}rtcEngine.sendMessage(JSON.stringify(jsonMsg));console.info('create and send offer message: ')}).catch(function (error) {console.error('offer setlocaldescription fail: ' + error);});}handleCreateOfferError = () => {console.error('handle offer fail: ' + error);}answer=()=>{this.pc.createAnswer().then(this.createAnswerAndSendMessage).catch(this.handleAnswerError) }createAnswerAndSendMessage=(session)=>{this.pc.setLocalDescription(session).then(function () {var jsonMsg = {cmd: SIGNAL_TYPE_ANSWER,roomId: roomId,uid: localUserId,remoteUid: remoteUid,msg: JSON.stringify(session)}rtcEngine.sendMessage(JSON.stringify(jsonMsg));console.info('create and send answer message: ')}).catch(function (error) {console.error('answer setLocalDescription fail: ' + error);});}handleAnswerError=()=>{console.error('handle answer fail: ' + error);}//#endregion}rtcEngine = new RtcEngine("ws://你的地址");rtcEngine.createWebsocket();
//--------------------------UI-------------------------------------
//#region
joinBtn.onclick = function () {roomId = idText.value;if (roomId)//初始化本地码流InitLocalStream();elsealert('please input roomId')
}leaveBtn.onclick = () => {roomId = idText.value;var leaveMsg = {cmd: SIGNAL_TYPE_LEAVE,roomId: roomId,uid: localUserId,}rtcEngine.sendMessage(JSON.stringify(leaveMsg));remoteVideo.srcObject = null;CloseLocalStream();if(this.pc){this.pc.close();this.pc = null;}
}
function CloseLocalStream (){if(localStream){localStream.getTracks().forEach(track=>{track.stop();})}
}
intensityBtn.onclick=()=>{let intensity = Math.random() + 0.5;var msgJson={cmd:SIGNAL_BTN_CLICK,roomId:roomId,uid:localUserId,eventcode:0,eventvalue:intensity,}rtcEngine.sendMessage(JSON.stringify(msgJson));
}let mouseX,mouseY;
remoteVideo.onmousedown =(e)=>{console.log(e);mouseX = e.offsetX,mouseY = e.offsetY;var msgJson={cmd:SIGNAL_CANVAS_INTERACT,roomId:roomId,uid:localUserId,eventcode:0,button:e.button,mouseDownX:e.offsetX/remoteVideo.clientWidth*1000,mouseDownY:e.offsetY/remoteVideo.clientHeight*1000}rtcEngine.sendMessage(JSON.stringify(msgJson));
}remoteVideo.onmouseup =(e)=>{var msgJson={cmd:SIGNAL_CANVAS_INTERACT,roomId:roomId,uid:localUserId,eventcode:1,mouseDownX:e.offsetX/remoteVideo.clientWidth*1000,mouseDownY:e.offsetY/remoteVideo.clientHeight*1000}rtcEngine.sendMessage(JSON.stringify(msgJson));
}remoteVideo.onmousemove = (e)=>{let offsetX = e.offsetX - mouseX;let offsetY = e.offsetY - mouseY;var msgJson={cmd:SIGNAL_CANVAS_INTERACT,roomId:roomId,uid:localUserId,eventcode:2,offsetX:offsetX/remoteVideo.clientWidth*1000,offsetY:offsetY/remoteVideo.clientHeight*1000}rtcEngine.sendMessage(JSON.stringify(msgJson));
}remoteVideo.onwheel=(e)=>{console.log(e);mouseX = e.offsetX,mouseY = e.offsetY;var msgJson={cmd:SIGNAL_CANVAS_INTERACT,roomId:roomId,uid:localUserId,eventcode:3,offsetX:e.offsetX/remoteVideo.clientWidth*1000,offsetY:e.offsetY/remoteVideo.clientHeight*1000,deltaY:e.deltaY,}rtcEngine.sendMessage(JSON.stringify(msgJson));
}function InitLocalStream() {// navigator.mediaDevices.getUserMedia({// audio:true,// vedio:true// }).then(OpenLocalStream).catch(function(e){// alert("getUserMedia() error: "+e.name)// })var stream = null//var stream = mxengine.canvas.captureStream();OpenLocalStream(stream);}function OpenLocalStream(stream) {localStream = stream;join(roomId);
}function join(roomId) {var joinMsg = {'cmd': SIGNAL_TYPE_JOIN,'roomId': roomId,'uid': localUserId,}var message = JSON.stringify(joinMsg);rtcEngine.sendMessage(message);
}function leave() {}
//#endregion3.6交互操作
往往用户需要操作b端来实现对a端的控制，具体实现就是在b端触发一系列事件，再将交互信息通过服务器传递给b端从而调用b端的方法来完成控制，具体实现也寂静给出，顺着b端的一些鼠标事件就可以找到具体的实现逻辑。
4.代码演示
请下载完整代码 链接：https://pan.baidu.com/s/1Tcrpviae_5vr6AeI6SSdWQ?pwd=3ncn
提取码：3ncn
①在server文件夹根目录下 运行 npm signal_server.js（请确保电脑安装了node）
②打开a_client中的html 输入一个房间号加入
③打开b_client中的html输入相同的房间号加入
④如果b端没有内容 请切换到a端再切回来 很可能是浏览器被最小化后不运作了
视频
renderstream-babylonjs webrtc
5.未完成的
这一套技术流程只能实现一对一且在一个局域网下的视频流传输，如果要在公网上提供服务，那么要用到流媒体服务器，如果有机会再来分享吧 。
本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！