Step 1: Fetch List of Ice Servers

The application will need to call the endpoint to get the list of ICE servers:

/accounts/${account}/devices/${deviceId}/videos/webrtcservers

An example of how this might look in JavaScript is:

await httpHelper.get(url, data);

The Twilio Service handles the available servers. Twilio will monitor and manage the STUN/TURN servers. If these servers go down, a Google version will spin up as a backup.

An Example Response is shown below:

Step 2: Compose SDP Offer

After retrieving the list of servers, the user will need to compose an SDP offer with the following data structure:

{
 uid: string,
 expiration_secs: 10,
 platform: 'mission_control_link',
 command: 'launch_webrtc_video',
 data: {
   'fr.webrtc.topics.uploadable': {},
   teleop_session: string,
   offer: {
     sdp: offer.sdp,
     type: offer.type
   },
   topic: string,
   max_fps: int,
   max_bandwidth: int,
   max_width: int,
   max_height: int,
   max_time_no_browser_ping: int,
   bandwidth_tuning: boolean,
   clean_start: boolean
 }
}

An example of this could look like:

{
  command: "launch_webrtc_video",
	data: {
  	bandwidth_tuning: true,
  	clean_start: false,
  	direct_video_source: null,
  	fr.webrtc.topics.uploadable: {/tf: 0.2, /tf_static: 5},
  	max_bandwidth: 1000,
  	max_fps: 30,
  	max_height: 480,
  	max_time_no_browser_ping: 20,
  	max_width: 640,
  	offer: {
    	sdp: "v=0\r\no=- 3075460600386794818 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS\r\nm=video 15531 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 122 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116 37\r\nc=IN IP4 34.203.250.39\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=candidate:204784042 1 udp 2113937151 f521e669-d813-4c52-9ecb-6255b7079e94.local 58369 typ host generation 0 network-cost 999\r\na=candidate:1477544869 1 udp 2113939711 b8598a9c-c604-4691-b780-27c6922c6507.local 63553 typ host generation 0 network-cost 999\r\na=candidate:842163049 1 udp 1677729535 70.119.66.130 58369 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:1658132311 1 udp 33562879 34.203.250.39 15531 typ relay raddr 70.119.66.130 rport 58369 generation 0 network-cost 999\r\na=candidate:765036628 1 udp 16785407 34.203.250.55 34682 typ relay raddr 70.119.66.130 rport 63757 generation 0 network-cost 999\r\na=candidate:765036628 1 udp 16785151 34.203.250.55 23816 typ relay raddr 70.119.66.130 rport 63758 generation 0 network-cost 999\r\na=ice-ufrag:fdON\r\na=ice-pwd:HyUkZT0kaIUe/5PMB1kq3Djk\r\na=ice-options:trickle\r\na=fingerprint:sha-256 03:CC:58:CF:8F:87:E5:24:C4:11:67:33:31:84:36:B7:4D:29:F1:CE:8E:D4:5E:FD:6E:A3:C9:50:0F:E3:52:57\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 urn:3gpp:video-orientation\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 VP9/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 profile-id=2\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:122 VP9/90000\r\na=rtcp-fb:122 goog-remb\r\na=rtcp-fb:122 transport-cc\r\na=rtcp-fb:122 ccm fir\r\na=rtcp-fb:122 nack\r\na=rtcp-fb:122 nack pli\r\na=fmtp:122 profile-id=1\r\na=rtpmap:102 H264/90000\r\na=rtcp-fb:102 goog-remb\r\na=rtcp-fb:102 transport-cc\r\na=rtcp-fb:102 ccm fir\r\na=rtcp-fb:102 nack\r\na=rtcp-fb:102 nack pli\r\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f\r\na=rtpmap:121 rtx/90000\r\na=fmtp:121 apt=102\r\na=rtpmap:127 H264/90000\r\na=rtcp-fb:127 goog-remb\r\na=rtcp-fb:127 transport-cc\r\na=rtcp-fb:127 ccm fir\r\na=rtcp-fb:127 nack\r\na=rtcp-fb:127 nack pli\r\na=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f\r\na=rtpmap:120 rtx/90000\r\na=fmtp:120 apt=127\r\na=rtpmap:125 H264/90000\r\na=rtcp-fb:125 goog-remb\r\na=rtcp-fb:125 transport-cc\r\na=rtcp-fb:125 ccm fir\r\na=rtcp-fb:125 nack\r\na=rtcp-fb:125 nack pli\r\na=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\na=rtpmap:107 rtx/90000\r\na=fmtp:107 apt=125\r\na=rtpmap:108 H264/90000\r\na=rtcp-fb:108 goog-remb\r\na=rtcp-fb:108 transport-cc\r\na=rtcp-fb:108 ccm fir\r\na=rtcp-fb:108 nack\r\na=rtcp-fb:108 nack pli\r\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\r\na=rtpmap:109 rtx/90000\r\na=fmtp:109 apt=108\r\na=rtpmap:35 AV1X/90000\r\na=rtcp-fb:35 goog-remb\r\na=rtcp-fb:35 transport-cc\r\na=rtcp-fb:35 ccm fir\r\na=rtcp-fb:35 nack\r\na=rtcp-fb:35 nack pli\r\na=rtpmap:36 rtx/90000\r\na=fmtp:36 apt=35\r\na=rtpmap:124 H264/90000\r\na=rtcp-fb:124 goog-remb\r\na=rtcp-fb:124 transport-cc\r\na=rtcp-fb:124 ccm fir\r\na=rtcp-fb:124 nack\r\na=rtcp-fb:124 nack pli\r\na=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032\r\na=rtpmap:119 rtx/90000\r\na=fmtp:119 apt=124\r\na=rtpmap:123 H264/90000\r\na=rtcp-fb:123 goog-remb\r\na=rtcp-fb:123 transport-cc\r\na=rtcp-fb:123 ccm fir\r\na=rtcp-fb:123 nack\r\na=rtcp-fb:123 nack pli\r\na=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032\r\na=rtpmap:118 rtx/90000\r\na=fmtp:118 apt=123\r\na=rtpmap:114 red/90000\r\na=rtpmap:115 rtx/90000\r\na=fmtp:115 apt=114\r\na=rtpmap:116 ulpfec/90000\r\na=rtpmap:37 flexfec-03/90000\r\na=rtcp-fb:37 goog-remb\r\na=rtcp-fb:37 transport-cc\r\na=fmtp:37 repair-window=10000000\r\nm=application 24454 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 34.203.250.39\r\na=candidate:204784042 1 udp 2113937151 f521e669-d813-4c52-9ecb-6255b7079e94.local 51223 typ host generation 0 network-cost 999\r\na=candidate:1477544869 1 udp 2113939711 b8598a9c-c604-4691-b780-27c6922c6507.local 56222 typ host generation 0 network-cost 999\r\na=candidate:842163049 1 udp 1677729535 70.119.66.130 51223 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\r\na=candidate:1658132311 1 udp 33562879 34.203.250.39 24454 typ relay raddr 70.119.66.130 rport 51223 generation 0 network-cost 999\r\na=candidate:765036628 1 udp 16785151 34.203.250.55 43331 typ relay raddr 70.119.66.130 rport 63760 generation 0 network-cost 999\r\na=candidate:765036628 1 udp 16785407 34.203.250.55 53331 typ relay raddr 70.119.66.130 rport 63759 generation 0 network-cost 999\r\na=ice-ufrag:fdON\r\na=ice-pwd:HyUkZT0kaIUe/5PMB1kq3Djk\r\na=ice-options:trickle\r\na=fingerprint:sha-256 03:CC:58:CF:8F:87:E5:24:C4:11:67:33:31:84:36:B7:4D:29:F1:CE:8E:D4:5E:FD:6E:A3:C9:50:0F:E3:52:57\r\na=setup:actpass\r\na=mid:1\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n",
    	type: "offer"
  	},
  	teleop_session: "wmsq39xu4r",
  	topic: "/head_camera/rgb/image_raw",
	}
	expiration_secs: 10,
	platform: "mission_control_link",
	uid: "wmsq39xu4r"
}

The API endpoint this sdp offer should call is:

/accounts/${account}/devices/${deviceId}/commands

an example in JavaScript is:

await httpHelper.put(url, data);

An example response is shown below:

Step 3: Read the Command Response

The application will have to poll the device making the webrtc connection to read the command response. An SDP answer will be returned.

To read the command response, poll the data endpoint for the device making the webrtc connection.

/accounts/${account}/devices/${deviceId}/data

The data endpoint will need params: utc_start and utc_now are in milliseconds from 1970.

const utc_start = Date.now() / 1000 - 90;
const utc_end = Date.now() / 1000 + 30;
let params = {
 platform: 'mission_control_link',
 utc_start,
 utc_end,
 tables: 'hourly'
};

an example in JavaScript would be:

await httpHelper.get(url, params);

While checking the WebRTC Session, the application should be looking for the object type "command_completed". This will indicate the connection completed.

type: "command_completed"

Step 4: Negotiate the Web RTC Connection

With the agent installed on the device, proceed to negotiate WebRTC connection. Negotiating the WebRTC connection will be dependent on the library/package you use to establish the connection with.

Step 5: Sending a Heartbeat

To retain the connection, a heart beat is required in the format of:

let command = {
    type: 'freedom_msgs/Ping',
    topic: 'ping',
    message: {command: 'ping'},
    expiration_secs: 5,
    platform: 'freedom_webrtc',
    uid: utils.getRandomKey(3), // must be unique on every heartbeat
    utc_ping_secs: time         // current utc_time
  };

The data channel returns an object with the current utc_time that should be sent to prevent webRTC from thinking the connection is faulty.

For example, you can create an event trigger when creating the data channel. This callback could be executed when a message is sent back from the data channel.

// a call back is created, so that when a message is returned
// the data channel is sent an empty command
pc.ondatachannel = event => {
  event.channel.onmessage = event => {
    sendCommand({});
  };
};

// send command will open the data channel and send an object with the current
// utc_time
sendCommand = () => {
  let commandDataChannel = pc.createDataChannel('CommandDataChannel');

  let cmd = {};
  cmd.utc_time = dateHelper.getNow(true);
  commandDataChannel.send(JSON.stringify(cmd));
}

Step 6: Sending Data through the Data Channel

This is up to the party that established the data channel. Work with the client to understand the expected payload and create an object the client can digest. Preferably it is in the form the client will interact with the data. For example, a ROS Topic like sensor_msgs/Joy .