TWebRTCCommunication

The TWebRTCCommunication is a component to enable communication between two or more peers by using the WebRTC API.

Connection overview

The flow of a connection is as follows:

1. The connection is initiated by peer1. peer1 creates an offer and sends this offer to the signaling server.
2. The signaling server forwards this offer to peer2.
3. When peer2 receives the offer, it can accept it. Upon accepting the offer from peer1 an answer will be generated. This answer then is sent to the signaling server.
4. The signaling server forwards the answer from peer2 to peer1.
5. When the answer arrives to peer1, it accepts it.
6. While the offer/answer is created, ICE candidates are started to being generated by each peer. These candidates are also sent to the signaling server where the signaling server is repsonsible for forwarding the candidates to the correct peers.*
7. When an ICE candidate arrives, the peer accepts it.*
8. The connection between two peers is now estabilished.

*Step 6-7 is optional. It is possible to wait with sending the offer and answer until all candidates are gathered. In that case, the offer and answer will contain the candidates. By doing that, it's guaranteed the candidate will arrive properly and at the correct time but it also results in a slower connection establishment.

Signaling server

It's not specified in the WebRTC standards what the signaling server should be. You can use anything that fits your application best as long as the offers, answers and candidates can be exchanged between peers.

Camera and microphone access

Requesting the devices

Use the RequestDevices method to request a list of available devices. By default this is not requested automatically as it requires user permission that results in a popup from the browser. In web applications it's best to let the request happen from a user interaction or from a page where the user anticipates the permissions will be requested.

procedure TForm1.WebFormCreate(Sender: TObject);
begin
  //Request both audio and video devices
  WebRTCCommunication1.RequestUserMedia(True, True);
end;

The OnDeviceListInitialized event will be triggered when the device list is returned. Use the DeviceList property to loop through the returned devices. Each DeviceList item will have a Name, ID and a Kind property mapping onto the underlying MediaDeviceInfo objects' label, deviceId and kind properties.

procedure TForm1.WebRTCCommunication1DeviceListInitialized(Sender: TObject);
var
  I: Integer;
  device: TRTCDevice;
begin
  camList.Clear;
  micList.Clear;

  for I := 0 to WebRTCCommunication1.DeviceList.Count - 1 do
  begin
    device := WebRTCCommunication1.DeviceList.Items[I];
    if device.Kind = rdkVideoInput then
      camList.Items.AddPair(device.Name, device.ID)
    else if device.Kind = rdkAudioInput then
      micList.Items.AddPair(device.Name, device.ID);
  end;
end;

Requesting device access

Use the RequestUserMedia method to request access to the user's camera and/or microphone.

If you request a device that does not exists (for example: AAudio parameter is True or AAudioID is a non-empty string but the user does not have a microphone), the request will be rejected. Unless you are 100% certain the user has the given devices, it's better to request the available devices and make the request based on that.

Before the request is executed, the OnBeforeRequestUserMedia event is triggered allowing the modification of the the constraints object.

Once the request was approved by the user, the OnAfterRequestUserMedia event will be triggered.

Mute and unmute

Use the LocalVideoSettings.EnableVideoTrack and LocalVideoSettings.EnableAudioTrack properties to show/hide video feed and mute/unmute audio feed.

Screen sharing

It is also possible for the user to share their screen as a video source in the WebRTC communication. This is done by calling RequestDisplay. RequestDisplay internally maps onto the MediaDevices.getDisplayMedia API.

No mobile support

The getDisplayMedia API is NOT supported on mobile platforms.

Calling RequestDisplay will prompt the user to select a tab/window/screen to share with the option to enable the audio if the AAudio parameter is set to True. This prompt is handled by the browser, it's not possible to list the available tabs/windows/screens.

procedure TForm1.WebButton1Click(Sender: TObject);
begin
  WebRTCCommunication1.RequestDisplay(True);
end;

The resulting video stream can be used instead of a camera feed as a video source in the WebRTC communication. It is also possible to change the video feed in an already negotiated connection by simply calling RequestUserMedia/RequestDisplay again allowing to switch between camera and screen feeds.

Adding peers

Peers can be added both at design-time and programmatically.

Each Peers collection item has a PeerMediaPlayer property to which a TWebMultimediaPlayer can be assigned. This way you can decide where the output video/audio stream should be shown and it's possible to take advantage of dynamic element placement via TWebMultimediaPlayer.ElementID too.

If there are multiple peer connections, it is recommended to fill in the PeerID. This ID will be used to search for the correct peer when calling CreateOffer, SendData (etc...) methods. It is also used in the OnOffer/OnAnswer/OnIceCandidate (etc...) events to identify which peer connection triggered the event.

ICE servers

Interactive Connectivity Establishment (ICE) is a framework to allow your web browser to connect with peers. ICE is using STUN and/or TURN servers to get the public IP address of a peer and/or relay data through the server if the router does not allow direct connection.

Use the DefaultIceServers property to set up the server list that you plan on sharing between multiple peers. Whenever a new peer is added its IceServer property will be identical to the DefaultIceServers. Modifying the DefaultIceServer property won't affect the existing peers' IceServer property.

Data channels

TWebRTCCommunication also provides a way to create data channels between peers. These data channels can be used to send some data directly to a peer through the browser connection (such as chat messages, images, etc...).

Creating data channels

Each Peer item in the Peers collection has its own DataChannels collection as a single peer connection can have multiple data channels at a time.

The offer needs to contain the data channels one wants to estabilish between the two peers. To put this simply, make sure your offer sending peer will always have the necessary data channels added to the DataChannels collection so it can create those data channels by sending them as part of the offer.

If a data channel is added later than the connection is established, the connection needs to be renegotiated.

Sending messages

Once the data channels connect, the OnDataChannelOpen event will be triggered. From that moment it's possible to send messages by using the SendData method. This method has two overloads, one expects a string for text based data and the other one expects an ArrayBuffer for binary data.

To send messages, call either the SendData method of the Peer item or the SendData method of the TWebRTCCommunication with the ID of the peer:

procedure TForm1.WebButton1Click(Sender: TObject);
begin
  WebRTCCommunication1.SendData('peer id', 'channel name', 'my text');
end;

Receiving messages

When a message arrives to a peer, it will trigger either the OnDataChannelStringMessage or OnDataChannelArrayBufferMessage message depending on the format the message was sent in.

Example of receiving string messages:

procedure TForm1.WebRTCConnection1DataChannelStringMessage(Sender: TObject; APeerID,
  AChannelName, AMessage: string);
begin
  Memo1.Lines.Add('A message arrived from ' + APeerID + ' on the ' + AChannelName + ' channel: ' + AMessage);
end;

Create and accept offers/answers

To initiate a connection from one peer, first an offer needs to be created.

procedure TForm1.Button1Click(Sender: TObject);
begin
  WebRTCCommunication1.CreateOffer('peerID');
end;

When the offer is created, the OnOffer event will be triggered where the AOffer.description JavaScript object parameter contains the offer. When this event is triggered, the offer does not contain any candidates yet. After the candidates are gathered, the OnOfferWithCandidates event will be triggered, where similar to OnOffer the AOffer.description object contains the offer.

procedure TForm1.WebRTCCommunication1Offer(Sender: TObject; AID: string;
  AOffer: TJSRTCSessionDescriptionRecord);
begin
  SendToServer('{"type": "offer", "peerId": "' + AID + '", "offer": ' + TJSJSON.stringify(AOffer.description) + '}');
end;

When a peer gets an offer (through the signaling server) it needs to accept, use the AcceptOffer method. The first AID parameter is the PeerID of the peer in the Peers collection. The AOfferparameter can be a string (must be the offer in a JSON string format) or the RTCSessionDescription offer object.

Upon accepting an offer, the peer will automatically create an answer and the OnAnswer event will be triggered. Similarly to OnOffer, the AAnswer.description parameter will contain the answer in JSON string format. The OnAnswerWithCandidates works the same way as OnOfferWithCandidates.

procedure TForm1.WebRTCCommunication1Answer(Sender: TObject; AID: string;
  AAnswer: TJSRTCSessionDescriptionRecord);
begin
  SendToServer('{"type": "answer", "peerId": "' + AID + '", "answer": ' + TJSJSON.stringify(AAnswer.description) + '}');
end;

Whenever an answer arrives through the signaling server, it can be accepted by calling AcceptAnswer where the first AID parameters is the PeerID of the peer in the Peers collection. The AAnswer parameter can be a string (must be the answer in a JSON string format) or the RTCSessionDescription answer object.

Changes in connection and reconnecting

Whenever the ICE connection state changes, the OnIceConnectionStateChange event triggers. For example, this event can be used to detect losing a connection if the parameter is a rcsFailed. This means the connection cannot be restored by itself and it needs to be created again.

To restart a connection, simply call RestartIce. It will request to gather the ICE candiates so the connection on both ends can be redone. It will also cause the OnNegotationNeeded event to be fired from where a new offer can be created to restore the connection - if the application logic requires that.

Properties

Property name	Description
DefaultIceServers	With multi-peer connection it's easier to set the ICE servers once via this property. When a new item is added to the Peer collection, the ICEServer property will automatically be set to this.
DeviceList	This property will contain the available devices after requesting them.
LocalVideoSettings	Various setting for the user's video access.
Peers	A collection of peer connections.
UserMediaPlayer	The TWebMultimediaPlayer control which plays the video/audio feed of the user.

Methods

Method name	Description
AcceptAnswer(AID: string; AAnswer: string)	Accepts an answer formatted in a JSON string. The AID represents the ID of the peer in the Peers collection.
AcceptAnswer(AID: string; AAnswer: TJSRTCSessionDescriptionInitRecord)	Accepts an answer as an RTCSessionDescription JavaScript object. The AID represents the ID of the peer in the Peers collection.
AcceptIceCandidate(AID: string; AIceCandidate: string)	Accepts an ICE candidate formatted in a JSON string. The AID represents the ID of the peer in the Peers collection.
AcceptIceCandidate(AID: string; AIceCandidate: TJSRTCIceCandidateInitRecord)	Accepts an ICE candidate as an RTCIceCandidate JavaScript object. The AID represents the ID of the peer in the Peers collection.
AcceptOffer(AID: string; AOffer: string)	Accepts an offer formatted in a JSON string. The AID represents the ID of the peer in the Peers collection.
AcceptOffer(AID: string; AOffer: TJSRTCSessionDescriptionInitRecord)	Accepts an offer as an RTCSessionDescription JavaScript object.. The AID represents the ID of the peer in the Peers collection.
AddPeerConnection(AID: string): TWebRTCPeer	Adds a peer to the Peers collection.
CloseConnection(AID: string)	Closes a connection to a peer. The AID represents the ID of the peer in the Peers collection.
CreateOffer(AID: string): TJSPromise	Creates an offer for a peer. The AID represents the ID of the peer in the Peers collection.
RequestDevices(AVideo: Boolean; AAudio: Boolean)	Request a list of the available devices from the user.
RequestDisplay	Requests the user to pick a tab/window/screen to share as a video stream.
RequestUserMedia(ACamera: Boolean; AAudio: Boolean)	Requests access to the user's video and audio devices.
RequestUserMedia(ACameraID: string; AAudioID: string)	Requests access to the user's video and audio devices by ID.
RestartIce(AID: string)	Call to restart the ICE candidate generation. The AID represents the ID of the peer in the Peers collection.
SendData(APeerId, AChannelName, AMessage: string)	Sends string data through an open data channel.
SendData(APeerId, AChannelName: string; AMessage: TJSArrayBufferRecord)	Sends ArrayBuffer data through an open data channel.
StopUserMedia	Stops the requested user media of the user.
StopDisplayStream	Stops the display stream of the user.

Events

Event name	Description
OnAfterRequestDisplayMedia	Event triggered after getDisplayMedia is successfully executed. The resulting media stream object can be accessed from this event.
OnAfterRequestUserMedia	Event triggered after getUserMedia is successfully executed. The resulting media stream object can be accessed from this event.
OnAnswer	Event triggered when an answer was generated.
OnAnswerWithCandidates	Event triggered when the candidates are gathered after generating an answer.
OnBeforeRequestUserMedia	Event triggered before getUserMedia is executed. From this event the constraints object can be modified.
OnConnectionStateChange	Event maps onto the connectionstatechange event.
OnDataChannelArrayBufferMessage	Event triggers when an ArrayBuffer message arrives through a data channel.
OnDataChannelOpen	Event triggers when a data channel opens.
OnDataChannelStringMessage	Event triggers when a string message arrives through a data channel.
OnDeviceListInitialized	Event triggered when the requested device list is initialized.
OnIceCandidate	Event triggered when an ICE candidate was generated.
OnIceConnectionStateChange	Event triggered when the ICE connection state changes. Maps onto the iceconnectionstatechange event.
OnNegotiationNeeded	Event triggered when the negotiation of the connection through the signaling server is required. Maps onto the negotiationneeded event.
OnOffer	Event triggered when an offer was generated.
OnOfferWithCandidates	Event triggered when the candidates are gathered after generating an offer.
OnRequestUserMediaError	Event triggers when an exception happens during the getUserMedia call. Reasons for this can be: unsecure context, user does not have a requested device, user rejects the request.
OnSignalingStateChange	Maps onto the signalingstatechange event.
OnTrack	Maps onto the track event.
OnUserMediaStart	Event triggered when the video or audio of the user starts playing.