TTMSFNCWXWebRTCCommunication

The TTMSFNCWXWebRTCCommunication is a component to enable communication between two or more peers by using the WebRTC API.

Connection overview

The flow of a connection is as follows:

The connection is initiated by peer1. peer1 creates an offer and sends this offer to the signaling server.
The signaling server forwards this offer to peer2.
When peer2 receives the offer, it can accept it. Upon accepting the offer from peer1 an answer will be generated. This answer then is sent to the signaling server.
The signaling server forwards the answer from peer2 to peer1.
When the answer arrives to peer1, it accepts it.
While the offer/answer is created, ICE candidates are started to being generated by each peer. These candidates are also sent to the signaling server where the signaling server is repsonsible for forwarding the candidates to the correct peers.*
When an ICE candidate arrives, the peer accepts it.*
The connection between two peers is now estabilished.

*Step 6-7 is optional. It is possible to wait with sending the offer and answer until all candidates are gathered. In that case, the offer and answer will contain the candidates. By doing that, it's guaranteed the candidate will arrive properly and at the correct time but it also results in a slower connection establishment.

Signaling server

It's not specified in the WebRTC standards what the signaling server should be. You can use anything that fits your application best as long as the offers, answers and candidates can be exchanged between peers.

Camera and microphone access

Requesting the devices

By default the device list is requested when the component has initialized. The OnDeviceListInitialized event will be triggered when the device list is returned. Use the DeviceList property to loop through the returned devices.

If the AutoRequestVideoDevices is set to False the component will only request the audio devices.

If the AutoRequestAudioDevices is set to False the component will only request the video devices.

Use the RequestDevices method to request a list of available devices again.

Important

On Android manually calling RequestDevices or automatically requesting the available devices via the AutoRequestVideoDevices and AutoRequestAudioDevices properties requires an active internet connection.

procedure TForm1.TMSFNCWXWebRTCCommunication1DeviceListInitialized(
  Sender: TObject);
var
  I: Integer;
  device: TTMSFNCWXWebRTCDevice;
begin
  camCB.Clear;
  micCB.Clear;

  for I := 0 to TMSFNCWXWebRTCCommunication1.DeviceList.Count - 1 do
  begin
    device := TMSFNCWXWebRTCCommunication1.DeviceList.Items[I];
    if device.Kind = wdkVideoInput then
      camCB.Items.AddPair(device.Name, device.DeviceID)
    else if device.Kind = wdkAudioInput then
      micCB.Items.AddPair(device.Name, device.DeviceID);
  end;
end;

Requesting device access

Use the RequestUserMedia method to request access to the user's camera and/or microphone.

If you request a device that does not exists (for example: AAudio parameter is True or AAudioID is a non-empty string but the user does not have a microphone), the request will be rejected. Unless you are 100% certain the user has the given devices, it's better to request the available devices and make the device request based on that.

It is also possible for the user to share their screen as a video source in the WebRTC communication. This is done by calling RequestDisplay. RequestDisplay internally maps onto the MediaDevices.getDisplayMedia API.

Limited support

The getDisplayMedia API is NOT supported on mobile platforms.
On native platforms only Windows (WebView2) and macOS (WebKit Safari) is supported.

Calling RequestDisplay will prompt the user to select a tab/window/screen to share with the option to enable the audio if the AAudio parameter is set to True. This prompt is handled by the embedded browser, it's not possible to list the available tabs/windows/screens. Make sure to have a sufficiently sized control to let the users see the prompt fully.

procedure TForm1.Button1Click(Sender: TObject);
begin
  TMSFNCWXWebRTCCommunication1.RequestDisplay(True);
end;

Adding peers

Peers can be added both at design-time and programmatically but the approach differs for each.

Design-time

When adding peers at design-time, make sure to fill the IDs of the peers beforehand. In addition to that, use the HTML and CSS properties to create a layout for the peers. When defining the layout via the HTML property, specify which element belongs to the user and the peers by setting the element ID the following way:

{%self%} - The feed of the user
{%peerX%} - The feed of the peer. X needs to be replaced with the index of the collection element.

For example, the following means the video element of the first collection item:
```
<video id={%peer1%} class="peer-alignment" playsinline autoplay></video>
```

Programmatically

When adding items programmatically it's required to use AddPeerConnection as these connections are added after the browser is intialized with the initial HTML and CSS content. This method will automatically add a new item in the collection with a given ID, and it will also insert the HTML element dynamically related to the item. Compared to design-time setting, when using AddPeerConnection the id is not required to be added.

For example:

procedure TServer1.OnNewMessage(Sender: TObject; AMessage: string);
var
  id: string;
begin
  //Parse message and get id
  id := GetIdFromMessage(AMessage);
  //Add new peer if not present in the collection yet
  if not Assigned(Peers.GetPeerById(id)) then
    TMSFNCWXWebRTCCommunication1.AddPeerConnection('peer_id_from_message', '<video class="peer-alignment" playsinline autoplay></video>');
end;

ICE servers

Interactive Connectivity Establishment (ICE) is a framework to allow your web browser to connect with peers. ICE is using STUN and/or TURN servers to get the public IP address of a peer and/or relay data through the server if the router does not allow direct connection.

Use the DefaultIceServers property to set up the server list that you plan on sharing between multiple peers. Whenever a new peer is added its IceServer property will be identical to the DefaultIceServers.

Data channels

TTMSFNCWXWebRTCCommunication also provides a way to create data channels between peers. These data channels can be used to send some data directly to a peer through the browser connection (such as chat messages, images, etc...).

Creating data channels

Each Peer item in the Peers collection has its own DataChannels collection as a single peer connection can have multiple data channels at a time.

The offer needs to contain the data channels one wants to estabilish between the two peers. To put this simply, make sure your offer sending peer will always have the necessary data channels added to the DataChannels collection so it can create those data channels by sending them as part of the offer.

If a data channel is added later than the connection is established, the connection needs to be renegotiated.

Sending messages

Once the data channels connect, the OnDataChannelOpen event will be triggered. From that moment it's possible to send messages by using the SendData method. This method has two overloads, one expects a string for text based data and the other one expects a stream for binary data.

To send messages, call either the SendData method of the Peer item or the SendData method of the TTMSFNCWXWebRTCCommunication with the ID of the peer:

procedure TForm1.Button1Click(Sender: TObject);
begin
  TMSFNCWXWebRTCCommunication1.SendData('peer id', 'channel name', 'my text');
end;

Receiving messages

When a message arrives to a peer, it will trigger either the OnDataChannelStringMessage or OnDataChannelBinaryMessage message depending on the format the message was sent in.

Example of receiving string messages:

procedure TForm1.TMSFNCWXWebRTCConnection1DataChannelStringMessage(Sender: TObject; APeerID,
  AChannelName, AMessage: string);
begin
  Memo1.Lines.Add('A message arrived from ' + APeerID + ' on the ' + AChannelName + ' channel: ' + AMessage);
end;

Create and accept offers/answers

To initiate a connection from one peer, first an offer needs to be created.

procedure TForm1.Button1Click(Sender: TObject);
begin
  TMSFNCWXWebRTCCommunication1.CreateOffer('peerID');
end;

When the offer is created, the OnOffer event will be triggered where the AOffer string parameter contains the offer in a JSON string format. When this event is triggered, the offer does not contain any candidates yet. After the candidates are gathered, the OnOfferWithCandidates event will be triggered, where similar to OnOffer the AOffer string parameter contains the offer.

procedure TForm1.TMSFNCWXWebRTCCommunication1Offer(Sender: TObject; AID: string;
  AOffer: string);
begin
  SendToServer('{"type": "offer", "peerId": "' + AID + '", "offer": ' + AOffer + '}');
end;

When a peer gets an offer (through the signaling server) that it needs to accept, it can use the AcceptOffer method. The first AID parameter is the PeerID of the peer in the Peers collection. The AOffer string parameter must be the offer in a JSON string format.

Upon accepting an offer, the peer will automatically create an answer and the OnAnswer event will be triggered. Similarly to OnOffer, the AAnswer parameter will contain the answer in JSON string format. The OnAnswerWithCandidates works the same way as OnOfferWithCandidates.

procedure TForm1.TMSFNCWXWebRTCCommunication1Answer(Sender: TObject; AID: string;
  AAnswer: string);
begin
  SendToServer('{"type": "answer", "peerId": "' + AID + '", "answer": ' + AAnswer + '}');
end;

Whenever an answer arrives through the signaling server, it can be accepted by calling AcceptAnswer where the first AID parameters is the PeerID of the peer in the Peers collection. The AAnswer string parameter must be the answer in JSON string format.

Changes in connection and reconnecting

Whenever the ICE connection state changes, the OnIceConnectionStateChange event triggers. For example, this event can be used to detect losing a connection if the parameter is a "failed". This means the connection cannot be restored by itself and it needs to be created again.

To restart a connection, simply call RestartIce. It will request to gather the ICE candiates so the connection on both ends can be redone. It will also cause the OnNegotationNeeded event to be fired from where a new offer can be created to restore the connection - if the application logic requires that.

Properties

Property name	Description
AutoRequestAudioDevices	If this property is enabled, the component will automatically request the audio devices of the user when it's initialized.
AutoRequestVideoDevices	If this property is enabled, the component will automatically request the video devices of the user when it's initialized.
CSS	Allows to add custom CSS to the control that can be used with HTML elements.
DefaultIceServers	With multi-peer connection it's easier to set the ICE servers once via this property. When a new item is added to the `Peer` collection, the ICEServer property will automatically be set to this.
DeviceList	This property will contain the available devices after requesting them.
HTML	Allows to add custom HTML to the control.
LocalVideoSettings	Various setting for the user's video access.
Peers	A collection of peer connections.

Methods

Method name	Description
AcceptAnswer(AID: string; AAnswer: string)	Accepts an answer formatted in a JSON string. The `AID` represents the ID of the peer in the `Peers` collection.
AcceptIceCandidate(AID: string; AIceCandidate: string)	Accepts an ICE candidate formatted in a JSON string. The `AID` represents the ID of the peer in the `Peers` collection.
AcceptOffer(AID: string; AOffer: string)	Accepts an offer formatted in a JSON string. The `AID` represents the ID of the peer in the `Peers` collection.
AddDataChannel(AID: string; AName: string)	Adds a new data channel with the `AName` channel name to the `DataChannels` collection of the `Peer` indicated with `AID`.
AddPeerConnection(AID: string; AHTML: string; ACSS: string = ''): TTMSFNCWXWebRTCPeer	Adds a peer to the `Peers` collection with HTML and optional CSS that is dynamically added to the `body`.
CloseConnection(AID: string)	Closes a connection to a peer. The `AID` represents the ID of the peer in the `Peers` collection.
CreateOffer(AID: string)	Creates an offer for a peer. The `AID` represents the ID of the peer in the `Peers` collection.
RemovePeerConnection(AID: string)	Removes a peer from the collection. The automatically added HTML is also removed.
RequestDevices(AVideo: Boolean; AAudio: Boolean)	Request a list of the available devices from the user.
RequestUserMedia(ACamera: Boolean; AAudio: Boolean)	Requests access to the user's video and audio devices.
RequestUserMedia(ACameraID: string; AAudioID: string)	Requests access to the user's video and audio devices by ID.
RestartIce(AID: string)	Call to restart the ICE candidate generation. The `AID` represents the ID of the peer in the `Peers` collection.
SendData(AID: string; AChannel: string; AData: string)	Sends string data through an open data channel. `AID` is the ID of the peer and `AChannel` is the channel name where `AData` should be sent.
SendData(AID: string; AChannel: string; AData: TStream)	Sends string data through an open data channel.

Events

Event name	Description
OnAnswer	Event triggered when an answer was generated.
OnAnswerWithCandidates	Event triggered when the candidates are gathered after generating an answer.
OnDataChannelBinaryMessage	Event triggers when a binary message arrives through a data channel.
OnDataChannelOpen	Event triggers when a data channel opens.
OnDataChannelStringMessage	Event triggers when a string message arrives through a data channel.
OnDeviceListInitialized	Event triggered when the requested device list is initialized.
OnIceCandidate	Event triggered when an ICE candidate was generated.
OnIceConnectionStateChange	Event triggered when the ICE connection state changes.
OnNegotiationNeeded	Event triggered when the negotiation of the connection through the signaling server is required.
OnOffer	Event triggered when an offer was generated.
OnOfferWithCandidates	Event triggered when the candidates are gathered after generating an offer.
OnUserMediaError	Event triggered when an error occurs while requesting a media device.
OnUserMediaStart	Event triggered when the video or audio of the user starts playing.