...

Operation flowchart

1. The browser connects to the server via the Websocket protocol and sends the publish command.

...

rtp_activity_video=false

If Chrome browser sends empty video due to web camera conflict

Some Chrome versions does not return an error if web camera is busy, but publish a stream with empty video (black screen). In this case, stream publishing can be stopped by two ways: using JavaScript and HTML5 on client, or using server settings.

Stopping a stream with empty video on client side

Videotrack that Chrome browsers creates for busy web camera, stops after no more than one second publishing, then stream is send without a videotrack. In this case videotrack state (readyState variable) changes to ended, and corresponding  onended event is generated that can be catched by web application. To use this event:

1. Add to web application script the registartion function for onended event handler, in which stream pub;ishing is stopped with stream.stop()

...

...

RTP activity can be checked for publishing streams only, not for playing streams.

Disable tracks activity checking by stream name

Since build 5.2.1784 it is possible to disable video and audio tracks activity checking for the streams with names matching a regular expression

rtp_activity_audio_exclude=stream1
rtp_activity_video_exclude=stream1

The feature may be useful for streams in which a media traffic can stop for a long time, for example, screen sharing streams from an application window

rtp_activity_audio_exclude=.*-screen$
rtp_activity_video_exclude=.*-screen$

In this case tracks activity checking will not be applied to the tracks named like conference-123-user-456-screen 

If Chrome browser sends empty video due to web camera conflict

Some Chrome versions does not return an error if web camera is busy, but publish a stream with empty video (black screen). In this case, stream publishing can be stopped by two ways: using JavaScript and HTML5 on client, or using server settings.

Stopping a stream with empty video on client side

Videotrack that Chrome browsers creates for busy web camera, stops after no more than one second publishing, then stream is send without a videotrack. In this case videotrack state (readyState variable) changes to ended, and corresponding  onended event is generated that can be catched by web application. To use this event:

1. Add to web application script the registartion function for onended event handler, in which stream pub;ishing is stopped with stream.stop()

function addVideoTrackEndedListener(localVideo, stream) {
    var videoTrack = extractVideoTrack(localVideo);
    if (videoTrack && videoTrack.readyState == 'ended') {
        console.error("Video source error. Disconnect...");
        stream.stop();
    } else if (videoTrack) {
        videoTrack.onended = function (event) {
            console.error("Video source error. Disconnect...");
            stream.stop();
        };
    }
}

...

publishStream = session.createStream({
    ...
    disableConstraintsNormalization: true,
    constraints = session.createStream({
    ...
    disableConstraintsNormalization: true,
    constraints {
        video: {
            width: {ideal: 1024},
            height: {ideal: 768}
        },
        audio: true
    }
}).on(STREAM_STATUS.PUBLISHING, function (publishStream) {
    ...
});
publishStream.publish();

H264 encoding profiles exclusion

Since build 5.2.620 some H264 encoding profiles can be excluded from remote SDP which is sent by server to browser. The profiles to exckude should be listed in the following parameter

webrtc_sdp_h264_exclude_profiles=4d,64

In this case, Main (4d) and High (64) profiles will be excluded, but Baseline (42) still remains:

a=rtpmap:102 H264/90000
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:125 H264/90000
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:127 H264/90000
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:108 H264/90000
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f

This setting may be useful if some browser encodes B-frames for example using high profiles while hardware acceleration is enabled.

By default, no profiles will be excluded from SDP if they are supported by browser

webrtc_sdp_h264_exclude_profiles=

Content type management while publishing from Chromium based browser

In some cases, most browsers based on Chromium 91 agressively assess a publishing channel quality and drop publishing resolution lower than set in constraints, even if channel is enough to publish 720p or 1080p stream. To workaround this behaviour, since WebSDK build  2.0.180 videoContentHint option was added:

    session.createStream({
        name: streamName,
        display: localVideo,
        cacheLocalResources: true,
        receiveVideo: false,
        receiveAudio: false,
        videoContentHint: "detail"
        ...
    }).publish();

In WebSDK builds before 2.0.242 this option is set to detail by default and forces browsers to keep the publishing resolution as set in constraints. However, browser can drop FPS in this case when publishing stream from som USB web cameras. If FPS should be kept mo matter to resolution, the option should be set to motion

    session.createStream({
        name: streamName,
        display: localVideo,
        cacheLocalResources: true,
        receiveVideo: false,
        receiveAudio: false,
        videoContentHint: "motion"
        ...
    }).publish();

Since WebSDK build 2.0.242, videoContentHint  is set to motion by default. The detail or text values should be set only for screen sharing streaming in browser.

Since WebSDK build 2.0.204 videoContentHint selection is available in Media Device example

Image Added

FPS management in Firefox browser

By default, Firefox is publishing video with maximum FPS shown by web camera driver fo requested resolution. This value is 30 FPS for most of modern web cameras. Publishing FPS can be defined more strictly if necessary. To do this, disable constraints normalization:

session.createStream({
    ...
    disableConstraintsNormalization: true,
    constraints: {
        video: {
            width: 640,
            height: 360,
            frameRate: { max: 15 }
        },
        audio: true
    }
}).on(STREAM_STATUS.PUBLISHING, function (publishStream) {
    ...
}).publish();

Note that Foirefox can exclude the camera from the list while requesting camera and microphone access if camera driver does not provide a required combination of resolution and FPS. Also, Firefox can change a publishing resolution if there is only one resolution with required FPS in camera driver response.

Stereo audio publishing in browser

Audio bitrate should be more than 60000 bps to publish stereo in Opus codec from browser. This can be done by setting Opus codec parameters on client side

session.createStream({
    name: streamName,
    display: remoteVideo,
    constraints: {
        audio: {
            bitrate: 64000
        },
        ...
    }
    ...
}).publish();

or on server side

opus_formats = maxaveragebitrate=64000;stereo=1;sprop-stereo=1;

In this case, Firefox browser publishes stereo audio without additional setup.

Stereo audio publishing in Chrome based browsers

A certain client setup is required to publish stereo audio from Chrome. Thre is two ways to set this up depending on client implementation

Using Web SDK

If Web SDK is used in project, it is necessary to set the fiollowing constraint option:

session.createStream({
    name: streamName,
    display: localVideo,
    constraints: {
        audio: {
            stereo: true
        },
        ...
    }
    ...
}).publish();

Using Websocket API

If only Websocket API is used in project, it is necessary to disable echo cancellation

    var constraints = {
        audio: {
            echoCancellation: false,
            googEchoCancellation: false
        },
        ...
    };
    ...
    navigator.getUserMedia(constraints, function (stream) {
        ...
    }, reject);

If echo cancellation is enabled, Chrome will publish mono audio even if stereo is set in Opus codec options.

How to bypass an encrypted UDP traffic blocking

Sometimes an encrypted UDP mediatraffic may be blocked by ISP. In this case, WebRTC stream publishing over UDP will fail with Failed by RTP activity error. To bypass this, it is recommended to use TCP transport at client side

session.createStream({
    name: streamName,
    display: localVideo,
    transport: "TCP"
    ...
}).publish();

Another option is to use external or internal TURN server or publish a stream via RTMP or RTSP.

Redundancy support while publishing audio

Since build 5.2.1969 a redundancy is supported while publishing audio data (RED, RFC2198). This allows to reduce audio packet loss when using opus codec. The feature may be enabled with the following parameter

codecs=red,opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv

Note that red codec should be set before opus codec. In this case a browser supporting RED will send a redundancy data in audio packets. Note that audio publishing bitrate will be raised.

RED should be excluded for the cases when it cannot be applied:

codecs_exclude_sip=red,mpeg4-generic,flv,mpv
codecs_exclude_sip_rtmp=red,opus,g729,g722,mpeg4-generic,vp8,mpv

Getting an additional information about publisher configuration

Since WebSDK 2.0.246 build shipped with WCS build 5.2.2015 and later an additional information about publisher configuration will be sent to server. The data may help to debug and solve a stream publishing issues.

The data are sent in REST hooks.

System and browser information

System and browser information is sent in /connect REST hook

{
  "nodeId" : "VuQnlozpitGdVKzoIZg3f2JmJdMldzPV@192.168.0.40",
  "appKey" : "defaultApp",
  "sessionId" : "/192.168.0.231:11641/192.168.0.40:8443-b7efee23-45ac-4591-9aec-37f896ef10f1",
  ...,
  "clientInfo" : {
    "architecture" : "x86",
    "bitness" : "64",
    "brands" : [ {
      "brand" : "Not/A)Brand",
      "version" : "8"
    }, {
      "brand" : "Chromium",
      "version" : "126"
    }, {
      "brand" : "Google Chrome",
      "version" : "126"
    } ],
    "fullVersionList" : [ {
      "brand" : "Not/A)Brand",
      "version" : "8.0.0.0"
    }, {
      "brand" : "Chromium",
      "version" : "126.0.6478.127"
    }, {
      "brand" : "Google Chrome",
      "version" : "126.0.6478.127"
    } ],
    "mobile" : false,
    "model" : "",
    "platform" : "Windows",
    "platformVersion" : "10.0.0"
  }
}

System and browser information is available at server side in response to /connection/find  and /connection/find_all REST queries.

Publisher media devices information

Publisher media devices information is sent in /publishStream REST hook

{
  "nodeId" : "d2hxbqNPE04vGeZ51NPhDuId6k3hUrBB@192.168.0.39",
  "appKey" : "defaultApp",
  "sessionId" : "/192.168.0.83:42172/192.168.0.39:8443-325c8258-81a6-43de-8c74-1bf582fb436a",
  "mediaSessionId" : "bb237f90-39be-11ef-81b8-bda3bf19742b",
  "name" : "test",
  "published" : true,
  ...,
  "localMediaInfo" : {
    "devices" : {
      "video" : [ {
        "id" : "bc18c5c2510d338d7b2c26bce4e37967feea3172f54ba2077558775d51839419",
        "label" : "HD Webcam C615 (046d:082c)",
        "type" : "camera"
      }, {
        "id" : "1f78289ccdbf27d67d605a35d6288acbdefe257275d4b5403525fb5cb1e1822e",
        "label" : "HP HD Camera (0408:5347)",
        "type" : "camera"
      } ],
      "audio" : [ {
        "id" : "de988681c02901db0bfe012cd393eb2db5245fc2fb34709a26686ae6ca85b3ba",
        "label" : "HD Webcam C615 Mono",
        "type" : "mic"
      }, {
        "id" : "default",
        "label" : "Default",
        "type" : "mic"
      }, {
        video"id" : {"e7f4beddb0d71ea1f618cf3bab0f7e94053b622ddaf312c403824caa006f5889",
        "label" : "Quantum  width: {ideal: 1024}600 Mono",
            height: {ideal: 768}"type" : "mic"
        }, {
        "id" audio: true"fc6ac664dec546102d3f83b7fb5981afe3f0e88f8b76ffbe660ecfdd989bcf96",
    }
}).on(STREAM_STATUS.PUBLISHING, function (publishStream) {
  "label"  ...
});
publishStream.publish();

H264 encoding profiles exclusion

Since build 5.2.620 some H264 encoding profiles can be excluded from remote SDP which is sent by server to browser. The profiles to exckude should be listed in the following parameter

webrtc_sdp_h264_exclude_profiles=4d,64

In this case, Main (4d) and High (64) profiles will be excluded, but Baseline (42) still remains:

a=rtpmap:102 H264/90000
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:125 H264/90000
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:127 H264/90000
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:108 H264/90000
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f

This setting may be useful if some browser encodes B-frames for example using high profiles while hardware acceleration is enabled.

By default, no profiles will be excluded from SDP if they are supported by browser

webrtc_sdp_h264_exclude_profiles=: "Family 17h (Models 10h-1fh) HD Audio Controller Analog Stereo",
        "type" : "mic"
      } ]
    },
    "tracks" : {
      "video" : [ {
        "trackId" : "1922f1d0-3a3f-4fa5-bd6e-e91ac84c666c",
        "device" : "HD Webcam C615 (046d:082c)"
      } ],
      "audio" : [ {
        "trackId" : "44187ea1-b756-4feb-80f0-ac57934c2200",
        "device" : "MediaStreamAudioDestinationNode"
      } ]
    }
  }
}

Where:

• devices - a media devices list available to the browser
• tracks - what devices audio and video tracks of the stream published are capturing from

Publisher media devices information is available at server side in response to /stream/find  and /stream/find_all REST queries.

Known issues

1. If the web app is inside an iframe element, publishing of the video stream may fail.

...

publishStream = session.createStream({
    ...
    constraints: {
        video: {
            width: 640,
            height: 360
        },
        audio: true
    }
}).on(STREAM_STATUS.PUBLISHING, function (publishStream)      },
        audio: true
    }
}).on(STREAM_STATUS.PUBLISHING, function (publishStream) {
    ...
});
publishStream.publish();

b) Update Web SDK to build 0.5.28.2753.153, where default resolution for Safari browser is adopted to 16:9

c) Update MacOS to build  11.3.1, Safari to build 14.1 (16611.1.21.161.6)

14. Streams published on Origin server in CDN may not be played from Edge if some H264 encoding profiles are excluded

Symptoms: WebRTC H264 stream published on Origin is playing as audio only from Edge with VP* codec shown in stream metrics

Solution: if some H264 encoding profiles are excluded on Origin

webrtc_sdp_h264_exclude_profiles=4d,64

the allowed H264 profiles should be explicitly set on Egde with the following parameter

profiles=42e01f

15. MacOS Safari 14.0.* stops sending video packets after video in muted and then unmuted due to Webkit bug

Symptoms: when muteVideo() then unmuteVideo() are applied, the publishing stops after a minute with Failed by Video RTP activity error

...

{
    ...
});
publishStream.publish();

b) Update Web SDK to build 0.5.28.2753.153, where default resolution for Safari browser is adopted to 16:9

c) Update MacOS to build  11.3.1, Safari to build 14.1 (16611.1.21.161.6)

14. Streams published on Origin server in CDN may not be played from Edge if some H264 encoding profiles are excluded

Symptoms: WebRTC H264 stream published on Origin is playing as audio only from Edge with VP* codec shown in stream metrics

Solution: if some H264 encoding profiles are excluded on Origin

webrtc_sdp_h264_exclude_profiles=4d,64

the allowed H264 profiles should be explicitly set on Egde with the following parameter

profiles=42e01f

15. MacOS Safari 14.0.* stops sending video packets after video in muted and then unmuted due to Webkit bug

Symptoms: when muteVideo() then unmuteVideo() are applied, the publishing stops after a minute with Failed by Video RTP activity error

Solution: update MacOS to build 11.3.1, Safari to build 14.1 (16611.1.21.161.6), in this build the Webkit bug seems to be fixed, the problem is not reproducing

16. When publishing from Google Pixel 3/3XL, the picture is strongly distorted in some resolutions

Symptoms: a local video is displaying normally, but is playing with a strong distortion (transverse stripes)

Solution: avoid the following resolutions while publishing stream from Google Pixel 3/3XL:

• 160x120
• 1920x1080

17. iOS Safari 15.1 requires from another side to enable image orientation extension when publishing H264 stream

Symptoms: webpage crashes in iOS Safari 15.1 when stream publishing is started (Webkit bugs https://bugs.webkit.org/show_bug.cgi?id=232381 and https://bugs.webkit.org/show_bug.cgi?id=231505)

Solution:

a) enable image orientation extension support on client side in iOS Safari

session.createStream({
    name: streamName,
    ...
    cvoExtension: true
}).publish();

and in WCS builds before 5.2.1074 disable RTP bundle support

rtp_bundle=false

Since WCS build 5.2.1074RTP bundle may not be disabled

b) use VP8 to publish a stream

session.createStream({
    name: streamName,
    ...
    stripCodecs: "H264"
}).publish();