...
then speech indicator frame will not be displayed until video data arrive.
Frame colour
...
Avatar pictures displaying for audio only streams
Warning |
---|
...
Code Block | ||
---|---|---|
| ||
mixer_voice_activity_colour=#FF0000 |
A colour can be set as hexadecimal value with # or 0x prefix, in #RRGGBB form. Using the settings above, speech indicator frame will be red.
Frame displaying when silence is in a stream
By default, speech indicator frame may blink at normal participant speech rate, which can be incomfortable for viewing. For this reason, since build 5.2.775 it is possible to set a time interval in which the frame will be displayed since silence is detected in the stream. The feature can be enabled using the following parameter
Code Block | ||
---|---|---|
| ||
mixer_voice_activity_switch_delay=500 |
In this case, speech indicator frame will still be displayed in 500 milliseconds since silence is detected in the stream.
By default, this interval is set to 0, frame will be hidden without delay.
Mixer background management
By default, mixer applies black background. Since build 5.2.645 it is possible to set PNG picture file which will be used as mixer background. For example, to change the background to blue, prepare blue.png picture and set the following parameter
Code Block | ||
---|---|---|
| ||
mixer_video_background_filename=blue.png |
Background picture file should be placed to /usr/local/FlashphonerWebCallServer/conf if full path is not set. The file can be placed to any folder on the server, in this case full path to the file should be set to the parameter
Code Block | ||
---|---|---|
| ||
mixer_video_background_filename=/opt/media/blue.png |
Background picture will be automatically scaled to mixer output stream resolution
Audio only streams picture aspect ratio management
By default, audio only streams picture aspect ratio is set according to mixer one (16:9)
Code Block | ||
---|---|---|
| ||
mixer_audio_only_width=640
mixer_audio_only_height=360
mixer_video_width=1280
mixer_video_height=720 |
It can be changed together with mixer if necessary, for example to 4:3
Code Block | ||
---|---|---|
| ||
mixer_audio_only_width=640
mixer_audio_only_height=480
mixer_video_width=1280
mixer_video_height=960 |
Note that audio only streams picture aspect ratio settings are applied only with WCS restart.
Multithreading support and hign load optimizations
...
1725 the feature is supported in Ubuntu 20.04 and other systems with glibc 2.31 and newer only Since build 5.2.1727 the feature is supported in Centos 7.6 (glibc 2.17) and above |
Since build 5.2.1710 it is possible to set an avatar picture in PNG (with transparency), JPG or BMP formats to an audio only stream in mixer. A picture may be set while adding a stream to mixer by REST API query /mixer/add
:
Code Block | ||||
---|---|---|---|---|
| ||||
{
"uri": "mixer://mixer1",
"remoteStreamName": "user2",
"hasVideo": false,
"hasAudio": true,
"avatar": "https://mystorage.com/storage/avatar.png"
} |
or by REST API query /mixer/set_stream_avatar
for the audio only stream which already has added to the mixer:
Code Block | ||||
---|---|---|---|---|
| ||||
{
"uri": "mixer://mixer1",
"remoteStreamName": "user2",
"avatar": "https://mystorage.com/storage/avatar.png"
} |
The following ways to set a picture are supported:
- HTTP URL:
https://mystorage.com/storage/avatar.png
- file URI:
file:///opt/avatar.png
- local file:
/opt/avatar.png
Audio only stream in mixer is displaying as follows
If stream names displaying in mixer is enabled, audio only stream name will be displayed like video stream one.
Avatar picture may be removed by REST API query /mixer/remove_stream_avatar
:
Code Block | ||||
---|---|---|---|---|
| ||||
{
"uri": "mixer://mixer1",
"remoteStreamName": "user2"
} |
Since WCS build 5.2.1858 it is possible to set avatar pictures automatically by stream name, not using REST API. In this case avatar picture files should be placed to the folder
Code Block | ||
---|---|---|
| ||
avatar_dir=/usr/local/FlashphonerWebCallServer/avatar |
A file name should be equal to stream name. For example, the picture file stream1.png
will be applied to the stream named stream1
. If the stream was added to the mixer automatically, e.g. stream1#mixer1
, the mixer name will not be used to get the picture, only the stream name before #
character.
Warning |
---|
Avatar picture setting with REST API query |
Frame colour
Since build 5.2.741 speech indicator frame colour can be changed with the following parameter
Code Block | ||
---|---|---|
| ||
mixer_type=MULTIvoice_THREADED_NATIVE mixer_mcu_multithreaded_mix=true |
CPU threads count to mix audio amd video can be adjusted by the following parameters
Code Block | ||
---|---|---|
| ||
mixer_audio_threads=10
mixer_video_threads=4 |
...
activity_colour=#FF0000 |
A colour can be set as hexadecimal value with # or 0x prefix, in #RRGGBB form. Using the settings above, speech indicator frame will be red.
Frame displaying when silence is in a stream
By default, speech indicator frame may blink at normal participant speech rate, which can be incomfortable for viewing. For this reason, since build 5.2.775 it is possible to set a time interval in which the frame will be displayed since silence is detected in the stream. The feature can be enabled using the following parameter
Code Block | ||
---|---|---|
| ||
mixer_mcuvoice_activity_multithreadedswitch_deliverydelay=true |
If participants count is lower than mixing threads count (for example, 3 participants), only one CPU thread will be used for mixing.
Real-time mixer tuning
Real-time mixer can be tuned using the following parameters
...
Parameter
...
Default value
...
Description
...
mixer_in_buffering_ms
...
200
...
Incoming stream video buffer in milliseconds
...
Testing
1. For test we use
- demo server demo.flashphoner.com;
- Chrome browser;
- MCU Client web application example for conferencing.
2. Open MCU Client web application page. Enter user name user1 and room name room1
2. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed without your microphone audio
3. Open MCU Client application page in another brower tab/browser/PC. Enter user name user2 and room name room1
4. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed with videos from both users but audio from user1 only
5. On user1 page two videos also are playing with user2 audio only
6. Click Leave to leave the room in both tabs/browsers
Call flow
Call flow for conferencing example based on real-time mixer with MCU function is described on MCU Client page.
Incoming streams tuning recommendations
When delay occurs in the incoming stream from one of participants, realtime mixer will freeze that stream. The following is recommended to minimize incoming stream delays:
1. For RTMP streams, adjust encoding parameter so that:
- client encoder perfomance to be enough to send picture frames in time
- stream resolution and bitrate to fit to publishers channel from client to server
2. For WebRTC streams, do not raise minimum video bitrate threshold higher than webrtc_cc_min_bitrate
server configuration parameter defines. By default, lower video bitrate threshold is set to 30 kbps
Code Block | ||
---|---|---|
| ||
webrtc_cc_min_bitrate=30000 |
The publisher client browser will adopt the stream to channel quality drops. The lower bitrate the lower picture quality, but the participant stream will not be freezed in this case.
Adding one stream to two or more realtime mixers simultaneously
Since build 5.2.732 one stream can be added to two ore more realtime mixers simultaneously. Note that realtime mixer should be enabled
Code Block | ||
---|---|---|
| ||
mixer_realtime=true |
and custom losless videoprocessor should be disabled
Code Block | ||
---|---|---|
| ||
mixer_lossless_video_processor_enabled=false |
Known issues
1. Real-time mixer functions are disabled when custom losless videoprocessor is used, mixer incoming streams cannot be played
Symptoms: stream stops playing after adding to mixer
Solution: do not use custom losless videoprocessor with real-time mixer
Code Block | ||
---|---|---|
| ||
mixer_lossless_video_processor_enabled=false |
2. To display stream captions in mixer, it would be necessary to install fontconfig library
Symptoms: streams cannot be added to mixer with the following exception in server log
Code Block | ||
---|---|---|
| ||
09:17:11,756 ERROR MixerAgent - MIXER-AGENT-mixer://mixervmixdr52-9d46cd04-5867-4d74-a9d9-baf67f74e7d2 Mixer closed due to error java.lang.InternalError: java.lang.reflect.InvocationTargetException at java.desktop/sun.font.FontManagerFactory$1.run(FontManagerFactory.java:86) at java.base/java.security.AccessController.doPrivileged(AccessController.java:310) at java.desktop/sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74) at java.desktop/sun.font.SunFontManager.getInstance(SunFontManager.java:247) at java.desktop/sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:265) at java.desktop/java.awt.Font.getStringBounds(Font.java:2606) at java.desktop/java.awt.Font.getStringBounds(Font.java:2516) at com.flashphoner.media.N.A.A.A(Unknown Source) at com.flashphoner.media.mixer.video.presentation.Canvas.computeTextScales(Unknown Source) at com.flashphoner.media.mixer.video.presentation.Canvas.writeNative(Unknown Source) at com.flashphoner.media.N.A.A(Unknown Source) at com.flashphoner.media.N.D.D(Unknown Source500 |
In this case, speech indicator frame will still be displayed in 500 milliseconds since silence is detected in the stream.
By default, this interval is set to 0, frame will be hidden without delay.
Mixer background management
By default, mixer applies black background. Since build 5.2.645 it is possible to set PNG picture file which will be used as mixer background. For example, to change the background to blue, prepare blue.png picture and set the following parameter
Code Block | ||
---|---|---|
| ||
mixer_video_background_filename=blue.png |
Background picture file should be placed to /usr/local/FlashphonerWebCallServer/conf if full path is not set. The file can be placed to any folder on the server, in this case full path to the file should be set to the parameter
Code Block | ||
---|---|---|
| ||
mixer_video_background_filename=/opt/media/blue.png |
Background picture will be automatically scaled to mixer output stream resolution
Audio only streams picture aspect ratio management
By default, audio only streams picture aspect ratio is set according to mixer one (16:9)
Code Block | ||
---|---|---|
| ||
mixer_audio_only_width=640
mixer_audio_only_height=360
mixer_video_width=1280
mixer_video_height=720 |
It can be changed together with mixer if necessary, for example to 4:3
Code Block | ||
---|---|---|
| ||
mixer_audio_only_width=640
mixer_audio_only_height=480
mixer_video_width=1280
mixer_video_height=960 |
Note that audio only streams picture aspect ratio settings are applied only with WCS restart.
Multithreading support and hign load optimizations
Since build 5.2.793 multithreading support is added to optimize a mixer for a big participants amount. This feature can be enabled by the following parameters
Code Block | ||
---|---|---|
| ||
mixer_type=MULTI_THREADED_NATIVE
mixer_mcu_multithreaded_mix=true |
CPU threads count to mix audio amd video can be adjusted by the following parameters
Code Block | ||
---|---|---|
| ||
mixer_audio_threads=10
mixer_video_threads=4 |
Usually, MCU mixer encodes one video stream and a many ausio streams: two per every participant plus one common audio track. Therefore, it is recommended to set more CPU threads for audio encoding than for video (as shown above). Additionally, if freezes occur in MCU mixer output stream, it is recommended to enable multithreaded delivery of mixing result to the main WCS engine, to send it to subscribers.
Code Block | ||
---|---|---|
| ||
mixer_mcu_multithreaded_delivery=true |
If participants count is lower than mixing threads count (for example, 3 participants), only one CPU thread will be used for mixing.
Real-time mixer tuning
Real-time mixer can be tuned using the following parameters
Parameter | Default value | Description |
---|---|---|
mixer_audio_silence_threshold | -50.00 | Incoming stream audio silence level in Db |
mixer_debug_mode | false | Adds some debug info to stream picture caption |
mixer_in_buffering_ms | 200 | Incoming stream video buffer in milliseconds |
mixer_incoming_time_rate_lower_threshold | 0.95 | Relative incoming stream time to mixer time rate lower threshold |
mixer_incoming_time_rate_upper_threshold | 1.05 | Relative incoming stream time to mixer time rate upper threshold |
mixer_video_stable_fps_threshold | 15 | Incoming stream FPS threshold, video buffer will be disabled for streams with low fps |
audio_mixer_max_delay | 300 | A maximum time interval to wait for any audio data in the incoming stream, in milliseconds, when this time is expired, the stream will not be mixed until some media data receiving |
Testing
1. For test we use
- demo server demo.flashphoner.com;
- Chrome browser;
- MCU Client web application example for conferencing.
2. Open MCU Client web application page. Enter user name user1 and room name room1
2. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed without your microphone audio
3. Open MCU Client application page in another brower tab/browser/PC. Enter user name user2 and room name room1
4. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed with videos from both users but audio from user1 only
5. On user1 page two videos also are playing with user2 audio only
6. Click Leave to leave the room in both tabs/browsers
Call flow
Call flow for conferencing example based on real-time mixer with MCU function is described on MCU Client page.
Incoming streams tuning recommendations
When delay occurs in the incoming stream from one of participants, realtime mixer will freeze that stream. The following is recommended to minimize incoming stream delays:
1. For RTMP streams, adjust encoding parameter so that:
- client encoder perfomance to be enough to send picture frames in time
- stream resolution and bitrate to fit to publishers channel from client to server
2. For WebRTC streams, do not raise minimum video bitrate threshold higher than webrtc_cc_min_bitrate
server configuration parameter defines. By default, lower video bitrate threshold is set to 30 kbps
Code Block | ||
---|---|---|
| ||
webrtc_cc_min_bitrate=30000 |
The publisher client browser will adopt the stream to channel quality drops. The lower bitrate the lower picture quality, but the participant stream will not be freezed in this case.
Adding one stream to two or more realtime mixers simultaneously
Since build 5.2.732 one stream can be added to two ore more realtime mixers simultaneously. Note that realtime mixer should be enabled
Code Block | ||
---|---|---|
| ||
mixer_realtime=true |
and custom losless videoprocessor should be disabled
Code Block | ||
---|---|---|
| ||
mixer_lossless_video_processor_enabled=false |
Known issues
1. Real-time mixer functions are disabled when custom losless videoprocessor is used, mixer incoming streams cannot be played
Symptoms: stream stops playing after adding to mixer
Solution: do not use custom losless videoprocessor with real-time mixer
Code Block | ||
---|---|---|
| ||
mixer_lossless_video_processor_enabled=false |
2. To display stream captions in mixer, it would be necessary to install fontconfig library
Symptoms: streams cannot be added to mixer with the following exception in server log
Code Block | ||
---|---|---|
| ||
09:17:11,756 ERROR MixerAgent - MIXER-AGENT-mixer://mixervmixdr52-9d46cd04-5867-4d74-a9d9-baf67f74e7d2 Mixer closed due to error java.lang.InternalError: java.lang.reflect.InvocationTargetException at java.desktop/sun.font.FontManagerFactory$1.run(FontManagerFactory.java:86) at java.base/java.security.AccessController.doPrivileged(AccessController.java:310) at java.desktop/sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74) at java.desktop/sun.font.SunFontManager.getInstance(SunFontManager.java:247) at java.desktop/sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:265) at java.desktop/java.awt.Font.getStringBounds(Font.java:2606) at java.desktop/java.awt.Font.getStringBounds(Font.java:2516) at com.flashphoner.media.N.DA.A.A(Unknown Source) at com.flashphoner.servermedia.mixer.remotevideo.Bpresentation.BCanvas.Г™computeTextScales(Unknown Source) at com.flashphoner.servermedia.mixer.remotevideo.Bpresentation.BCanvas.runwriteNative(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException at com.flashphoner.media.N.A.A(Unknown Source) at javacom.base/jdkflashphoner.internalmedia.reflectN.NativeConstructorAccessorImplD.newInstance0D(NativeUnknown MethodSource) at javacom.base/jdkflashphoner.internalmedia.reflectN.NativeConstructorAccessorImplD.newInstance(NativeConstructorAccessorImpl.java:62A(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45com.flashphoner.server.remote.B.B.Г™(Unknown Source) at java.base/com.flashphoner.server.remote.B.B.run(Unknown Source) Caused by: java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)InvocationTargetException at java.base/javajdk.langinternal.reflect.ConstructorNativeConstructorAccessorImpl.newInstance(Constructor.java:481newInstance0(Native Method) at java.desktopbase/sunjdk.internal.fontreflect.FontManagerFactory$1NativeConstructorAccessorImpl.runnewInstance(FontManagerFactoryNativeConstructorAccessorImpl.java:84) ... 14 more Caused by: java.lang.NullPointerException 62) at java.desktopbase/sunjdk.internal.awtreflect.FontConfigurationDelegatingConstructorAccessorImpl.getVersionnewInstance(FontConfigurationDelegatingConstructorAccessorImpl.java:126245) at java.desktopbase/sunjava.lang.awtreflect.FontConfigurationConstructor.readFontConfigFilenewInstanceWithCaller(FontConfigurationConstructor.java:225500) at java.desktopbase/sunjava.awtlang.FontConfigurationreflect.initConstructor.newInstance(FontConfigurationConstructor.java:107481) at java.desktop/sun.awtfont.X11FontManagerFontManagerFactory$1.createFontConfigurationrun(X11FontManagerFontManagerFactory.java:719) 84) ... 14 more Caused by: java.lang.NullPointerException at java.desktop/sun.fontawt.SunFontManager$2FontConfiguration.rungetVersion(SunFontManagerFontConfiguration.java:3671262) at java.basedesktop/javasun.securityawt.AccessControllerFontConfiguration.doPrivilegedreadFontConfigFile(AccessControllerFontConfiguration.java:310225) at java.desktop/sun.font.SunFontManager.<init>(SunFontManager.java:312) at java.desktop/sun.awt.FcFontManager.<init>(FcFontManager.java:35) at java.desktop/sun.awt.X11FontManager.<init>(X11FontManager.java:56) ... 20 more |
Solution: install fontconfig library
Code Block | ||||
---|---|---|---|---|
| ||||
sudo yum install -y fontconfig |
3. A participant name can be drawn over speech indicator frame if there are many participants in mixer
Symptoms: for small participant pictures (in desktop layout, for example) participants name can be drawn over speech indicator frame
...
sun.awt.FontConfiguration.init(FontConfiguration.java:107)
at java.desktop/sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:719)
at java.desktop/sun.font.SunFontManager$2.run(SunFontManager.java:367)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:310)
at java.desktop/sun.font.SunFontManager.<init>(SunFontManager.java:312)
at java.desktop/sun.awt.FcFontManager.<init>(FcFontManager.java:35)
at java.desktop/sun.awt.X11FontManager.<init>(X11FontManager.java:56)
... 20 more |
Solution: install fontconfig library
Code Block | ||||
---|---|---|---|---|
| ||||
sudo yum install -y fontconfig |
3. A participant name can be drawn over speech indicator frame if there are many participants in mixer
Symptoms: for small participant pictures (in desktop layout, for example) participants name can be drawn over speech indicator frame
Solution: enable inner speech indicator frame position and decrease frame thickness
Code Block | ||
---|---|---|
| ||
mixer_voice_activity_frame_position_inner=true
mixer_voice_activity_frame_thickness=2 |
4. B-frames should be excluded from mixer input stream, or mixer input buffer should be increased
Symptoms: when mixer input stream is encoded by Main profile with B-frames, the mixer output stream is plaing unsmoothly, there are short freezes with sound pauses
Solution:
a) encode stream without B-frames (preferrable)
b) increase mixer input buffer (may give an additional delay)
Code Block | ||
---|---|---|
| ||
mixer_in_buffering_ms=600 |
5. 60 FPS stream publishing to mixer gives additional CPU and system memory load
Symptoms: CPU load increasing and system memory (not Java heap) consumption growing while publishing 60 FPS stream to a mixer
Solution: publish 30 FPS streams to mixer or use more powerful server
6. Captions may twitch for many participants in MCU mixer
Symptoms: for many (over 10) participants in MCU mixer captions may periodically twitch
Solution: apply the following settings
Code Block | ||
---|---|---|
| ||
mixer_voicetext_activity_frame_position_inner=truebulk_write=false mixer_voice_activity_frame_thickness=2 |
4. B-frames should be excluded from mixer input stream, or mixer input buffer should be increased
Symptoms: when mixer input stream is encoded by Main profile with B-frames, the mixer output stream is plaing unsmoothly, there are short freezes with sound pauses
Solution:
a) encode stream without B-frames (preferrable)
b) increase mixer input buffer (may give an additional delay)
Code Block | ||
---|---|---|
| ||
mixer_in_buffering_ms=600 |
5. 60 FPS stream publishing to mixer gives additional CPU and system memory load
Symptoms: CPU load increasing and system memory (not Java heap) consumption growing while publishing 60 FPS stream to a mixer
Solution: publish 30 FPS streams to mixer or use more powerful server
6. Captions may twitch for many participants in MCU mixer
Symptoms: for many (over 10) participants in MCU mixer captions may periodically twitch
Solution: apply the following settings
Code Block | ||
---|---|---|
| ||
mixer_text_bulk_write=false
mixer_text_bulk_write_with_buffer=false |
7. When a number of participants are speaking simultaneously, some participants are less audible then others. The problem is typical for mixing WebRTC streams, and may ocuur in any solutions using WebRTC and audio mixing, in Discord for example: more data are encoded to one stream, more audio samples may be dropped due to fixed samplerate.
Symptoms: when three or more participants are speaking simultaneously, one of them is less audible then two others
Solution: raise the audio publishing bitrate on client to send more data per one participant
Code Block | ||||
---|---|---|---|---|
| ||||
constraints: {
audio: {
bitrate: 128000
}
} |
and raise Opus encoding bitrate on server
Code Block | ||
---|---|---|
| ||
opus.encoder.bitrate=128000text_bulk_write_with_buffer=false |
7. When a number of participants are speaking simultaneously, some participants are less audible then others. The problem is typical for mixing WebRTC streams, and may ocuur in any solutions using WebRTC and audio mixing, in Discord for example: more data are encoded to one stream, more audio samples may be dropped due to fixed samplerate.
Symptoms: when three or more participants are speaking simultaneously, one of them is less audible then two others
Solution: raise the audio publishing bitrate on client to send more data per one participant
Code Block | ||||
---|---|---|---|---|
| ||||
constraints: {
audio: {
bitrate: 128000
}
} |
and raise Opus encoding bitrate on server
Code Block | ||
---|---|---|
| ||
opus.encoder.bitrate=128000 |
8. An incoming stream will be never encoded in realtime mixer if media traffic is stopped in the stream
Symptoms: an incoming stream freezes in the mixer when media traffic is stopped in the stream, for example, an application window captured by screen sharing is minimized to task bar
Solution: decrease a minimal FPS threshold for mixer incoming streams
Code Block | ||
---|---|---|
| ||
mixer_video_stable_fps_threshold=0 |
9. Mixer output stream bitrate may be unstable when incoming stream bitrate or fps drops
Symptoms: mixer output stream bitrate becomes unstable when incoming stream bitrate or fps drops
Solution: update WCS to build 5.2.1843 or newer and set the parameter
Code Block | ||
---|---|---|
| ||
h264_encoder_filler_data_padding=true |
10. There may be a stuttering in mixer audio for a stream published with a packets loss
Symptoms: micro freezes and stuttering for a certain stream in mixer
Solution: increase the mixer incoming streams buffer
Code Block | ||
---|---|---|
| ||
mixer_in_buffering_ms=1000 |