Since build 5.2.607 real-time stream mixer function is added. This function with MCU support is designed for real-time video confereincing. Unlike the previous implementation, a real-time mixer does not stop the output stream when some of incoming streams is late (usually due to packet loss or channel interference), and does not wait for bad quality stream to restore.
Configuration
Real-time mixer is enabled by default
mixer_realtime=true
Mixer automatic creation when publishing a stream named like user1#room1 is also enabled by default
mixer_auto_start=true
For conferencing, MCU support should be enabled
mixer_mcu_audio=true mixer_mcu_video=true
It is recommended to reduce mixer idle timeout
mixer_idle_timeout=10000
Stream name displaying in stream picture caption can be enabled if necessary
mixer_display_stream_name=true
in this case caption will be displayed in picture bottom left corner for video streams and in picture center for audio only streams.
Speech indicator greenframe is enabled by default
mixer_voice_activity=true
Another mixer settings are also supported. Note that it is not recommended to enable custom losless video processor because real time function will not work in this case.
Captions management
Basic parameters and font size
Mixer captions font size can be changed if necessary depending on mixer output stream resolution:
- for video streams, caption font size if 20pt by default
mixer_font_size=20
- for audio only streams, caption font size if 40pt by default
mixer_font_size_audio_only=40
Caption text is also displayed on black rectangle background. The following caption background parameters can be tuned:
Parameter | Default value in pixels | Description |
---|---|---|
mixer_text_cut_top | 3 | Text top cut |
mixer_text_padding_bottom | 5 | Text bottom padding |
mixer_text_padding_left | 5 | Text left padding |
mixer_text_padding_right | 4 | Text right padding |
mixer_text_padding_top | 5 | Text top padding |
Text colour, background colour and opacity
Since build 5.2.741 caption text colour and background colour can be changed with the following parameters
mixer_text_colour=0xFFFF00 mixer_text_background_colour=0x006666
A colour can be set as hexadecimal value with # or 0x prefix, in #RRGGBB form. Using the settings above, yellow text will be displayed on cian background:
For audio only participants, background colour fills the rectangle.
Since build 5.2.770 caption background opacity can be changed using the following parameter
mixer_text_background_opacity=100
Background opacity should be set in percents: 0 for full transparency, 100 for full opacity (text background will be coloured to defined colour). The default value is 100 (fully opaque).
Font
Since build 5.2.770 it is possible to set captions font using the following parameter
mixer_text_font=Serif
The default value is Serif. Font can be chosen from X11 fonts list only, for example:
[root@centos76 ~]# fc-list | grep X11 /usr/share/X11/fonts/Type1/c0611bt_.pfb: Courier 10 Pitch:style=Bold Italic /usr/share/X11/fonts/Type1/UTBI____.pfa: Utopia:style=Bold Italic /usr/share/X11/fonts/Type1/c0419bt_.pfb: Courier 10 Pitch:style=Regular /usr/share/X11/fonts/Type1/c0648bt_.pfb: Bitstream Charter:style=Regular /usr/share/X11/fonts/Type1/cursor.pfa: Cursor:style=Regular /usr/share/X11/fonts/Type1/UTB_____.pfa: Utopia:style=Bold /usr/share/X11/fonts/Type1/c0583bt_.pfb: Courier 10 Pitch:style=Bold /usr/share/X11/fonts/Type1/UTI_____.pfa: Utopia:style=Italic /usr/share/X11/fonts/Type1/c0582bt_.pfb: Courier 10 Pitch:style=Italic /usr/share/X11/fonts/Type1/c0633bt_.pfb: Bitstream Charter:style=Bold Italic /usr/share/X11/fonts/Type1/c0649bt_.pfb: Bitstream Charter:style=Italic /usr/share/X11/fonts/Type1/c0632bt_.pfb: Bitstream Charter:style=Bold /usr/share/X11/fonts/Type1/UTRG____.pfa: Utopia:style=Regular
A full font name should be set, for example
mixer_text_font=Courier 10 Pitch
The default system font will be used if there is no the chosen font in the system.
If the chosen font contains no characters which are in caption text, those characters will not be displayed.
Caption text autoscaling
Since build 5.2.709 caption text can be autoscaled to fit the picture. The feature is enabled by default with the following parameter
mixer_text_autoscale=true
If one or more publishers share a screen, a special desktop layout is enabled, in this case desktop stream caption is scaled separately
mixer_autoscale_desktop=true
A minimal font size for text scaling is set by the following parameter
mixer_minimal_font_size=1
to 1 pt by default
Room (mixer) name displaying
Since build 5.2.770 it is possible to disable room name displaying (which is set to stream name while automatically adding stream to mixer). This can be done with the following parameter
mixer_text_display_room=false
In this case, for stream named "user1#room1" only "user1" will be displayed (without room name and special character).
Custom stream label displaying
Since build 5.2.816 it is possible to display custom stream label in mixer instead of stream and room name. In fact, almost any name can be assigned including national characters.
A label should be set in stream name while publishing, for example
session.createStream({ streamName: "test1#m1?label=John Doe", display: localDisplay, ... }).publish();
In this case the label will be displayed in mixer, including audio only streams
National characters are supported along with the following special characters
~!@#№$%^*()-_,.;:[]{}<>/|\
Character '+' will be replaced by whitespace, i.e. in the example above stream can be published with label
test1#m1?label=John+Doe
Caption location
Since build 5.2.844, it is possible to change location of video participant's caption. By default, caption is located in left bottom corner of video.
It is possible to display caption above video using the following parameter
mixer_text_outside_frame=TOP
or under video
mixer_text_outside_frame=BOTTOM
The distance between video and its caption can be changed using the following parameter
mixer_text_outside_frame_padding=70
Captions placement above or below a picture affects picture placement even if captions diplaying is disabled.
Speech indicator management
Thickness and position
Speech indicator frame thickness can be set (6 pixels by default)
mixer_voice_activity_frame_thickness=6
The frame is displayed outside astream picture by default. However, if stream pictures are too close, foe example, if CenterNoPaddingGridLayout
is used, the frame can affect neighbor pictures
In this case, inner speech frame displaying should be enabled
mixer_voice_activity_frame_position_inner=true
Audio only streams displaying
By default, if audio only track from some stream was added to mixer, this stream is shown in separate frame with speech indicator (see above). Sound only can be added to mixer without showing a separate participant (for example, to comment or to dub) with the following parameter if necessary
mixer_show_separate_audio_frame=false
The stream for dubbing must contain audio and video, and must be added to mixer with the following /mixer/add REST query
{ "uri": "mixer://mixer1", "remoteStreamName": "stream_dub", "hasVideo":false, "hasAudio":true }
Frame colour
Since build 5.2.741 speech indicator frame colour can be changed with the following parameter
mixer_voice_activity_colour=#FF0000
A colour can be set as hexadecimal value with # or 0x prefix, in #RRGGBB form. Using the settings above, speech indicator frame will be red.
Frame displaying when silence is in a stream
By default, speech indicator frame may blink at normal participant speech rate, which can be incomfortable for viewing. For this reason, since build 5.2.775 it is possible to set a time interval in which the frame will be displayed since silence is detected in the stream. The feature can be enabled using the following parameter
mixer_voice_activity_switch_delay=500
In this case, speech indicator frame will still be displayed in 500 milliseconds since silence is detected in the stream.
By default, this interval is set to 0, frame will be hidden without delay.
Mixer background management
By default, mixer applies black background. Since build 5.2.645 it is possible to set PNG picture file which will be used as mixer background. For example, to change the background to blue, prepare blue.png picture and set the following parameter
mixer_video_background_filename=blue.png
Background picture file should be placed to /usr/local/FlashphonerWebCallServer/conf if full path is not set. The file can be placed to any folder on the server, in this case full path to the file should be set to the parameter
mixer_video_background_filename=/opt/media/blue.png
Background picture will be automatically scaled to mixer output stream resolution
Audio only streams picture aspect ratio management
By default, audio only streams picture aspect ratio is set according to mixer one (16:9)
mixer_audio_only_width=640 mixer_audio_only_height=360 mixer_video_width=1280 mixer_video_height=720
It can be changed together with mixer if necessary, for example to 4:3
mixer_audio_only_width=640 mixer_audio_only_height=480 mixer_video_width=1280 mixer_video_height=960
Note that audio only streams picture aspect ratio settings are applied only with WCS restart.
Multithreading support and hign load optimizations
Since build 5.2.793 multithreading support is added to optimize a mixer for a big participants amount. This feature can be enabled by the following parameters
mixer_type=MULTI_THREADED_NATIVE mixer_mcu_multithreaded_mix=true
CPU threads count to mix audio amd video can be adjusted by the following parameters
mixer_audio_threads=10 mixer_video_threads=4
Usually, MCU mixer encodes one video stream and a many ausio streams: two per every participant plus one common audio track. Therefore, it is recommended to set more CPU threads for audio encoding than for video (as shown above). Additionally, if freezes occur in MCU mixer output stream, it is recommended to enable multithreaded delivery of mixing result to the main WCS engine, to send it to subscribers.
mixer_mcu_multithreaded_delivery=true
If participants count is lower than mixing threads count (for example, 3 participants), only one CPU thread will be used for mixing.
Real-time mixer tuning
Real-time mixer can be tuned using the following parameters
Parameter | Default value | Description |
---|---|---|
mixer_audio_silence_threshold | -50.00 | Incoming stream audio silence level in Db |
mixer_debug_mode | false | Adds some debug info to stream picture caption |
mixer_in_buffering_ms | 200 | Incoming stream video buffer in milliseconds |
mixer_incoming_time_rate_lower_threshold | 0.95 | Relative incoming stream time to mixer time rate lower threshold |
mixer_incoming_time_rate_upper_threshold | 1.05 | Relative incoming stream time to mixer time rate upper threshold |
mixer_video_stable_fps_threshold | 15 | Incoming stream FPS threshold, video buffer will be disabled for streams with low fps |
Testing
1. For test we use
- demo server demo.flashphoner.com;
- Chrome browser;
- MCU Client web application example for conferencing.
2. Open MCU Client web application page. Enter user name user1 and room name room1
2. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed without your microphone audio
3. Open MCU Client application page in another brower tab/browser/PC. Enter user name user2 and room name room1
4. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed with videos from both users but audio from user1 only
5. On user1 page two videos also are playing with user2 audio only
6. Click Leave to leave the room in both tabs/browsers
Call flow
Call flow for conferencing example based on real-time mixer with MCU function is described on MCU Client page.
Incoming streams tuning recommendations
When delay occurs in the incoming stream from one of participants, realtime mixer will freeze that stream. The following is recommended to minimize incoming stream delays:
1. For RTMP streams, adjust encoding parameter so that:
- client encoder perfomance to be enough to send picture frames in time
- stream resolution and bitrate to fit to publishers channel from client to server
2. For WebRTC streams, do not raise minimum video bitrate threshold higher than webrtc_cc_min_bitrate
server configuration parameter defines. By default, lower video bitrate threshold is set to 30 kbps
webrtc_cc_min_bitrate=30000
The publisher client browser will adopt the stream to channel quality drops. The lower bitrate the lower picture quality, but the participant stream will not be freezed in this case.
Adding one stream to two or more realtime mixers simultaneously
Since build 5.2.732 one stream can be added to two ore more realtime mixers simultaneously. Note that realtime mixer should be enabled
mixer_realtime=true
and custom losless videoprocessor should be disabled
mixer_lossless_video_processor_enabled=false
Known issues
1. Real-time mixer functions are disabled when custom losless videoprocessor is used, mixer incoming streams cannot be played
Symptoms: stream stops playing after adding to mixer
Solution: do not use custom losless videoprocessor with real-time mixer
mixer_lossless_video_processor_enabled=false
2. To display stream captions in mixer, it would be necessary to install fontconfig library
Symptoms: streams cannot be added to mixer with the following exception in server log
09:17:11,756 ERROR MixerAgent - MIXER-AGENT-mixer://mixervmixdr52-9d46cd04-5867-4d74-a9d9-baf67f74e7d2 Mixer closed due to error java.lang.InternalError: java.lang.reflect.InvocationTargetException at java.desktop/sun.font.FontManagerFactory$1.run(FontManagerFactory.java:86) at java.base/java.security.AccessController.doPrivileged(AccessController.java:310) at java.desktop/sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74) at java.desktop/sun.font.SunFontManager.getInstance(SunFontManager.java:247) at java.desktop/sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:265) at java.desktop/java.awt.Font.getStringBounds(Font.java:2606) at java.desktop/java.awt.Font.getStringBounds(Font.java:2516) at com.flashphoner.media.N.A.A.A(Unknown Source) at com.flashphoner.media.mixer.video.presentation.Canvas.computeTextScales(Unknown Source) at com.flashphoner.media.mixer.video.presentation.Canvas.writeNative(Unknown Source) at com.flashphoner.media.N.A.A(Unknown Source) at com.flashphoner.media.N.D.D(Unknown Source) at com.flashphoner.media.N.D.A(Unknown Source) at com.flashphoner.server.remote.B.B.Г™(Unknown Source) at com.flashphoner.server.remote.B.B.run(Unknown Source) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) at java.desktop/sun.font.FontManagerFactory$1.run(FontManagerFactory.java:84) ... 14 more Caused by: java.lang.NullPointerException at java.desktop/sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1262) at java.desktop/sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:225) at java.desktop/sun.awt.FontConfiguration.init(FontConfiguration.java:107) at java.desktop/sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:719) at java.desktop/sun.font.SunFontManager$2.run(SunFontManager.java:367) at java.base/java.security.AccessController.doPrivileged(AccessController.java:310) at java.desktop/sun.font.SunFontManager.<init>(SunFontManager.java:312) at java.desktop/sun.awt.FcFontManager.<init>(FcFontManager.java:35) at java.desktop/sun.awt.X11FontManager.<init>(X11FontManager.java:56) ... 20 more
Solution: install fontconfig library
sudo yum install -y fontconfig
3. A participant name can be drawn over speech indicator frame if there are many participants in mixer
Symptoms: for small participant pictures (in desktop layout, for example) participants name can be drawn over speech indicator frame
Solution: enable inner speech indicator frame position and decrease frame thickness
mixer_voice_activity_frame_position_inner=true mixer_voice_activity_frame_thickness=2
4. B-frames should be excluded from mixer input stream, or mixer input buffer should be increased
Symptoms: when mixer input stream is encoded by Main profile with B-frames, the mixer output stream is plaing unsmoothly, there are short freezes with sound pauses
Solution:
a) encode stream without B-frames (preferrable)
b) increase mixer input buffer (may give an additional delay)
mixer_in_buffering_ms=600
5. 60 FPS stream publishing to mixer gives additional CPU and system memory load
Symptoms: CPU load increasing and system memory (not Java heap) consumption growing while publishing 60 FPS stream to a mixer
Solution: publish 30 FPS streams to mixer or use more powerful server
6. Captions may twitch for many participants in MCU mixer
Symptoms: for many (over 10) participants in MCU mixer captions may periodically twitch
Solution: apply the following settings
mixer_text_bulk_write=false mixer_text_bulk_write_with_buffer=false