Skip to end of metadata
Go to start of metadata

Since build 5.2.607 real-time stream mixer function is added. This function with MCU support is designed for real-time video confereincing. Unlike the previous implementation, a real-time mixer does not stop the output stream when some of incoming streams is late (usually due to packet loss or channel interference), and does not wait for bad quality stream to restore.

Configuration

Real-time mixer is enabled by default

mixer_realtime=true

Mixer automatic creation when publishing a stream named like user1#room1 is also enabled by default

mixer_auto_start=true

For conferencing, MCU support should be enabled

mixer_mcu_audio=true
mixer_mcu_video=true

It is recommended to reduce mixer idle timeout

mixer_idle_timeout=10000

Stream name displaying in stream picture caption can be enabled if necessary

mixer_display_stream_name=true

in this case caption will be displayed in picture bottom left corner for video streams and in picture center for audio only streams.

Speech indicator greenframe is enabled by default

mixer_voice_activity=true

Another mixer settings are also supported. Note that it is not recommended to enable custom losless video processor because real time function will not work in this case.

Captions management

Mixer captions font size can be changed if necessary depending on mixer outpout stream resolution:

  • for video streams, caption font size if 20pt by default
mixer_font_size=20
  • for audio only streams, caption font size if 40pt by default
mixer_font_size_audio_only=40

Caption text is also displayed on black rectangle background. The following caption background parameters can be tuned:

Parameter

Default value in pixels

Description

mixer_text_cut_top3Text top cut
mixer_text_padding_bottom5Text bottom padding
mixer_text_padding_left5Text left padding
mixer_text_padding_right4Text right padding
mixer_text_padding_top5Text top padding

Caption text autoscaling

Since build 5.2.709 caption text can be autoscaled to fit the picture. The feature is enabled by default with thr following parameter

mixer_text_autoscale=true

If one or more publishers share a screen, a special desktop layout is enabled, in this case desktop stream caption is scaled separately

mixer_autoscale_desktop=true

A minimal font size for text scaling is set by the following parameter

mixer_minimal_font_size=1

to 1 pt by default

Speech indicator management

Speech indicator frame thikness can be set (6 pixels by default)

mixer_voice_activity_frame_thickness=6

The frame is displayed outside astream picture by default. However, if stream pictures are too close, foe example, if CenterNoPaddingGridLayout is used, the frame can affect neighbor pictures

In this case, inner speech frame displaying should be enabled

mixer_voice_activity_frame_position_inner=true

By default, if audio only track from some stream was added to mixer, this stream is shown in separate frame with speech indicator (see above). Sound only can be added to mixer without showing a separate participant (for example, to comment or to dub) with the following parameter if necessary

mixer_show_separate_audio_frame=false

Mixer background management

By default, mixer applies black background. Since build 5.2.645 it is possible to set PNG picture file which will be used as mixer background. For example, to change the background to blue, prepare blue.png picture and set the following parameter

mixer_video_background_filename=blue.png

Background picture file should be placed to /usr/local/FlashphonerWebCallServer/conf if full path is not set. The file can be placed to any folder on the server, in this case full path to the file should be set to the parameter

mixer_video_background_filename=/opt/media/blue.png

Background picture will be automatically scaled to mixer output stream resolution

Audio only streams picture aspect ratio management

By default, audio only streams picture aspect ratio is set according to mixer one (16:9)

mixer_audio_only_width=640
mixer_audio_only_height=360
mixer_video_width=1280
mixer_video_height=720

It can be changed together with mixer if necessary, for example to 4:3

mixer_audio_only_width=640
mixer_audio_only_height=480
mixer_video_width=1280
mixer_video_height=960

Note that audio only streams picture aspect ratio settings are applied only with WCS restart.

Real-time mixer tuning

Real-time mixer can be tuned using the following parameters

Parameter

Default value

Description

mixer_audio_silence_threshold-50.00Incoming stream audio silence level in Db
mixer_debug_modefalseAdds some debug info to stream picture caption

mixer_in_buffering_ms

200

Incoming stream video buffer in milliseconds

mixer_incoming_time_rate_lower_threshold0.95Relative incoming stream time to mixer time rate lower threshold
mixer_incoming_time_rate_upper_threshold1.05Relative incoming stream time to mixer time rate upper threshold
mixer_video_stable_fps_threshold15Incoming stream FPS threshold, video buffer will be disabled for streams with low fps

Testing

1. For test we use

2. Open MCU Client web application page. Enter user name user1 and room name room1

2. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed without your microphone audio

3. Open MCU Client application page in another brower tab/browser/PC. Enter user name user2 and room name room1

4. Click Join. A stream from your web camera will be published and added to mixer, then mixer output stream will be displayed with videos from both users but audio from user1 only

5. On user1 page two videos also are playing with user2 audio only

6. Click Leave to leave the room in both tabs/browsers

Call flow

Call flow for conferencing example based on real-time mixer with MCU function is described on MCU Client page.

Incoming streams tuning recommendations

When delay occurs in the incoming stream from one of participants, realtime mixer will freeze that stream. The following is recommended to minimize incoming stream delays:

1. For RTMP streams, adjust encoding parameter so that:

  • client encoder perfomance to be enough to send picture frames in time
  • stream resolution and bitrate to fit to publishers channel from client to server

2. For WebRTC streams, do not raise minimum video bitrate threshold higher than webrtc_cc_min_bitrate server configuration parameter defines. By default, lower video bitrate threshold is set to 30 kbps

webrtc_cc_min_bitrate=30000

The publisher client browser will adopt the stream to channel quality drops. The lower bitrate the lower picture quality, but the participant stream will not be freezed in this case.

Known issues

1. Real-time mixer functions are disabled when custom losless videoprocessor is used, mixer incoming streams cannot be played

Symptoms: stream stops playing after adding to mixer

Solution: do not use custom losless videoprocessor with real-time mixer

mixer_lossless_video_processor_enabled=false

2. To display stream captions in mixer, it would be necessary to install fontconfig library

Symptoms: streams cannot be added to mixer with the following exception in server log

09:17:11,756 ERROR           MixerAgent - MIXER-AGENT-mixer://mixervmixdr52-9d46cd04-5867-4d74-a9d9-baf67f74e7d2 Mixer closed due to error
java.lang.InternalError: java.lang.reflect.InvocationTargetException
	at java.desktop/sun.font.FontManagerFactory$1.run(FontManagerFactory.java:86)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:310)
	at java.desktop/sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74)
	at java.desktop/sun.font.SunFontManager.getInstance(SunFontManager.java:247)
	at java.desktop/sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:265)
	at java.desktop/java.awt.Font.getStringBounds(Font.java:2606)
	at java.desktop/java.awt.Font.getStringBounds(Font.java:2516)
	at com.flashphoner.media.N.A.A.A(Unknown Source)
	at com.flashphoner.media.mixer.video.presentation.Canvas.computeTextScales(Unknown Source)
	at com.flashphoner.media.mixer.video.presentation.Canvas.writeNative(Unknown Source)
	at com.flashphoner.media.N.A.A(Unknown Source)
	at com.flashphoner.media.N.D.D(Unknown Source)
	at com.flashphoner.media.N.D.A(Unknown Source)
	at com.flashphoner.server.remote.B.B.Г™(Unknown Source)
	at com.flashphoner.server.remote.B.B.run(Unknown Source)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
	at java.desktop/sun.font.FontManagerFactory$1.run(FontManagerFactory.java:84)
	... 14 more
Caused by: java.lang.NullPointerException
	at java.desktop/sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1262)
	at java.desktop/sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:225)
	at java.desktop/sun.awt.FontConfiguration.init(FontConfiguration.java:107)
	at java.desktop/sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:719)
	at java.desktop/sun.font.SunFontManager$2.run(SunFontManager.java:367)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:310)
	at java.desktop/sun.font.SunFontManager.<init>(SunFontManager.java:312)
	at java.desktop/sun.awt.FcFontManager.<init>(FcFontManager.java:35)
	at java.desktop/sun.awt.X11FontManager.<init>(X11FontManager.java:56)
	... 20 more

Solution: install fontconfig library

sudo yum install -y fontconfig
  • No labels