...

Server default settings are mostly universal and need to be tuned to certain client case.

Server load and memory consumption optimization

Garbage collector tuning

Garbage collector (GC) is an important part of Java VM. When GC is running, it dramatically increases the server load and may stop another tasks execution, therefore it is recommended to minimize GC launch with the following settings in wcs-core.properties file

#Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly

#Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70

# Use System.gc() concurrently in CMS
-XX:+ExplicitGCInvokesConcurrent

# Disable System.gc() for RMI, for 10000 hours
-Dsun.rmi.dgc.client.gcInterval=36000000000
-Dsun.rmi.dgc.server.gcInterval=36000000000

The Z Garbage Collector

The Z Garbage Collector (ZGC) is a scalable low latency garbage collector for Java 12. ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10ms, which makes is suitable for applications which require low latency and/or use a very large heap. You should pay attention to ZGC requires more processor resources compared to CMS GC.

Example installation of ZGC with OpenJDK version 12:

1. Download the latest release of OpenJDK 12 from URL http://jdk.java.net/12/:

wget https://download.java.net/java/GA/jdk12.0.2/e482c34c86bd4bf8b56c0b35558996b9/10/GPL/openjdk-12.0.2_linux-x64_bin.tar.gz

2. Extract the OpenJDK to the folder and move its contents to the working directory:

tar xvf openjdk-12.0.2_linux-x64_bin.tar.gz
mv jdk-12.0.2 /usr/java/jdk-12.0.2

3. Create symbolic links to OpenJDK 12:

ln -sf /usr/java/jdk-12.0.2 /usr/java/default
ln -sf /usr/java/default/bin/java /usr/bin/java
ln -sf /usr/java/default/bin/jstack /usr/bin/jstack
ln -sf /usr/java/default/bin/jcmd /usr/bin/jcmd
ln -sf /usr/java/default/bin/jmap /usr/bin/jmap

4. Verify your Java installation:

java --version

openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10)
OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

5. Install the WCS (if required).

6. Tune the wcs-core.properties (for example, allocating 24G under memory heap):

# ZGC
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g -XX:+UseLargePages -XX:ZPath=/hugepages

7. Сonfigure ZGC according to the recommendations (the number of memory pages (2048K each) with a margin of a quarter more than that obtained from calculating the memory for heap (1.25 * 24G / 2M)) and add the required parameters in the server startup:

mkdir /hugepages
echo "echo 13051 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" >>/etc/rc.local
echo "mount -t hugetlbfs -o uid=0 nodev /hugepages" >>/etc/rc.local
chmod +x /etc/rc.d/rc.local
systemctl enable rc-local.service
systemctl restart rc-local.service

8.  After restarting WCS, the gc-core.log log files show the periodic operation of the garbage collector. To understand the working model of Z Garbage Collector, you can read the presentation.

Image Removed

Heap memory tuning

Many objects with data are created and destroyed in memory while streaming. Therefore it is recommended to allocate at least 1/2 of server physical memory for Java memory heap. For example, if server RAM is 32 Gb, then it is recommended to allocate 16 Gb with the following settings in wcs-core.properties file

-Xmx16g
-Xms16g

REST client tuning

When REST hooks are used, on every WCS server action (establishing client connection, publishing and playing a stream, making a SIP call etc) HTTP REST connection to backend server is established. With a large number of simultaneously publishing clients or subscribers, with the default WCS settings it is possible to exhaust the WCS REST client thread pool, that is lead to deadlocks. Then, server stops to publish and play streams.

...

Adjusting the maximum number of opened files

Legacy settings (before build 5.2.762)

In the launch script webcallserver that is in subfolder bin in WCS home folder, for example

...

By default, this value is set to 20000, but it may be increased if necessary, following the limitations of the operating system used.

Using environment variable (since build 5.2.762)

Since build 5.2.762, maximum opened files limit can be set using the following environment variable

WCS_FD_LIMIT=20000

in setenv.sh file. When updating WCS from previous builds, this variable should be added to setenv.sh manually, for example

export WCS_FD_LIMIT=100000

Unlike the webcallserver startup script, the setenv.sh file is not overwritten on subsequent updates, therefore it is not necessary to restore this setting after every update.

Using service parameter while launching from non-root user (since build 5.2.801)

Since build 5.2.801, WCS is launching from 'flashphoner' user for better security. In this case, maximum opened files limit can be set using service parameters

sudo nano /etc/systemd/system/webcallserver.service

Maximum opened files limit is set with LimitNOFILE parameter, for example

[Service]
User=flashphoner
Group=flashphoner
LimitNOFILE=100000
...

Internal command to change file descriptors limit

Since build 5.2.1255 the following command can be used to set file descriptors limit:

sudo ./webcallserver set-fd-limit 100000

WCS will be stopped before settings changing and will be automatically started after settings changing to apply them.

If a new value is less than the default one (20000), erroe message will be displayed, and changes will not be applied.

Traffic encryption in a separate thread for each client session

By default, one CPU thread encrypts medai traffic for all the client sessions. This leads to one CPU core overload by such thread, espacyally on low-power servers, for big subscribers amount. Then, server can not send mediapackets to all subscribers, and streams viewed are degrading, FPS lowering and freezing.

To distribute the load evenly across the CPU cores, it is necessary to enable traffic encryption in a separate thread for each client session with the following parameters

rtp_paced_sender=true
rtp_paced_sender_initial_rate=200000
rtp_paced_sender_increase_interval=50
rtp_paced_sender_k_up=0.9

and restart WCS.

Stream distribution optimization

A stream playback quality may drop when a number of subscribers are viewing it simutlaneouly (from 100 and more): low FPS, freezes. However, server capacity and channel bandwidth may be enough. In this case it is recommended to enable multithreaded stream distribution to subscribers using the following parameter

streaming_distributor_subgroup_enabled=true

In this case, audio and video client sessions are distributed by groups.

Maximum number of video sessions per group can be set with the following parameter

streaming_distributor_subgroup_size=50

Maximum number of audio sessions per group can be set with the following parameter 

streaming_distributor_audio_subgroup_size=500

Frame queue size per group and maximum frame waiting time (in milliseconds) are set by the following parameters

streaming_distributor_subgroup_queue_size=300
streaming_distributor_subgroup_queue_max_waiting_time=5000

for video and

streaming_distributor_audio_subgroup_queue_size=300
streaming_distributor_audio_subgroup_queue_max_waiting_time=5000

for audio sessions respectively.