Server default settings are mostly universal and need to be tuned to certain client case.
Server load and memory consumption optimization
Garbage collector tuning
Garbage collector (GC) is an important part of Java VM. When GC is running, it dramatically increases the server load and may stop another tasks execution, therefore it is recommended to minimize GC launch with the following settings in wcs-core.properties file
#Disable heuristic rules -XX:+UseCMSInitiatingOccupancyOnly #Reduce Old Gen threshold -XX:CMSInitiatingOccupancyFraction=70 # Use System.gc() concurrently in CMS -XX:+ExplicitGCInvokesConcurrent # Disable System.gc() for RMI, for 10000 hours -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000
The Z Garbage Collector
The Z Garbage Collector (ZGC) is a scalable low latency garbage collector for Java 12. ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10ms, which makes is suitable for applications which require low latency and/or use a very large heap. You should pay attention to ZGC requires more processor resources compared to CMS GC.
Example installation of ZGC with OpenJDK version 12:
1. Download the latest release of OpenJDK 12 from URL http://jdk.java.net/12/:
2. Extract the OpenJDK to the folder and move its contents to the working directory:
tar xvf openjdk-12.0.2_linux-x64_bin.tar.gz mv jdk-12.0.2 /usr/java/jdk-12.0.2
3. Create symbolic links to OpenJDK 12:
ln -sf /usr/java/jdk-12.0.2 /usr/java/default ln -sf /usr/java/default/bin/java /usr/bin/java ln -sf /usr/java/default/bin/jstack /usr/bin/jstack ln -sf /usr/java/default/bin/jcmd /usr/bin/jcmd ln -sf /usr/java/default/bin/jmap /usr/bin/jmap
4. Verify your Java installation:
java --version openjdk 12.0.2 2019-07-16 OpenJDK Runtime Environment (build 12.0.2+10) OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)
5. Install the WCS (if required).
6. Tune the wcs-core.properties (for example, allocating 24G under memory heap):
# ZGC -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g -XX:+UseLargePages -XX:ZPath=/hugepages
7. Сonfigure ZGC according to the recommendations (the number of memory pages (2048K each) with a margin of a quarter more than that obtained from calculating the memory for heap (1.25 * 24G / 2M)) and add the required parameters in the server startup:
mkdir /hugepages echo "echo 13051 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" >>/etc/rc.local echo "mount -t hugetlbfs -o uid=0 nodev /hugepages" >>/etc/rc.local chmod +x /etc/rc.d/rc.local systemctl enable rc-local.service systemctl restart rc-local.service
8. After restarting WCS, the gc-core.log log files show the periodic operation of the garbage collector. To understand the working model of Z Garbage Collector, you can read the presentation.
Heap memory tuning
Many objects with data are created and destroyed in memory while streaming. Therefore it is recommended to allocate at least 1/2 of server physical memory for Java memory heap. For example, if server RAM is 32 Gb, then it is recommended to allocate 16 Gb with the following settings in wcs-core.properties file
REST client tuning
When REST hooks are used, on every WCS server action (establishing client connection, publishing and playing a stream, making a SIP call etc) HTTP REST connection to backend server is established. With a large number of simultaneously publishing clients or subscribers, with the default WCS settings it is possible to exhaust the WCS REST client thread pool, that is lead to deadlocks. Then, server stops to publish and play streams.