Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Heap memory tuning

Many data objects are created and destroyed in memory while streaming. Therefore, it is recommended to allocate at least 1/2 of server physical memory for Java memory heap. For example, if server RAM is 32 Gb, then it is recommended to allocate 16 Gb with the following settings in wcs-core.properties file:

...

Code Block
languagebash
themeRDark
#Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly

#Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70

# Use System.gc() concurrently in CMS
-XX:+ExplicitGCInvokesConcurrent

# Disable System.gc() for RMI, for 10000 hours
-Dsun.rmi.dgc.client.gcInterval=36000000000
-Dsun.rmi.dgc.server.gcInterval=36000000000

...

Code Block
languagebash
themeRDark
# Used CMS GC
-XX:+UseConcMarkSweepGC -Xms24g -Xmx24g -XX:NewSize=6144m -XX:MaxNewSize=6144m

# Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly

# Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70

# Log
-Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-
-XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log

...

The Z Garbage Collector (ZGC) is a scalable low latency garbage collector for Java 12. ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10 milliseconds, which makes it suitable for applications requiring low latency and/or use a very large heap. It should be noted that ZGC requires more processor resources than CMS GC.

Example of installing of ZGC with OpenJDK version Here is the example of ZGC setup using OpenJDK 12:

1. Download the latest release of OpenJDK 12 from http://jdk.java.net/12/:

Code Block
languagebash
themeRDark
wget https://download.java.net/java/GA/jdk12.0.2/e482c34c86bd4bf8b56c0b35558996b9/10/GPL/openjdk-12.0.2_linux-x64_bin.tar.gz

2. Extract the archive to folder and move its contents to the working directory:

Code Block
languagebash
themeRDark
tar xvf openjdk-12.0.2_linux-x64_bin.tar.gz
mv jdk-12.0.2 /usr/java/jdk-12.0.2

3. Create symbolic links to OpenJDK 12:

Code Block
languagebash
themeRDark
ln -sfn /usr/java/jdk-12.0.2 /usr/java/default
ln -sfn /usr/java/default/bin/java /usr/bin/java
ln -sfn /usr/java/default/bin/jstack /usr/bin/jstack
ln -sfn /usr/java/default/bin/jcmd /usr/bin/jcmd
ln -sfn /usr/java/default/bin/jmap /usr/bin/jmap

4. Install OpenJDK 12 as described here

2. Verify your Java installation:

Code Block
languagebash
themeRDark
java --version

openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10)
OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

53. Install the WCS (if required).6. Tune the wcs

-core.properties 4. If WCS is already installed, comment or remove the following lines in wcs-core.properties file

Code Block
themeRDark
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails

Change the following line from

Code Block
themeRDark
-Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-

to

Code Block
themeRDark
-Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time

5. Add the following setting to wcs-core.properties (for example, allocating 24G under memory heap):

language
Code Block
bashthemeRDark
# ZGC
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g

6. If hugepages is planning to use, add the following settings to wcs-core.properties:

  • in JDK 12 or 14
Code Block
themeRDark
-XX:+UseLargePages -XX:ZPath=/hugepages
  • in JDK 15 and newer
Code Block
themeRDark
 -XX:+UseLargePages -XX:ZPathAllocateHeapAt=/hugepages

7. Сonfigure ZGC Then configure hugepages according to the recommendations (the number of memory pages (2048K each) with a margin of a quarter more than that obtained from calculating to the memory for heap (1.25 ,125 * 24G * 1024 / 2M)) and add the required parameters in the server startup (Centos example):

Code Block
languagebash
themeRDark
sudo mkdir /hugepages
sudo echo "echo 1305113824 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" >>/etc/rc.local
sudo echo "mount -t hugetlbfs -o uid=0,mode=0757 nodev /hugepages" >>/etc/rc.local
sudo chmod +x /etc/rc.d/rc.local
sudo systemctl enable rc-local.service
sudo systemctl restart rc-local.service

...


sudo chmod o+w /hugepages

7.  After restarting the WCS, the gc-core.log log files show the periodic operation of the garbage collector. To understand the working model of Z Garbage Collector, you can see this presentation.

Image Modified

Physical memory allocation tuning on system level

When server is under a high load, there can be not enough a physical memory map areas which are avalable to a proccess in system by default. This can lead to JVM crash due to lack of native memory. In this case, crash log contains the following comment:

Code Block
themeRDark
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
...

To prevent such crashes, increase memory map areas count available to a procces with the following system parameter

Code Block
languagebash
themeRDark
sysctl -w vm.max_map_count=131072

and restart WCS.

Known issues

1. CPU load average is higher when ZGC is used, especially in JDK 15

Symptoms: CPU load average measured at system level (using htop for example) grows after update from JDK 12 or 14 to 15 if ZGC is used

Solution: use ZGC in JDK 12 or 14 only fro high loaded servers if GC pauses minimizing is required