Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Heap memory tuning

Many data objects with data are created and destroyed in memory while streaming. Therefore, it is recommended to allocate at least 1/2 of server physical memory for Java memory heap. For example, if server RAM is 32 Gb, then it is recommended to allocate 16 Gb with the following settings in wcs-core.properties file:

...

Code Block
languagebash
themeRDark
#Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly

#Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70

# Use System.gc() concurrently in CMS
-XX:+ExplicitGCInvokesConcurrent

# Disable System.gc() for RMI, for 10000 hours
-Dsun.rmi.dgc.client.gcInterval=36000000000
-Dsun.rmi.dgc.server.gcInterval=36000000000

...

The Concurrent Mark Sweep (CMS) collector is designed for applications that prefer shorter garbage collection pauses and that can afford to share processor resources with the garbage collector while the application is running. This collector should be considered for any application with a low pause time requirement.

...

Code Block
languagebash
themeRDark
# Used CMS GC
-XX:+UseConcMarkSweepGC -Xms24g -Xmx24g -XX:NewSize=6144m -XX:MaxNewSize=6144m

# Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly

# Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70

# Log
-Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-
-XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log

2. After restarting WCS, we can see the result of the garbage collector in the gc-core.log log files we see the result of the garbage collectorfile. The output may vary depending on the installed version of Java. For example:,

openjdk version "1.8.0_222":

...

The Z Garbage Collector (ZGC) is a scalable low latency garbage collector for Java 12. ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10ms10 milliseconds, which makes is it suitable for applications which require requiring low latency and/or use a very large heap.  You should pay attention to It should be noted that ZGC requires more processor resources compared to than CMS GC.

Example installation Here is the example of ZGC with setup using OpenJDK version 12:

1. Download the latest release of OpenJDK 12 from URL http://jdk.java.net/12/:

Code Block
languagebash
themeRDark
wget https://download.java.net/java/GA/jdk12.0.2/e482c34c86bd4bf8b56c0b35558996b9/10/GPL/openjdk-12.0.2_linux-x64_bin.tar.gz

2. Extract the OpenJDK to the folder and move its contents to the working directory:

Code Block
languagebash
themeRDark
tar xvf openjdk-12.0.2_linux-x64_bin.tar.gz
mv jdk-12.0.2 /usr/java/jdk-12.0.2

3. Create symbolic links to OpenJDK 12:

Code Block
languagebash
themeRDark
ln -sfn /usr/java/jdk-12.0.2 /usr/java/default
ln -sfn /usr/java/default/bin/java /usr/bin/java
ln -sfn /usr/java/default/bin/jstack /usr/bin/jstack
ln -sfn /usr/java/default/bin/jcmd /usr/bin/jcmd
ln -sfn /usr/java/default/bin/jmap /usr/bin/jmap

4. Install OpenJDK 12 as described here

2. Verify your Java installation:

Code Block
languagebash
themeRDark
java --version

openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10)
OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

53. Install the WCS (if required).

4. 6. Tune the wcs-core.properties If WCS is already installed, comment or remove the following lines in wcs-core.properties file

Code Block
themeRDark
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails

Change the following line from

Code Block
themeRDark
-Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-

to

Code Block
themeRDark
-Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time

5. Add the following setting to wcs-core.properties (for example, allocating 24G under memory heap):

bash
Code Block
language
themeRDark
# ZGC
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g

6. If hugepages is planning to use, add the following settings to wcs-core.properties:

  • in JDK 12 or 14
Code Block
themeRDark
-XX:+UseLargePages -XX:ZPath=/hugepages
  • in JDK 15 and newer
Code Block
themeRDark
 -XX:+UseLargePages -XX:ZPathAllocateHeapAt=/hugepages

7. Сonfigure ZGC Then configure hugepages according to the recommendations (the number of memory pages (2048K each) with a margin of a quarter more than that obtained from calculating to the memory for heap (1.25 ,125 * 24G * 1024 / 2M)) and add the required parameters in the server startup (Centos example):

Code Block
languagebash
themeRDark
sudo mkdir /hugepages
sudo echo "echo 1305113824 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" >>/etc/rc.local
sudo echo "mount -t hugetlbfs -o uid=0,mode=0757 nodev /hugepages" >>/etc/rc.local
sudo chmod +x /etc/rc.d/rc.local
sudo systemctl enable rc-local.service
sudo systemctl restart rc-local.service

...


sudo chmod o+w /hugepages

7.  After restarting the WCS, the gc-core.log log files show the periodic operation of the garbage collector. To understand the working model of Z Garbage Collector, you can read the presentation.

...

see this presentation.

Image Added

Physical memory allocation tuning on system level

When server is under a high load, there can be not enough a physical memory map areas which are avalable to a proccess in system by default. This can lead to JVM crash due to lack of native memory. In this case, crash log contains the following comment:

Code Block
themeRDark
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
...

To prevent such crashes, increase memory map areas count available to a procces with the following system parameter

Code Block
languagebash
themeRDark
sysctl -w vm.max_map_count=131072

and restart WCS.

Known issues

1. CPU load average is higher when ZGC is used, especially in JDK 15

Symptoms: CPU load average measured at system level (using htop for example) grows after update from JDK 12 or 14 to 15 if ZGC is used

Solution: use ZGC in JDK 12 or 14 only fro high loaded servers if GC pauses minimizing is required

2. ZGC logs may occupy a huge disk space with default output configuration

Symptoms: gc-core*.log files occupy a huge disk space

Solution: limit log tags set to write to GC logs with the following parameter in wcs-core.properties

Code Block
themeRDark
-Xlog:gc,gc+start,gc+phases:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time