Heap memory tuning
Many data objects are created and destroyed in memory while streaming. Therefore, it is recommended to allocate at least 1/2 of server physical memory for Java memory heap. For example, if server RAM is 32 Gb, then it is recommended to allocate 16 Gb with the following settings in wcs-core.properties file:
...
Code Block |
---|
|
#Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly
#Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70
# Use System.gc() concurrently in CMS
-XX:+ExplicitGCInvokesConcurrent
# Disable System.gc() for RMI, for 10000 hours
-Dsun.rmi.dgc.client.gcInterval=36000000000
-Dsun.rmi.dgc.server.gcInterval=36000000000 |
...
Code Block |
---|
|
# Used CMS GC
-XX:+UseConcMarkSweepGC -Xms24g -Xmx24g -XX:NewSize=6144m -XX:MaxNewSize=6144m
# Disable heuristic rules
-XX:+UseCMSInitiatingOccupancyOnly
# Reduce Old Gen threshold
-XX:CMSInitiatingOccupancyFraction=70
# Log
-Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-
-XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log |
...
The Z Garbage Collector (ZGC) is a scalable low latency garbage collector for Java 12. ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10 milliseconds, which makes it suitable for applications requiring low latency and/or use a very large heap. It should be noted that ZGC requires more processor resources than CMS GC.
Example of installing Here is the example of ZGC with setup using OpenJDK version 12:
1. Download the latest release of OpenJDK 12 from http://jdk.java.net/12/:
Code Block |
---|
|
wget https://download.java.net/java/GA/jdk12.0.2/e482c34c86bd4bf8b56c0b35558996b9/10/GPL/openjdk-12.0.2_linux-x64_bin.tar.gz |
2. Extract the archive to folder and move its contents to the working directory:
Code Block |
---|
|
tar xvf openjdk-12.0.2_linux-x64_bin.tar.gz
mv jdk-12.0.2 /usr/java/jdk-12.0.2 |
3. Create symbolic links to OpenJDK 12:
Code Block |
---|
|
ln -sfn /usr/java/jdk-12.0.2 /usr/java/default
ln -sfn /usr/java/default/bin/java /usr/bin/java
ln -sfn /usr/java/default/bin/jstack /usr/bin/jstack
ln -sfn /usr/java/default/bin/jcmd /usr/bin/jcmd
ln -sfn /usr/java/default/bin/jmap /usr/bin/jmap
|
4. Install OpenJDK 12 as described here
2. Verify your Java installation:
Code Block |
---|
|
java --version
openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10)
OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing) |
53. Install WCS (if required).6. Tune the wcs-core
.properties 4. If WCS is already installed, comment or remove the following lines in wcs-core.properties file
Code Block |
---|
|
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails |
Change the following line from
Code Block |
---|
|
-Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core- |
to
Code Block |
---|
|
-Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time |
5. Add the following setting to wcs-core.properties (for example, allocating 24G under memory heap):
Code Block |
---|
|
# ZGC
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms24g -Xmx24g |
Code Block |
---|
|
# ZGC
-XX:+UseZGC -Xms24g -Xmx24g |
6. If hugepages is planning to use, add the following settings to wcs-core.properties:
Code Block |
---|
|
-XX:+UseLargePages -XX:ZPath=/hugepages |
Code Block |
---|
|
-XX:+UseLargePages -XX:AllocateHeapAt=/hugepages |
7. Сonfigure ZGC Then configure hugepages according to the recommendations (the number of memory pages (2048K each) with a margin of a quarter more than that obtained from calculating to the memory for heap (1.25 ,125 * 24G * 1024 / 2M)) and add the required parameters in the server startup (Centos example):
Code Block |
---|
|
sudo mkdir /hugepages
sudo echo "echo 1305113824 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" >>/etc/rc.local
sudo echo "mount -t hugetlbfs -o uid=0,mode=0757 nodev /hugepages" >>/etc/rc.local
sudo chmod +x /etc/rc.d/rc.local
sudo systemctl enable rc-local.service
sudo systemctl restart rc-local.service |
...
sudo chmod o+w /hugepages |
7. After restarting the WCS, the gc-core.log log files show the periodic operation of the garbage collector. To understand the working model of Z Garbage Collector, you can see this presentation.
Image Modified
Physical memory allocation tuning on system level
When server is under a high load, there can be not enough a physical memory map areas which are avalable to a proccess in system by default. This can lead to JVM crash due to lack of native memory. In this case, crash log contains the following comment:
Code Block |
---|
|
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
... |
To prevent such crashes, increase memory map areas count available to a procces with the following system parameter
Code Block |
---|
|
sysctl -w vm.max_map_count=131072 |
and restart WCS.
Known issues
1. CPU load average is higher when ZGC is used, especially in JDK 15
Symptoms: CPU load average measured at system level (using htop for example) grows after update from JDK 12 or 14 to 15 if ZGC is used
Solution: use ZGC in JDK 12 or 14 only fro high loaded servers if GC pauses minimizing is required
2. ZGC logs may occupy a huge disk space with default output configuration
Symptoms: gc-core*.log
files occupy a huge disk space
Solution: limit log tags set to write to GC logs with the following parameter in wcs-core.properties
Code Block |
---|
|
-Xlog:gc,gc+start,gc+phases:/usr/local/FlashphonerWebCallServer/logs/gc-core-:time |