Server default settings are mostly universal and need to be tuned to certain client case.
Server load and memory consumption optimization
Garbage collector tuning
Garbage collector (GC) is an important part of Java VM. When GC is running, it dramatically increases the server load and may stop another tasks execution, therefore it is recommended to minimize GC launch with the following settings in wcs-core.properties file
The Z Garbage Collector
The Z Garbage Collector (ZGC) is a scalable low latency garbage collector for Java 12. ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10ms, which makes is suitable for applications which require low latency and/or use a very large heap. You should pay attention to ZGC requires more processor resources compared to CMS GC.
Example installation of ZGC with OpenJDK version 12:
1. Download the latest release of OpenJDK 12 from URL http://jdk.java.net/12/:
2. Extract the OpenJDK to the folder and move its contents to the working directory:
3. Create symbolic links to OpenJDK 12:
4. Verify your Java installation:
5. Install the WCS (if required).
6. Tune the wcs-core.properties (for example, allocating 24G under memory heap):
7. Сonfigure ZGC according to the recommendations (the number of memory pages (2048K each) with a margin of a quarter more than that obtained from calculating the memory for heap (1.25 * 24G / 2M)) and add the required parameters in the server startup:
8. After restarting WCS, the gc-core.log log files show the periodic operation of the garbage collector. To understand the working model of Z Garbage Collector, you can read the presentation.
Heap memory tuning
Many objects with data are created and destroyed in memory while streaming. Therefore it is recommended to allocate at least 1/2 of server physical memory for Java memory heap. For example, if server RAM is 32 Gb, then it is recommended to allocate 16 Gb with the following settings in wcs-core.properties file
REST client tuning
When REST hooks are used, on every WCS server action (establishing client connection, publishing and playing a stream, making a SIP call etc) HTTP REST connection to backend server is established. With a large number of simultaneously publishing clients or subscribers, with the default WCS settings it is possible to exhaust the WCS REST client thread pool, that is lead to deadlocks. Then, server stops to publish and play streams.
By default, a maximum number of simultaneous REST connections is set to 200 with the following parameter in flashphoner.properties file
To escape thred poolexhausting and deadlocks this value should be reduced, for example
If REST hooks are not used, REST client can be disabled with the following parameter
Excessive logging supression
When REST hooks are used, REST client operations, EchoApp default backend operations and REST API server operations are written to WCS core logs. That leads to large number of entries in the log file and, therefore, inceases the server load. The excessive logging may be decreased if necessary using the following parameters in log4j.properties file:
Streaming mediadata are transferred with UDP packets. Those packets can be dropped, for example if server does not have enough time to parse packet queue, that leads to picture quality loss and freezes. To escape this it is necessary to tune UDP sockets buffers with the following settings in flashphoner.properties file
and to tune system queues with command
To diagnose UDP problem, it is necessary to track UDP packets dropping with command
Channel load optimization
Users' playback picture quality depends on bitrate: the higher the bitrate, the higher the quality. However, the higher the bitrate, the higher data transfer channel load and, if the bandwidth between the server and clients is limited, there is a possibility that the channel will be fully loaded. This leads the bitrate dropping and a sharp decline in quality.
In this regard, it is necessary to limit the bitrate to ensure sufficient picture quality with an acceptable channel load.
Publisher bitrate limiting
Server bitrate limiting
Minimum and maximum bitrate values in bps on server may be set with the following parameters in flashphoner.properties file
To exclude fast bitrate rise bu=y browser, the following parameter should be set
Stream decoding on demand only must be switched on to reduce server load:
Changing dynamic ports range in Linux
Dynamic or ephemeral port is a temporary port that is opened when establishing IP-connection from certain range of TCP/IP stack. Many Linux kernel versions use ports range 32768 — 61000 as dymanic ports. Enter the following command to check what range is used on server
If this range overlaps with WCS standard ports, it should be changed with the following command
Adjusting the maximum number of opened files
In the launch script
webcallserver that is in subfolder
bin in WCS home folder, for example
start() function the maximum number of opened files is set
By default, this value is set to 20000, but it may be increased if necessary, following the limitations of the operating system used.