HAProxy is a reliable open source tool to implement reverse proxy server and load balancers. Most of well know load balancers, AWS LB for example, are based on HAProxy modified source code. Let's see how to setup a custom load balancer using HAProxy.
To deploy load balancer, prepare the following:
If WCS servers supposed to be in CDN, CDN setup should be done before. For example, if the goal is to balance publishers to a number of Origin servers, or subscribers to a number of Edges, all those instances should be configured before deploying the load balancer.
Open all the necessary ports for incoming connections on every WCS server (if this is not already done). Look at the minimal ports setup example for AWS EC2 instance
Note that TCP port 9707 should be added. This port will be used by HAProxy to check a current server state.
Media traffic ports (30000-33000 in the example above) should be available from outside networks if server is behind a NAT, because HAProxy may proxy Websocket connectinos only, but not WebRTC.
Add the following parameters to flashphoner.properties
file to use a real client IP addresses in session identifiers
ws.map_custom_headers=true ws.ip_forward_header=X-Client-IP |
If servers load supposed to be balanced depenfing on channel bandwidth, add the following setting too
global_bandwidth_check_enabled=true |
Then restart WCS
sudo systemctl restart webcallserver |
3.1. Install all the necessary dependencies to the server
yum install jq bc xinetd telnet |
3.2. Copy the scripts haproxy-agent-check.sh
and haproxy-agent-check-launch.sh
to /usr/local/bin
folder and allow execution
sudo cp haproxy-agent-check* /usr/local/bin/ sudo chmod +x /usr/local/bin/haproxy-agent-check* |
#! /bin/bash /usr/local/bin/haproxy-agent-check.sh cpu 70 |
#!/bin/bash CPU_MAX_LOAD=90 MAX_PUBLISHERS=100 MAX_SUBSCRIBERS=100 MAX_HLS_STREAMS=100 MAX_BANDWIDTH_IN=100 MAX_BANDWIDTH_OUT=100 function isTreshold_Cpu() { local load=$(uptime | grep -E -o 'load average[s:][: ].*' | sed 's/,//g' | cut -d' ' -f3-5) local cpus=$(grep processor /proc/cpuinfo | wc -l) local l5util=0 while read -r l1 l5 l15; do { l5util=$(echo "pct=$l5/$cpus*100; if(pct<1) print 0; pct" | bc -l | cut -d"." -f1); if [[ $l5util -lt $CPU_MAX_LOAD ]]; then true; return else false; return fi }; done < <(echo $load) } function isTreshold_Publishers() { local statsJson=$1 local webrtcPublishers=$(echo $statsJson | jq '.streams_stats.streams_webrtc_in' | bc -l) local rtmpPublishers=$(echo $statsJson | jq '.streams_stats.streams_rtmp_in' | bc -l) local rtspStreamsIn=$(echo $statsJson | jq '.streams_stats.streams_rtsp_in' | bc -l) local rtspPublishers=$(echo $statsJson | jq '.streams_stats.streams_rtsp_push_in' | bc -l) local publishers=$(($webrtcPublishers + $rtmpPublishers + $rtspStreamsIn + $rtspPublishers)) if [[ $publishers -lt $MAX_PUBLISHERS ]]; then true; return else false; return fi } function isTreshold_Subscribers() { local statsJson=$1 local webrtcSubscribers=$(echo $statsJson | jq '.streams_stats.streams_webrtc_out' | bc -l) local rtmpSubscribers=$(echo $statsJson | jq '.streams_stats.streams_rtmp_out' | bc -l) local rtmpRepublishers=$(echo $statsJson | jq '.streams_stats.streams_rtmp_client_out' | bc -l) local rtspSubscribers=$(echo $statsJson | jq '.streams_stats.streams_rtsp_out' | bc -l) local websocketSubscribers=$(echo $statsJson | jq '.streams_stats.streams_websocket_out' | bc -l) local subscribers=$(($webrtcSubscribers + $rtmpSubscribers + $rtmpRepublishers + $rtspSubscribers + $websocketSubscribers)) if [[ $subscribers -lt $MAX_SUBSCRIBERS ]]; then true; return else false; return fi } function isTreshold_HlsStreams() { local statsJson=$1 local hlsStreams=$(echo $statsJson | jq '.streams_stats.streams_hls' | bc -l) if [[ $hlsStreams -lt $MAX_HLS_STREAMS ]]; then true; return else false; return fi } function isTreshold_BandwidthIn() { local statsJson=$1 local bandwidthIn=$(echo $statsJson | jq '.network_stats.global_bandwidth_in' | bc -l) local comparison=$(echo "$bandwidthIn < $MAX_BANDWIDTH_IN" | bc -l) if [[ $comparison -ne 0 ]]; then true; return else false; return fi } function isTreshold_BandwidthOut() { local statsJson=$1 local bandwidthOut=$(echo $statsJson | jq '.network_stats.global_bandwidth_out' | bc -l) local comparison=$(echo "$bandwidthOut < $MAX_BANDWIDTH_OUT" | bc -l) if [[ $comparison -ne 0 ]]; then true; return else false; return fi } function usage() { echo "Usage: $(basename $0) [OPTIONS]" echo -e " cpu [<treshold>]\t\tcheck CPU load (default 90 %)" echo -e " publishers [<treshold>]\tcheck publishers count (default 100)" echo -e " subscribers [<treshold>]\tcheck subscribers count (default 100)" echo -e " hls [<treshold>]\t\tcheck HLS streams count (default 100)" echo -e " band-in [<treshold>]\t\tcheck incoming channel bandwidth (default 100 Mbps)" echo -e " band-out [<treshold>]\t\tcheck outgoing channel bandwidth (default 100 Mbps)" echo "" echo -e "Example: $(basename $0) cpu 90 publishers 100 subscribers 100 hls 100 band-in 100 band-out 100" exit 0 } function main() { local checklist=() local statsJson="" local check="" if [[ $# -eq 0 ]]; then checklist=( 'Cpu' 'Publishers' 'Subscribers' 'HlsStreams' 'BandwidthIn' 'BandwidthOut' ) else while [[ $# -gt 0 ]]; do case $1 in cpu) checklist+=('Cpu') if [ -z "${2//[0-9]}" ]; then CPU_MAX_LOAD=$2 shift fi shift ;; publishers) checklist+=('Publishers') if [ -z "${2//[0-9]}" ]; then MAX_PUBLISHERS=$2 shift fi shift ;; subscribers) checklist+=('Subscribers') if [ -z "${2//[0-9]}" ]; then MAX_SUBSCRIBERS=$2 shift fi shift ;; hls) checklist+=('HlsStreams') if [ -z "${2//[0-9]}" ]; then MAX_HLS_STREAMS=$2 shift fi shift ;; band-in) checklist+=('BandwidthIn') if [ -z "${2//[0-9]}" ]; then MAX_BANDWIDTH_IN=$2 shift fi shift ;; band-out) checklist+=('BandwidthOut') if [ -z "${2//[0-9]}" ]; then MAX_BANDWIDTH_OUT=$2 shift fi shift ;; help|*) usage ;; esac done fi if [[ -z "${checklist[@]}" ]]; then usage return 1 fi statsJson=$(curl -s 'http://localhost:8081/?action=stat&format=json') if [[ -z "$statsJson" ]]; then echo "down" return 1 fi for check in ${checklist[@]}; do if ! isTreshold_$check $statsJson; then echo "down" return 1 fi done echo "up 100%" return 0 } main "$@" exit $? |
The haproxy-agent-check.sh
is used to check server state according to system information and WCS statistics. If any of thresholds passed to the script is reached, the script will return down
state. HAProxy, in its turn, will not dispatch a new connections to the server until the agent script returns up
.
The following tresholds are supported:
For example, to check if CPU LA is below 70%, the script sholud be launched as
/usr/local/bin/haproxy-agent-check.sh cpu 70 |
3.3. Add the following string to /etc/services
file
haproxy-agent-check 9707/tcp # haproxy-agent-check |
3.4. Add the file haproxy-agent-check
with the following content to the folder /etc/xinetd.d
# default: on # description: haproxy-agent-check service haproxy-agent-check { disable = no flags = REUSE socket_type = stream port = 9707 wait = no user = nobody server = /usr/local/bin/haproxy-agent-check-launch.sh log_on_failure += USERID only_from = 172.31.42.154 127.0.0.1 per_source = UNLIMITED } |
The helper script haproxy-agent-check-launch.sh
is used because xinetd does not support any command line keys in server
parameter
The only_from
parameters allows connections to the port 9707 only from load balancer server where HAProxy will be installed, and from localhost for testing purposes.
3.5. Allow haproxy-agent-check
execution
sudo chmod +x /etc/xinetd/haproxy-agent-check |
3.6. Restart xinetd
sudo systemctl restart xinetd |
3.7. Test the agent work
telnet localhost 9707 |
1.1. Install nginx
sudo yum install nginx |
1.2. Change default port in /etc/nginx/nginx.conf
file, and set the server name as localhost
server { listen 8180; listen [::]:8180; server_name localhost; root /usr/share/nginx/html; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; error_page 404 /404.html; location = /404.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } |
nginx will be available only locally because HAProxy will provide an entry point for clients.
1.3. Restart nginx
sudo systemctl restart nginx |
1.4. Download WebSDK actual build bundle
wget https://flashphoner.com/downloads/builds/flashphoner_client/wcs_api-2.0/flashphoner-api-2.0.206-7d9863ae4de631a59ff8793ddecd104ca2fd4a22.tar.gz |
and unpack it to the /usr/share/nginx/html/wcs
folder
sudo mkdir /usr/share/nginx/html/wcs cd /usr/share/nginx/html/wcs sudo tar -xzf ~/flashphoner-api-2.0.206-7d9863ae4de631a59ff8793ddecd104ca2fd4a22.tar.gz --strip-components=2 |
2.1. Create a full certificate file in PEM format (must include all the certificates and a private key) and copy to a folder whele certificate file should be available
cat cert.crt ca.crt cert.key >> cert.pem sudo mkdir -p /etc/pki/tls/mydomain.com sudo cp cert.pem /etc/pki/tls/mydomain.com |
3.1. Install HAProxy
sudo yum install haproxy |
3.2. Edit the file /etc/haproxy/haproxy.cfg
#--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log /dev/log local0 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend wcs-balancer bind *:443 ssl crt /etc/pki/tls/mydomain.com/cert.pem acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket hdr(Sec-WebSocket-Key) -m found use_backend wcs_back if is_websocket default_backend wcs_web_admin #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend wcs_back http-request add-header X-Client-IP %ci:%cp balance roundrobin server wcs1_ws 172.31.44.243:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 server wcs2_ws 172.31.33.112:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 #--------------------------------------------------------------------- # WCS web admin dashboard #--------------------------------------------------------------------- backend wcs_web_admin server wcs_web_http localhost:8180 maxconn 100 check |
All the parameters in global
and defaults
sections may be left by default. Configure frontend
frontend wcs-balancer bind *:443 ssl crt /etc/pki/tls/mydomain.com/cert.pem acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket hdr(Sec-WebSocket-Key) -m found use_backend wcs_back if is_websocket default_backend wcs_web_admin |
Set nginx with WebSDK examples as default backend
backend wcs_web_admin server wcs_web_http localhost:8180 maxconn 100 check |
Backen to balance a load between two instances (IP addresses are private and shown for example only)
backend wcs_back http-request add-header X-Client-IP %ci:%cp balance roundrobin server wcs1_ws 172.31.44.243:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 server wcs2_ws 172.31.33.112:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 |
Session stickiness may be set up as follows
backend wcs_back http-request add-header X-Client-IP %ci:%cp balance roundrobin cookie SERVERID insert indirect nocache server wcs1_ws 172.31.44.243:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 cookie wcs1_ws server wcs2_ws 172.31.33.112:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 cookie wcs1_ws |
In this case, all the connections from a certain client will be redirected to the same server unless it returns down
state
Load balancing by maximum client connections to the server may be configured as follows
backend wcs_back http-request add-header X-Client-IP %ci:%cp balance leastconn server wcs1_ws 172.31.44.243:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 server wcs2_ws 172.31.33.112:8080 maxconn 100 weight 100 check agent-check agent-inter 5s agent-port 9707 |
In this case all the clients will be redirected to the first server until either maxconn
is reached or or server returns down
state
3.3. Restart HAProxy
sudo systemctl restart haproxy |
1. Open Two Way Streaming example, set port 443 in Websocket URL input field and publish a stream
2. Check the statistics page on the first WCS server
One Websocket connection (1), one incoming stream (2) named test
(3) are displayed
3. Check session Id
Client IP address and port are used in session Id.
4. Open Two Way Streaming example in another browser window, set port 443 in Websocket URL input field and publish a second stream
5. Check the statistics page on the second WCS server
One Websocket connection (1), one incoming stream (2) named test2
(3) are displayed