Information on load and resources of the WCS server is available via the port 8081 at this URL

http://test.flashphoner.com:8081/?action=stat

where test.flashphoner.com - is the actual address of the WCS server.

The information includes data on published and played streams, SIP calls and server specifications.

Statistics parameters

Connection

Connections to the WCS server

connections

Overall number of connections

connections_rtmfp

Number of RTMFP connections (Flash)

connections_websocket

Number of WebSocket connections

Port

Ports listened by the WCS server

ports_media_free

Free media ports

ports_media_busy

Busy media ports

ports_media_quarantine

Media ports in quarantine

ports_wcs_agents_free

Free CDN used ports

ports_wcs_agents_busy

Busy CDN used ports

ports_wcs_agents_quarantine

CDN used ports in quarantine

Stream

Audio and video stream sessions on the WCS server

streams_rtsp_in

Number of active RTSP sessions sending traffic

streams_rtsp_out

Number of active RTSP sessions receiving traffic

streams_websocket_out

Number of active streams played by WebSocket

streams_rtmp_out

Number of active RTMP streams

streams_webrtc_in

Number of active streams published via WebRTC

streams_webrtc_out

Number of active streams played via WebRTC

streams_rtmfp_in

Number of active streams published via RTMFP

streams_rtmfp_out

Number of active streams played via RTMFP

streams_viewers

Number of active clients playing published streams

streams_synchronizationStreams published synchronization

Native resources

Information about resource usage

native_resources

Information about encoding/decoding of the media stream

Core

System parameters (obtained using Java methods) and parameters of the WCS server

core_threads

Number of active threads of the WCS server process

core_java_threads

Current number of live threads

core_java_committedMemory

Amount of guaranteed available virtual memory, in bytes

core_java_freePhysicalMemorySize

Amount of free physical memory, in bytes

core_java_arch

Operating system architecture

core_java_availableProcessors

Number of processors available for the Java virtual machine

core_java_freeSwapSpaceSize

Amount of free swap space, in bytes

core_java_maxFileDescriptorCount

Maximum number of file descriptors

core_java_open_file_descriptors

Number of file descriptors opened in the Java virtual machine

core_java_cpu_usage

"Recent CPU usage" for a process in the Java virtual machine

core_java_totalPhysicalMemorySize

Amount of physical memory, in bytes

core_java_totalSwapSpaceSize

Amount of swap space, in bytes

core_java_uptime

Uptime since Java virtual machine started, in milliseconds

core_heap_memory_used

Heap memory usage

core_rss_memory

Memory usage

core_open_file_descriptors

Number of opened file descriptors

core_cpu_usage

CPU time percentage used by processes

core_gc

Information about "garbage collecting"

core_gc_manager

Information about "garbage collecting" in the Manager

Call

SIP calls on the WCS server

sip_calls

Number of SIP calls

sip_calls_established

Number of active SIP calls

sip_calls_in

Number of incoming SIP calls

sip_calls_out

Number of outgoing SIP calls

sip_calls_per_second (cps)

Number of SIP calls per minute

SIP


sip_registered

Number of client in the REGISTERED state

Recording

Audio and video file recording

recording_sessions

Number of active recording sessions

System

system parameters

system_java_cpu_usage

"Recent CPU usage" for the entire system (obtained using Java methods)

system_java_load_average

Average system load for the last minute (obtained using Java methods)

system_uptime

Uptime since starting of the Linux server

system_memory_total

Available RAM, in kilobytes

system_memory_free

Amount of physical RAM not used by the system, in kilobytes

system_cpu_usage

CPU time percentage used by the processes of the core

Network Stats (Mbit/s)

Network traffic statistics

global_bandwidth_in

Incoming connection channel bandwidth

global_bandwidth_out

Outgoing connection channel bandwidth

Version info

WCS version info

wcs_version

Current version of the WCS server

wcs_client_version

Current version of the WCS API

Errors info

Error information (obtained based on caught Java exceptions)

java.io.IOException

Number of IO errors

java.lang.ArrayIndexOutOfBoundsException

Number of array out of bounds errors

java.lang.IllegalArgumentException

Number of invalid function argument errors

com.flashphoner.server.license.activation.A.C

Number of errors while activating the license

java.lang.NullPointerException

Number of null pointer jump errors

Degraded streamsDegraded streams information
degraded_streamsDegraded streams quantity
degraded_streams_percentDegraded streams percentage of total number of streams on server

Transcoding stats

Streams transcoding information

transcoding_video_decoding_resolutions

List of resolutions decoded in "resolution/streams count" form

transcoding_video_decoding_average_time

Average decoding time in "resolution/time, ms" form

transcoding_video_decoding_max_time

Maximum decoding time in "resolution/time, ms" form

transcoding_video_decoding_average_queue_size

Average decoding queue size in "resolution/queue size" form

transcoding_video_decoding_max_queue_size

Maximum decoding queue size in "resolution/queue size" form

transcoding_video_encoding_resolutions

List of resolutions encoded in "resolution/streams count" form

transcoding_video_encoding_average_time

Average encoding time in "resolution/time, ms" form

transcoding_video_encoding_max_time

Maximum encoding time in "resolution/time, ms" form

transcoding_video_encoding_average_queue_size

Average encoding queue size in "resolution/queue size" form

transcoding_video_encoding_max_queue_size

Maximum encoding queue size in "resolution/queue size" form
Transcoding video full infoStream transcoding details (JSON format only)
CDN statsCDN information
cdn_version
CDN version supported by server
cdn_role
Server role in CDN
cdn_group
CDN group server belongs to

Every parameter can be requested separately, for example:

http://test.flashphoner.com:8081/?action=stat&params=native_resources

Load and resources usage information of the WCS server, combined by groups, can be requested by the group name (connections_stats, streams_stats, port_stats, call_stats, degraded_streams_stats, system_stats, core_stats are available)

http://test.flashphoner.com:8081/?action=stat&groups=call_stats

You can include several group names in the request

http://test.flashphoner.com:8081/?action=stat&groups=call_stats,connections_stats

System parameters can be excluded from full report using the following parameter in flashphoner.properties file:

parse_system_stats=false

Statistics output formats

By default, statistics is shown as plain text

-----Connection Stats-----
connections=1
connections_rtmfp=0
connections_websocket=1
-----Port Stats-----
ports_media_free=495
ports_media_busy=4
ports_media_quarantine=0
-----Stream Stats-----
streams_webrtc_in=1
streams_webrtc_out=1
streams_websocket_out=0
streams_rtmfp_in=0
streams_rtmfp_out=0
streams_rtmp_in=0
streams_rtmp_out=0
streams_viewers=test/1
streams_rtsp_in=0
streams_rtsp_out=0
streams_rtmp_client_out=0
streams_play_rate=0
streams_stop_rate=0
-----Native Resources-----
native_resources=140537376620832,NENC:H264/FFMPEG,1859;140537236444800,FFDecoderNative:H264/FFMPEG,39404417
native_resources.audio_codecs=0
native_resources.audio_resamplers=0
native_resources.video_transcoders=0
native_resources.video_decoders=1
native_resources.video_encoders=1
native_resources.writers=0
-----Core Stats-----
core_java_threads=71
core_java_committedMemory=3127017472
core_java_freePhysicalMemorySize=69820416
core_java_arch=amd64
core_java_availableProcessors=2
core_java_freeSwapSpaceSize=1044369408
core_java_maxFileDescriptorCount=20000
core_java_open_file_descriptors=206
core_java_cpu_usage=39.88
core_java_totalPhysicalMemorySize=1927303168
core_java_totalSwapSpaceSize=1073737728
core_java_uptime=102191
-----Call Stats-----
sip_processed_calls=0
sip_calls_state=established/0;trying/0;ringing/0;ring/0;ring_media/0;hold/0;busy/0;finish/0;session_progress/0;pending/0;failed/0
sip_calls=0
sip_calls_established=0
sip_calls_in=0
sip_calls_out=0
sip_calls_per_second=0.00
-----Sip Stats-----
sip_registered=0
-----Recording Stats-----
recording_sessions=0
-----System Stats-----
system_java_cpu_usage=75.00
system_java_load_average=0.87
-----Network Stats (Mbit/s)-----
global_bandwidth_in=0.000
global_bandwidth_out=0.000
-----Version info-----
wcs_version=5.2.416-32aab7dc90527bfe2ffb4711090fe75c6613a2bb
wcs_client_version=0.5.28.2753-9ba78705727d3ee9d75c1c10f488d21cd00a601c
-----Errors info-----
-----Degraded streams-----
degraded_streams=
degraded_streams_percent=0

Since build 5.2.416, JSON and Prometheus formats are supported. To get JSON statistics, set the format in request URI

curl -s 'http://localhost:8081/?action=stat&format=json'
{
  "connections_stats": {
    "connections": "1",
    "connections_rtmfp": "0",
    "connections_websocket": "1"
  },
  "port_stats": {
    "ports_media_free": "495",
    "ports_media_busy": "4",
    "ports_media_quarantine": "0"
  },
  "streams_stats": {
    "streams_webrtc_in": "1",
    "streams_webrtc_out": "1",
    "streams_websocket_out": "0",
    "streams_rtmfp_in": "0",
    "streams_rtmfp_out": "0",
    "streams_rtmp_in": "0",
    "streams_rtmp_out": "0",
    "streams_viewers": [
      "test/1"
    ],
    "streams_rtsp_in": "0",
    "streams_rtsp_out": "0",
    "streams_rtmp_client_out": "0",
    "streams_play_rate": "0",
    "streams_stop_rate": "0"
  },
  "native_resources": {
    "native_resources": [
      "140537376620832,NENC:H264/FFMPEG,9819",
      "140537236444800,FFDecoderNative:H264/FFMPEG,209561611"
    ],
    "native_resources.audio_codecs": "0",
    "native_resources.audio_resamplers": "0",
    "native_resources.video_transcoders": "0",
    "native_resources.video_decoders": "1",
    "native_resources.video_encoders": "1",
    "native_resources.writers": "0"
  },
  "core_stats": {
    "core_java_threads": "67",
    "core_java_committedMemory": "3127017472",
    "core_java_freePhysicalMemorySize": "73224192",
    "core_java_arch": "amd64",
    "core_java_availableProcessors": "2",
    "core_java_freeSwapSpaceSize": "1044107264",
    "core_java_maxFileDescriptorCount": "20000",
    "core_java_open_file_descriptors": "188",
    "core_java_cpu_usage": "37.19",
    "core_java_totalPhysicalMemorySize": "1927303168",
    "core_java_totalSwapSpaceSize": "1073737728",
    "core_java_uptime": "358833"
  },
  "call_stats": {
    "sip_processed_calls": "0",
    "sip_calls_state": [
      "established/0",
      "trying/0",
      "ringing/0",
      "ring/0",
      "ring_media/0",
      "hold/0",
      "busy/0",
      "finish/0",
      "session_progress/0",
      "pending/0",
      "failed/0"
    ],
    "sip_calls": "0",
    "sip_calls_established": "0",
    "sip_calls_in": "0",
    "sip_calls_out": "0",
    "sip_calls_per_second": "0.00"
  },
  "sip_stats": {
    "sip_registered": "0"
  },
  "recording_stats": {
    "recording_sessions": "0"
  },
  "system_stats": {
    "system_java_cpu_usage": "50.00",
    "system_java_load_average": "0.73"
  },
  "network_stats": {
    "global_bandwidth_in": "0.000",
    "global_bandwidth_out": "0.000"
  },
  "version_stats": {
    "wcs_version": "5.2.416-32aab7dc90527bfe2ffb4711090fe75c6613a2bb",
    "wcs_client_version": "0.5.28.2753-9ba78705727d3ee9d75c1c10f488d21cd00a601c"
  },
  "errors_info": {},
  "degraded_streams_stats": {
    "degraded_streams": [],
    "degraded_streams_percent": "0"
  }
}

To get Prometheus statistics, set the format in request URI

curl -s 'http://localhost:8081/?action=stat&format=prometheus'
connections_stats{param="connections"} 1
connections_stats{param="connections_rtmfp"} 0
connections_stats{param="connections_websocket"} 1
port_stats{param="ports_media_free"} 495
port_stats{param="ports_media_busy"} 4
port_stats{param="ports_media_quarantine"} 0
streams_stats{param="streams_webrtc_in"} 1
streams_stats{param="streams_webrtc_out"} 1
streams_stats{param="streams_websocket_out"} 0
streams_stats{param="streams_rtmfp_in"} 0
streams_stats{param="streams_rtmfp_out"} 0
streams_stats{param="streams_rtmp_in"} 0
streams_stats{param="streams_rtmp_out"} 0
streams_stats{param="streams_viewers",name="test"} 1
streams_stats{param="streams_rtsp_in"} 0
streams_stats{param="streams_rtsp_out"} 0
streams_stats{param="streams_rtmp_client_out"} 0
streams_stats{param="streams_play_rate"} 0
streams_stats{param="streams_stop_rate"} 0
native_resources{param="native_resources",id="140537376620832",name="NENC:H264/FFMPEG"} 11129
native_resources{param="native_resources",id="140537236444800",name="FFDecoderNative:H264/FFMPEG"} 239113192
native_resources{param="native_resources.audio_codecs"} 0
native_resources{param="native_resources.audio_resamplers"} 0
native_resources{param="native_resources.video_transcoders"} 0
native_resources{param="native_resources.video_decoders"} 1
native_resources{param="native_resources.video_encoders"} 1
native_resources{param="native_resources.writers"} 0
core_stats{param="core_java_threads"} 63
core_stats{param="core_java_committedMemory"} 3127017472
core_stats{param="core_java_freePhysicalMemorySize"} 67538944
core_stats{core_java_arch="amd64"} 1
core_stats{param="core_java_availableProcessors"} 2
core_stats{param="core_java_freeSwapSpaceSize"} 1043853312
core_stats{param="core_java_maxFileDescriptorCount"} 20000
core_stats{param="core_java_open_file_descriptors"} 188
core_stats{param="core_java_cpu_usage"} 37.02
core_stats{param="core_java_totalPhysicalMemorySize"} 1927303168
core_stats{param="core_java_totalSwapSpaceSize"} 1073737728
core_stats{param="core_java_uptime"} 401113
call_stats{param="sip_processed_calls"} 0
call_stats{param="sip_calls_state",name="established"} 0
call_stats{param="sip_calls_state",name="trying"} 0
call_stats{param="sip_calls_state",name="ringing"} 0
call_stats{param="sip_calls_state",name="ring"} 0
call_stats{param="sip_calls_state",name="ring_media"} 0
call_stats{param="sip_calls_state",name="hold"} 0
call_stats{param="sip_calls_state",name="busy"} 0
call_stats{param="sip_calls_state",name="finish"} 0
call_stats{param="sip_calls_state",name="session_progress"} 0
call_stats{param="sip_calls_state",name="pending"} 0
call_stats{param="sip_calls_state",name="failed"} 0
call_stats{param="sip_calls"} 0
call_stats{param="sip_calls_established"} 0
call_stats{param="sip_calls_in"} 0
call_stats{param="sip_calls_out"} 0
call_stats{param="sip_calls_per_second"} 0.00
sip_stats{param="sip_registered"} 0
recording_stats{param="recording_sessions"} 0
system_stats{param="system_java_cpu_usage"} 66.67
system_stats{param="system_java_load_average"} 0.84
network_stats{param="global_bandwidth_in"} 0.000
network_stats{param="global_bandwidth_out"} 0.000
version_stats{wcs_version="5.2.416-32aab7dc90527bfe2ffb4711090fe75c6613a2bb"} 1
version_stats{wcs_client_version="0.5.28.2753-9ba78705727d3ee9d75c1c10f488d21cd00a601c"} 1
degraded_streams_stats{param="degraded_streams"} 0
degraded_streams_stats{param="degraded_streams_percent"} 0

Stream transcoding statistics

Since build 5.2.443 brief or detailed stream transcoding statistics can be shown. The brief stream transcoding information is grouped by stream resolutions and is available in all the formats, for example:

-----Transcoding info-----
transcoding_video_decoding_resolutions=640x360/1
transcoding_video_decoding_average_time=640x360/2.0
transcoding_video_decoding_max_time=640x360/2
transcoding_video_decoding_average_queue_size=640x360/0.0
transcoding_video_decoding_max_queue_size=640x360/0
transcoding_video_encoding_resolutions=426x240/1;640x360/1;852x480/1
transcoding_video_encoding_average_time=426x240/2.0;640x360/2.0;852x480/6.0
transcoding_video_encoding_max_time=426x240/2;640x360/2;852x480/6
transcoding_video_encoding_average_queue_size=426x240/0.0;640x360/0.0;852x480/0.0
transcoding_video_encoding_max_queue_size=426x240/0;640x360/0;852x480/0

Here the following parameters are shown:

The detailed stream transcoding information is available in JSON format only

curl -s 'http://localhost:8081/?action=stat&format=json&groups=transcoding_stats' | jq

and is grouped by streams published

    "transcoding_video_full_info": {
      "test1": {
        "codec": "H264",
        "queueSize": 0,
        "distributors": [
          {
            "codec": "H264",
            "queueSize": 0,
            "resolution": "426x240",
            "averageProcessingTime": 3
          },
          {
            "codec": "H264",
            "queueSize": 0,
            "resolution": "640x360",
            "averageProcessingTime": 5
          },
          {
            "codec": "H264",
            "queueSize": 0,
            "resolution": "852x480",
            "averageProcessingTime": 10
          }
        ],
        "resolution": "640x360",
        "averageProcessingTime": 3
      }
    }

Where:

CDN information

Since build 5.2.471 CDN statistics output was added for server participating in CDN

curl -s 'http://localhost:8081/?action=stat&groups=cdn_stats'
-----CDN info-----
cdn_version=2.3
cdn_role=ORIGIN
cdn_group=

Where:

Streams published synchronization

Since build 5.2.508 audio and video synchronization information for streams published is added to stream statistics:

-----Stream Stats-----
streams_synchronization=stream1/90,stream2/-11

Synchronization metric value is calculated as difference between current audio and video synchronization values:

var metric = lastAudioSync - lastVideoSync;

Therefore, a positive metric value means audio is currently ahead of video, and negative means audio is currently behind video.

This metric normally changes withis small limits. If synchronization metric for the stream remaihs high with constant sign, it mean some problem with stream publishing.