...
connections_stats | Connections to the WCS server |
---|---|
connections | Overall number of connections |
connections_rtmfp | Number of RTMFP connections (Flash) |
connections_websocket | Number of WebSocket connections |
port_stats | Ports listened by the WCS server |
ports_media_free | Free media ports |
ports_media_busy | Busy media ports |
ports_media_quarantine | Media ports in quarantine |
ports_wcs_agents_free | Free CDN used ports |
ports_wcs_agents_busy | Busy CDN used ports |
ports_wcs_agents_quarantine | CDN used ports in quarantine |
streams_stats | Audio and video stream sessions on the WCS server |
streams_rtsp_in | Number of active RTSP sessions sending traffic |
streams_rtsp_out | Number of active RTSP sessions receiving traffic |
streams_websocket_out | Number of active streams played by WebSocket |
streams_rtmp_out | Number of active RTMP streams |
streams_webrtc_in | Number of active streams published via WebRTC |
streams_webrtc_out | Number of active streams played via WebRTC |
streams_rtmfp_in | Number of active streams published via RTMFP |
streams_rtmfp_out | Number of active streams played via RTMFP |
streams_viewers | Number of active clients playing published streams |
streams_synchronization | Streams published synchronization |
native_resources | Information about resource usage |
native_resources | Information about encoding/decoding of the media stream |
core_stats | System parameters (obtained using Java methods) and parameters of the WCS server |
core_threads | Number of active threads of the WCS server process |
core_java_threads | Current number of live threads |
core_java_threads_blocked | Current number of blocked threads |
core_java_committedMemory | Amount of guaranteed available virtual memory, in bytes |
core_java_freePhysicalMemorySize | Amount of free physical memory, in bytes |
core_java_arch | Operating system architecture |
core_java_availableProcessors | Number of processors available for the Java virtual machine |
core_java_freeSwapSpaceSize | Amount of free swap space, in bytes |
core_java_maxFileDescriptorCount | Maximum number of file descriptors |
core_java_open_file_descriptors | Number of file descriptors opened in the Java virtual machine |
core_java_cpu_usage | "Recent CPU usage" for a process in the Java virtual machine |
core_java_totalPhysicalMemorySize | Amount of physical memory, in bytes |
core_java_totalSwapSpaceSize | Amount of swap space, in bytes |
core_java_uptime | Uptime since Java virtual machine started, in milliseconds |
core_heap_memory_used | Heap memory usage |
core_rss_memory | Memory usage |
core_open_file_descriptors | Number of opened file descriptors |
core_cpu_usage | CPU time percentage used by processes |
core_gc | Information about "garbage collecting" |
core_gc_manager | Information about "garbage collecting" in the Manager |
core_heap_memory_used | Heap memory used |
core_java_version | Java version |
core_java_nio_memory_buffer_count | TCP NIO buffers used count |
core_java_nio_memory_used | Memory used by TCP NIO buffers |
core_java_nio_memory_capacity | TCP NIO buffers memory capacity |
call_stats | SIP calls on the WCS server |
sip_calls | Number of SIP calls |
sip_calls_established | Number of active SIP calls |
sip_calls_in | Number of incoming SIP calls |
sip_calls_out | Number of outgoing SIP calls |
sip_calls_per_second (cps) | Number of SIP calls per minute |
sip_stats | SIP clients |
sip_registered | Number of client in the REGISTERED state |
recording_stats | Audio and video file recording |
recording_sessions | Number of active recording sessions |
recording_threads | Recording threads count |
recording_thread_min_writers | Minimal writers count per one thread |
recording_thread_max_writers | Maximal writers count per one thread |
recording_thread_average_writers | Average writers count per one thread |
recording_writers_list | Active writers list |
recording_writers_with_max_queue | Writers with maximum recording data queue size |
recording_writers_with_min_queue | Writers with minimum recording data queue size |
recording_min_writers_queue | Minimal recording data queue size |
recording_average_writers_queue | Average recording data queue size |
recording_max_writers_queue | Maximal recording data queue size |
system_stats | System parameters |
system_java_cpu_usage | "Recent CPU usage" for the entire system (obtained using Java methods) |
system_java_load_average | Average system load for the last minute (obtained using Java methods) |
system_uptime | Uptime since starting of the Linux server |
system_memory_total | Available RAM, in kilobytes |
system_memory_free | Amount of physical RAM not used by the system, in kilobytes |
system_cpu_usage | CPU time percentage used by the processes of the core |
network_stats | Network traffic statistics |
global_bandwidth_in | Incoming connection channel bandwidth |
global_bandwidth_out | Outgoing connection channel bandwidth |
version_stats | WCS version information |
wcs_version | Current version of the WCS server |
wcs_client_version | Current version of the Web SDK |
gc_stats | Last GC cycle information |
gc_last_pause_ms | Last GC pause, in milliseconds |
gc_last_duration_ms | Last GC cycle duration, in millisecinds |
gc_last_heap_before_mb | Last heap used before GC |
gc_last_heap_after_mb | Last heap used after GC |
errors_info | Error information (obtained based on caught Java exceptions) |
java.io.IOException | Number of IO errors |
java.lang.ArrayIndexOutOfBoundsException | Number of array out of bounds errors |
java.lang.IllegalArgumentException | Number of invalid function argument errors |
com.flashphoner.server.license.activation.A.C | Number of errors while activating the license |
java.lang.NullPointerException | Number of null pointer jump errors |
java.lang.OutOfMemoryError | Number of memory allocation errors (server must be restarted) |
degraded_streams_stats | Degraded streams information |
degraded_streams | Degraded streams quantity |
degraded_streams_percent | Degraded streams percentage of total number of streams on server |
transcoding_stats | Streams transcoding information |
transcoding_video_decoding_resolutions | List of resolutions decoded in "resolution/streams count" form |
transcoding_video_decoding_average_time | Average decoding time in "resolution/time, ms" form |
transcoding_video_decoding_max_time | Maximum decoding time in "resolution/time, ms" form |
transcoding_video_decoding_average_queue_size | Average decoding queue size in "resolution/queue size" form |
transcoding_video_decoding_max_queue_size | Maximum decoding queue size in "resolution/queue size" form |
transcoding_video_encoding_resolutions | List of resolutions encoded in "resolution/streams count" form |
transcoding_video_encoding_average_time | Average encoding time in "resolution/time, ms" form |
transcoding_video_encoding_max_time | Maximum encoding time in "resolution/time, ms" form |
transcoding_video_encoding_average_queue_size | Average encoding queue size in "resolution/queue size" form |
transcoding_video_encoding_max_queue_size | Maximum encoding queue size in "resolution/queue size" form |
transcoding_video_decoding_load | Integral decoding load value |
transcoding_video_encoding_load | Integral encoding load value |
buffer_output_stats | Pacer buffer usage information (JSON format only) |
cdn_stats | CDN information |
cdn_version | CDN version supported by server |
cdn_role | Server role in CDN |
cdn_group | CDN group server belongs to |
log_stats | Parameters collected with custom script |
Every parameter can be requested separately, for example:
Code Block | ||
---|---|---|
| ||
http://test.flashphoner.com:8081/?action=stat¶ms=native_resources |
Load and resources usage information of the WCS server, combined by groups, can be requested by the group name (connections_stats, streams_stats, port_stats, call_stats, degraded_streams_stats, system_stats, core_stats are available)
Code Block | ||
---|---|---|
| ||
http://test.flashphoner.com:8081/?action=stat&groups=call_stats |
You can include several group names in the request
Code Block | ||
---|---|---|
| ||
http://test.flashphoner.com:8081/?action=stat&groups=call_stats,connections_stats |
System parameters can be excluded from full report using the following parameter in flashphoner.properties file:
Code Block | ||
---|---|---|
| ||
parse_system_stats=false |
Statistics output formats
By default, statistics is shown as plain text
...
theme | RDark |
---|---|
title | Plain text statistics example |
collapse | true |
...
Logging stats | |
log_msg_per_sec | Logging messages written per second |
log_mbit_per_sec | Logging volume in megabits per second |
tcp_channel_stats | TCP channel stats |
channels_not_writable | TCP channels not writable events count, per stream |
jitter_buffer_stats | Jitter buffer stats for incoming media streams |
decodable_drops_old | H264 decodable frames dropped count, per stream |
incomplete_drops_old | H264 incomplete frames dropped count, per stream |
decodable_drops_reset | H264 decodable frames dropped before a new decoding point count, per stream |
incomplete_drops_reset | H264 incomplete frames dropped before a new decoding point count, per stream |
decodable_drops_pli | H264 decodable frames dropped on PLI receiving count, per stream |
incomplete_drops_pli | H264 incomplete frames dropped on PLI receiving count, per stream |
data_packets_with_empty_payload | Data packets with empty payload sent to test a channel quality when TWCC is enabled count, per stream |
h264_depacketizer_stats | H264 depacketizer stats |
missed_h264_units | Missed H264 units count, per stream |
rtp_audio_player_stats | RTP audio player stats |
dropped_audio_data_packets | Audio packets dropped before passing them to server engine, per stream |
custom_stats | Parameters collected with custom script |
Every parameter can be requested separately, for example:
Code Block | ||
---|---|---|
| ||
http://test.flashphoner.com:8081/?action=stat¶ms=native_resources |
Load and resources usage information of the WCS server, combined by groups, can be requested by the group name (connections_stats, streams_stats, port_stats, call_stats, degraded_streams_stats, system_stats, core_stats are available)
Code Block | ||
---|---|---|
| ||
http://test.flashphoner.com:8081/?action=stat&groups=call_stats |
You can include several group names in the request
Code Block | ||
---|---|---|
| ||
http://test.flashphoner.com:8081/?action=stat&groups=call_stats,connections_stats |
System parameters can be excluded from full report using the following parameter in flashphoner.properties file:
Code Block | ||
---|---|---|
| ||
parse_system_stats=false |
Statistics output formats
By default, statistics is shown as plain text
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
-----Connection Stats----- connections=1 connections_rtmfp=0 connections_websocket=1 -----Port Stats----- ports_media_free=495 ports_media_busy=4 ports_media_quarantine=0 -----Stream Stats----- corestreams_javawebrtc_threadsin=711 corestreams_javawebrtc_committedMemoryout=31270174721 corestreams_javawebsocket_freePhysicalMemorySizeout=698204160 corestreams_javartmfp_archin=amd640 corestreams_javartmfp_availableProcessorsout=20 corestreams_javartmp_freeSwapSpaceSizein=10443694080 corestreams_javartmp_maxFileDescriptorCountout=200000 core_java_open_file_descriptors=206 core_java_cpu_usage=39.88 core_java_totalPhysicalMemorySize=1927303168 core_java_totalSwapSpaceSize=1073737728 core_java_uptime=102191streams_viewers=test/1 streams_rtsp_in=0 streams_rtsp_out=0 streams_rtmp_client_out=0 streams_play_rate=0 streams_stop_rate=0 -----CallNative StatsResources----- sipnative_processed_calls=0 sip_calls_state=established/0;trying/0;ringing/0;ring/0;ring_media/0;hold/0;busy/0;finish/0;session_progress/0;pending/0;failed/0 sip_calls=0 sip_calls_established=0 sip_calls_in=0 sip_calls_out=0 sip_calls_per_second=0.00resources=140537376620832,NENC:H264/FFMPEG,1859;140537236444800,FFDecoderNative:H264/FFMPEG,39404417 native_resources.audio_codecs=0 native_resources.audio_resamplers=0 native_resources.video_transcoders=0 native_resources.video_decoders=1 native_resources.video_encoders=1 native_resources.writers=0 -----SipCore Stats----- sipcore_java_registeredthreads=0 -----Recording Stats----- recording_sessions=0 -----System Stats----- system_java_cpu_usage=75.00 system_java_load_average=0.87 -----Network Stats (Mbit/s)----- global_bandwidth_in=0.000 global_bandwidth_out=0.00071 core_java_committedMemory=3127017472 core_java_freePhysicalMemorySize=69820416 core_java_arch=amd64 core_java_availableProcessors=2 core_java_freeSwapSpaceSize=1044369408 core_java_maxFileDescriptorCount=20000 core_java_open_file_descriptors=206 core_java_cpu_usage=39.88 core_java_totalPhysicalMemorySize=1927303168 core_java_totalSwapSpaceSize=1073737728 core_java_uptime=102191 -----VersionCall infoStats----- wcssip_processed_version=5.2.416-32aab7dc90527bfe2ffb4711090fe75c6613a2bb wcs_client_version=0.5.28.2753-9ba78705727d3ee9d75c1c10f488d21cd00a601c -----Errors info-----calls=0 sip_calls_state=established/0;trying/0;ringing/0;ring/0;ring_media/0;hold/0;busy/0;finish/0;session_progress/0;pending/0;failed/0 sip_calls=0 sip_calls_established=0 sip_calls_in=0 sip_calls_out=0 sip_calls_per_second=0.00 -----Sip Stats----- sip_registered=0 -----DegradedRecording streamsStats----- degraded_streams= degraded_streams_percent=0 |
...
recording_sessions=0
-----System Stats-----
system_java_cpu_usage=75.00
system_java_load_average=0.87
-----Network Stats (Mbit/s)-----
global_bandwidth_in=0.000
global_bandwidth_out=0.000
-----Version info-----
wcs_version=5.2.416-32aab7dc90527bfe2ffb4711090fe75c6613a2bb
wcs_client_version=0.5.28.2753-9ba78705727d3ee9d75c1c10f488d21cd00a601c
-----Errors info-----
-----Degraded streams-----
degraded_streams=
degraded_streams_percent=0
|
Since build 5.2.416, JSON and Prometheus formats are supported. To get JSON statistics, set the format in request URI
...
Since build 5.2.992 it is possible to get video recording statistics, including HLS segments recording to a disk, for example
Code Block | ||
---|---|---|
| ||
-----Recording Stats----- recording_sessions=10 recording_threads=48 recording_thread_min_writers=21 recording_thread_max_writers=32 recording_thread_average_writers=31 recording_writers_list=17cb0c0895c8f5d3/1;cb7e90f53881dab6/1;d06bbc41545e59b8/1;8eba221954e7a01c/1;104f63c406de077d/1;3da56751b71fa871/1;474e59e9193aa3bb/1;1088894d91219b12/1;b95459b9c314201c/1;b6a574016f5241be/1 recording_writers_with_max_queue=17cb0c0895c8f5d3/23;104f63c4545e59b8/3;474e59e9/2;b95459b9/1;06de077d/4;b71fa871/5;193aa3bb/5;91219b12/1;c314201c/7;6f5241be/3 recording_writers_with_min_queue=d06bbc41/0;8eba2219/2;474e59e9/2;b95459b9/195c8f5d3/3;545e59b8/3;06de077d/4;b71fa871/5;193aa3bb/5;91219b12/1;c314201c/7;6f5241be/3 recording_min_writers_queue=01 recording_average_writers_queue=24 recording_max_writers_queue=37 |
The following are displayed:
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "recording_stats": { "recording_sessions": "10", "recording_threads": "48", "recording_thread_min_writers": "21", "recording_thread_max_writers": "32", "recording_thread_average_writers": "31", "recording_writers_list": [ "17cb0c0895c8f5d3", "cb7e90f53881dab6", "d06bbc41545e59b8", "8eba221954e7a01c", "104f63c406de077d", "3da56751b71fa871", "474e59e9193aa3bb", "1088894d91219b12", "b95459b9c314201c", "b6a574016f5241be" ], ... "recording_full_info": [ { "threadId": 97109, "writersCount": 32, "writersInfousageCounter": [2, { "queueSize": 3, "streams"writersInfo": [ "17cb0c08" ] }, { "queueSize": 0, "streams": [ "cb7e90f595c8f5d3" ] }, { "queueSize": 20, "streams": [ "d06bbc413881dab6" ] } ] }, ... ] } } |
Where:
...
...
]
}
} |
Where:
- queueSize - current recording data queue size
- streams - recording streams list
- threadId - CPU thread Id which is writing data to disk
- writersCount - writers count per CPU thread
- usageCount - CPU thread usages count
Logging statistics
Since build 5.2.1210 it is possible to get logging statistics, that may be useful on the server under high load
Code Block | ||||
---|---|---|---|---|
| ||||
curl -s 'http://localhost:8081/?action=stat&groups=log_stats' |
Logging statistics collection may increase CPU load if there are many media streams on the server. Therefore since build 5.2.1252 logging statistics collection is disabled by default, and may be enabled with the following parameter
Code Block | ||
---|---|---|
| ||
log_metrics_stats=true |
The following parameters are measured
Code Block | ||
---|---|---|
| ||
-----Logger info-----
log_msg_per_sec=0.30
log_mbit_per_sec=0.00 |
- log_msg_per_sec - logging messages count (all logs) per second
- log_mbit_per_sec - logging volume (all logs) in megabits per second
For example, statistics is buffered on 10 seconds interval. If there is no any message logged during this interval, the statistic values are cleaned. The interval can be set by the following parameter
Code Block | ||
---|---|---|
| ||
log_metrics_time_buffer=10000 |
The interval cannot be set less than 1 second, the default value will be used in such case.
Incoming stream statistics
Since build 5.2.1257 a certain incoming stream statistics can be collected
Code Block | ||||
---|---|---|---|---|
| ||||
curl -s "http://localhost:8081/?action=stat&format=json&groups=transcoding_stats" | jq '.[].transcoding_video_full_info' |
The statistics is available in JSON format only and includes the following parameters
Code Block | ||||
---|---|---|---|---|
| ||||
{
"test": {
"codec": "H264",
"queueSize": 0,
"distributors": {
...
},
"minDeltaArrivalTime": 1,
"maxDeltaArrivalTime": 62,
...,
"streamDelay": 38
}
} |
- stream name
- codec - publishing codec
- queueSize - decoding queue size
- distributors - subscribers statistics
- minDeltaArrivalTime - a minimal time between two subsequent packets receiving, ms
- maxDeltaArrivalTime - a maximal time between two subsequent packets receiving, ms
- streamDelay - stream delay, ms
Packets receiving time statistics is collected on time interval set by the following parameter in milliseconds
Code Block | ||
---|---|---|
| ||
media_processor_incoming_stat_window=30000 |
TCP channel statistics
Since build 5.2.1883 a TCP channel level statistics may be collected
Code Block | ||||
---|---|---|---|---|
| ||||
curl -s "http://localhost:8081/?action=stat&groups=tcp_channel_stats" |
The statistics contains the following parameters
Code Block | ||
---|---|---|
| ||
-----Tcp channel info-----
channels_not_writable= |
- channels_not_writable - TCP channels not writable events count, per mediasession
The statistics is displayed for currently published streams only.
Jitter buffer stats for incoming media streams
Since build 5.2.1883 a jitter buffer statistics may be collected
Code Block | ||||
---|---|---|---|---|
| ||||
curl -s "http://localhost:8081/?action=stat&groups=jitter_buffer_stats" |
The statistics contains the following parameters
Code Block | ||
---|---|---|
| ||
-----JitterBuffer info-----
decodable_drops_old=
incomplete_drops_old=f49f8cb0-dc52-11ee-81df-51ad589334c0/7;
decodable_drops_reset=
incomplete_drops_reset=
decodable_drops_pli=
incomplete_drops_pli=f49f8cb0-dc52-11ee-81df-51ad589334c0/10;
data_packets_with_empty_payload= |
- decodable_drops_old - H264 decodable frames dropped count, per mediasession
- incomplete_drops_old - H264 incomplete frames dropped count, per mediasession
- decodable_drops_reset - H264 decodable frames dropped before a new decoding point count, per mediasession
- incomplete_drops_reset - H264 incomplete frames dropped before a new decoding point count, per mediasession
- decodable_drops_pli - H264 decodable frames dropped on PLI receiving count, per mediasession
- incomplete_drops_pli - H264 incomplete frames dropped on PLI receiving count, per mediasession
- data_packets_with_empty_payload - data packets with empty payload sent to test a channel quality when TWCC is enabled count, per mediasession
The statistics is displayed for currently published streams only.
H264 depacketizer statistics
Since build 5.2.1883 an H264 depacketizer statistics may be collected
Code Block | ||||
---|---|---|---|---|
| ||||
curl -s "http://localhost:8081/?action=stat&groups=h264_depacketizer_stats" |
The statistics contains the following parameters
Code Block | ||
---|---|---|
| ||
-----H264 RTPDepacketizer info-----
missed_h264_units=f49f8cb0-dc52-11ee-81df-51ad589334c0/443; |
- missed_h264_units - missed H264 units count, per mediasession
The statistics is displayed for currently published streams only.
RTP audio player statistics
Since build 5.2.1923 an RTP audio player statistics may be collected
Code Block | ||||
---|---|---|---|---|
| ||||
curl -s "http://localhost:8081/?action=stat&groups=rtp_audio_player_stats" |
The statistics contains the following parameters
Code Block | ||
---|---|---|
| ||
-----RtpAudioPlayer info-----
dropped_audio_data_packets= |
- dropped_audio_data_packets - audio packets dropped before passing them to server engine, per mediasession
The statistics is displayed for currently published streams only.