Overview
It may be necessary to collect stream, client connections and CDN events data while managing a big number of WCS servers, to debug a streaming problems. In fact, the information that logged on every server, should be collected at one point. Note that logging itself is minimized in production use, to prevent server disk excessive load.
To collect such big amount of data, time series databases are good choice. Since build 5.2.774, the Remote Event Logging System (RELS) based on open source time series DB ClickHouse can be used to collect server logs.
Architecture
Every WCS server sends logging data to ClickHouse DB independently using JDBC-driver and HTTP connection. To optimize ClickHouse server load, the data are buffered and sent by time interval batch
Data table description
The logging data are collected to ClickHouse tables listed below. To speed up, an integer event identifiers are written to the tables. There is a textual string dictionary decribing all the events for each table to display human readable selection results.
Connections data (table ConnectionEvent)
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
sessionId | String | Session Id |
eventType | UInt64 | Event type Id |
eventPayload | String | Event payload |
Streams data (table StreamEvent)
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
sessionId | String | Session Id |
mediaSessionId | String | Media session Id |
streamName | String | Stream name |
eventType | UInt64 | Event type Id |
eventPayload | String | Event payload |
CDN data (table CDNEvent)
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
nodeId | String | Node Id (IP address string) |
eventType | UInt64 | Event type Id |
eventPayload | String | Event payload |
Stream metrics data (table MediaSessionEvents)
Since build 5.2.1896 it is possible to collect a certain stream metrics data
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
mediaSessionId | String | Media session Id |
streamName | String | Stream name |
videoProfileId | UInt32 | Video profile Id |
videoWidth | UInt32 | Pictiure height |
videoHeight | UInt32 | Picture width |
videoFrameRate | UInt32 | Video frame rate per second |
videoKframes | UInt64 | Key frames number (I-frame) |
videoPframes | UInt64 | P-frames number |
videoBframes | UInt64 | B-frames number |
videoRate | UInt64 | Video bitrate, bps |
audioRate | UInt64 | Audio bitrate, bps |
videoSyncTime | UInt64 | Video synchronization value |
audioSyncTime | UInt64 | Audio synchronization value |
videoTimestamp | UInt64 | Video packet timestamp |
audioTimestamp | UInt64 | Audio packet timestamp |
lastKeyFrameSyncTime | UInt64 | Video synchronization value from the last key frame |
sendNACK | UInt64 | NACK sent count |
recvNACK | UInt64 | NACK received count |
videoFramesLost | UInt64 | Lost video frames count |
audioPacketsLost | UInt64 | Lost audio frames count |
audioPlaybackSpeed | Float32 | Audio publishing/playback speed |
videoPlaybackSpeed | Float32 | Video publishing/playback speed |
HLS stream data (table HlsStreamEvents)
Since build 5.2.1917 it is possible to collect a certain HLS stream events data
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
severity | UInt8 | Event severity level: INFO, WARNING, ERROR |
messageType | UInt16 | Event type |
streamId | String | HLS stream Id |
variantName | String | Quality variant name |
segmentId | String | HLS segment Id |
message | String | Logged event description |
HLS segments data (table HlsSegmenterEvents)
Since build 5.2.1917 it is possible to collect a certain HLS stream segments data
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
streamId | String | HLS stream Id |
variantName | String | Quality variant name |
segmentId | String | Segment Id |
segmentStartPts | UInt64 | Segment start PTS |
videoStartPts | UInt64 | Video start PTS |
audioStartPts | UInt64 | Audio start PTS |
videoWidth | UInt32 | Picture width |
videoHeight | UInt32 | Picture height |
videoFrameCount | UInt32 | Video frames count |
audioPacketCount | UInt32 | Audio packets count |
segmentDuration | UInt64 | Segment duration, ms |
independent | Bool | Independent segment |
gap | Bool | GAP segment |
discontinuity | Bool | DISCONTINUTY segment |
segmentInterval | UInt64 | Segments interval, ms |
partial | Bool | Partial segment |
playbackSpeed | Float32 | Playback speed |
HLS subscribers data (table HlsClientEvents)
Since build 5.2.1929 it is possible to collect a certain HLS stream subscribers data
Field | Type | Description |
---|---|---|
creationTime | UInt64 | Subscriber creation time (a first request) |
responseTime | UInt64 | Subscriber response time |
streamId | String | HLS stream Id |
variantName | String | Quality variant name |
uri | String | Playlist URI |
localIp | IPv4 | Server address |
remoteIp | IPv4 | Client address |
remotePort | UInt32 | Client port |
userAgent | String | User-Agent data |
httpStatus | UInt32 | Response status |
clientId | UInt64 | Client session Id |
Mixer metrics data (table MixerEvent)
Since build 5.2.1923 it is possible to collect a certain mixer metrics data
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
mixerMediaSessionId | String | Mixer media session Id |
mixerStreamName | String | Mixer output stream name |
mediaSessionId | String | Incoming stream media session Id |
streamName | String | Incoming stream name |
mixerAverageTickTimeInMs | Int64 | Mixer tick average duration, ms |
audioMixerSync | Int64 | Mixer output stream audio synchronization value |
videoMixerSync | Int64 | Mixer output stream audio synchronization value |
nextAudioDataTime | Int64 | Next audio packet time |
nextVideoDataTime | Int64 | Next video packet time |
audioBuffered | Int64 | Incoming stream audio packets buffered count |
videoBuffered | Int64 | Incoming stream video packets buffered count |
audioDropsCounter | Int64 | Incoming stream audio packets dropped count |
audioDropsSizeInBytes | Int64 | Incoming stream audio dropped data in bytes |
videoDropsCounter | Int64 | Incoming stream video packets dropped count |
videoDropsSizeInBytes | Int64 | Incoming stream video dropped data in bytes |
videoFps | Int64 | Mixer output stream framerate per second |
audioRate | DOUBLE | Mixer output stream audio bitrate, bps |
videoRate | DOUBLE | Mixer output stream audio bitrate, bps |
eventType | UInt32 | Mixer event type |
eventPayload | String | Event description |
Incoming audio recovery data (table AudioRecoveryEvent)
Since build 5.2.1969 it is possible to collect a certain stream audio recovery data
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
mediaSessionId | String | Media session Id |
type | UInt32 | Event type |
rtpTimestamp | UInt64 | RTP stream timestamp |
RTMP incoming streams buffer metrics data (table RtmpInBufferEvent)
Since build 5.2.1978 it is possible to collect a certain incoming RTMP stream buffer metrics data
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
streamClockTime | UInt64 | Stream clock time |
mediaSessionId | String | Media session Id |
streamName | String | Stream name |
nextAudioDataTime | Int64 | Next audio data packet time |
nextVideoDataTime | Int64 | Next video data packet time |
audioBuffered | Int64 | Audio packets buffered count |
videoBuffered | Int64 | Video packets buffered count |
maximumAllowedBuffer | Int64 | Maximum packet allowed in the buffer |
bufferingCounter | Int64 | Bufferings counter |
lastAudioDataTime | Int64 | Last audio data packet time |
lastVideoDataTime | Int64 | Last video data packet time |
bufferState | UInt32 | Buffer state |
Configuration
ClickHouse installation and setup
Server requirement
- CPU from 4 physical cores, frequency from 3 GHz, for example Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
- RAM from 32 Gb
- HDD from 2 Tb
ClickHouse installation in CentOS 7
1. Create repository file altinity_clickhouse.repo in /etc/yum.repos.d folder
sudo cat <<EOF > /etc/yum.repos.d/altinity_clickhouse.repo [altinity_clickhouse] name=altinity_clickhouse baseurl=https://packagecloud.io/altinity/clickhouse/el/7/$basearch repo_gpgcheck=1 gpgcheck=0 enabled=1 gpgkey=https://packagecloud.io/altinity/clickhouse/gpgkey sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=300 [altinity_clickhouse-source] name=altinity_clickhouse-source baseurl=https://packagecloud.io/altinity/clickhouse/el/7/SRPMS repo_gpgcheck=1 gpgcheck=0 enabled=1 gpgkey=https://packagecloud.io/altinity/clickhouse/gpgkey sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=300 EOF
2. Add repository
sudo yum -q makecache -y --enablerepo='altinity_clickhouse'
3. Install ClickHouse
sudo yum install -y clickhouse-server clickhouse-client
4. Launch ClickHouse
systemctl start clickhouse-server
ClickHouse configuration
1. Uncomment the following string in /etc/clickhouse-server/config.xml file to listen all the server network interfaces
<listen_host>::</listen_host>
2. Set the following parameter for default user in /etc/clickhouse-server/users.xml file to temporary allow users managent
<access_management>1</access_management>
3. Restart ClickHouse
systemctl restart clickhouse-server
4. Create wcs database and tables
cat wcs_clickhouse.sql | clickhouse-client -mn
5. Create wcs user and grant permissions to all the tables in wcs database
cat wcs_clickhouse_users.sql | clickhouse-client -mn
6. Disable users management for default user by setting the following parameter in /etc/clickhouse-server/users.xml file
<access_management>0</access_management>
7. Restart ClickHouse
systemctl restart clickhouse-server
WCS configuration
Data logging to ClickHouse is enabled by the following list of the data to collect
rels_enabled=CONNECTION,STREAM,CDN,MEDIA_SESSION
The following data types are available:
CONNECTION, STREAM, CDN, MEDIA_SESSION, HLS_SEGMENTER, HLS_STREAM, HLS_CLIENT, MIXER, AUDIO_RECOVERY, RTMP_IN_BUFFER
Type | Description |
---|---|
CONNECTION | Client session events |
STREAM | Stream events |
CDN | CDN events |
MEDIA_SESSION | Stream metrics |
HLS_SEGMENTER | HLS stream segment metrics |
HLS_STREAM | HLS stream events |
HLS_CLIENT | HLS clients statistics |
MIXER | Mixer events and metrics |
AUDIO_RECOVERY | Audio packets recovery events |
RTMP_IN_BUFFER | RTMP incoming streams buffer metrics |
ClickHouse server, database address and protocol are set by the following parameters
rels_client_type=HTTP rels_database_address=http://clickhouseserver:8123/wcs?user=wcs&password=wcs
HTTP protocol is recommended and used by default. But it can be switched to JDBC driver if needed
rels_client_type=JDBC rels_database_address=jdbc:clickhouse://clickhouseserver:8123/wcs?user=wcs&password=wcs
Stop data logging without WCS server restart
Data logging can be stopped without WCS restart if necessary. To do this:
1. Disable data logging in server settings
rels_enabled=false
2. Reload settings using CLI command
reload node-settings
ClickHouse server address changing without WCS server restart
ClickHouse server address can be changed without WCS restart. To do this:
1. Change address in server settings
rels_database_address=jdbc:clickhouse://newclickhouseserver:8123/wcs?user=wcs&password=wcs
2. Disable data logging in server settings
rels_enabled=false
3. Reload settings using CLI command
reload node-settings
4. Enable data logging in server settings
rels_enabled=true
5. Reload settings using CLI command
reload node-settings
Data retrieving from DB
Logging data can be retrieved usin SQL queries in CliskHouse client
select timestamp,ip,sessionId,mediaSessionId,streamName,dictGetString('wcs.DictionaryStreamEvents','type', eventType) as eventType from wcs.StreamEvent where streamName = 'test'
select timestamp,ip,sessionId,dictGetString('wcs.DictionaryConnectionEvents','type', eventType) as eventType from wcs.ConnectionEvent
select timestamp,ip,nodeId,dictGetString('wcs.DictionaryCDNEvents','type', eventType) as eventType,eventPayload from wcs.CDNEvent