Overview

It may be necessary to collect stream, client connections and CDN events data while managing a big number of WCS servers, to debug a streaming problems. In fact, the information that logged on every server, should be collected at one point. Note that logging itself is minimized in production use, to prevent server disk excessive load.

To collect such big amount of data, time series databases are good choice. Since build 5.2.774, the Remote Event Logging System (RELS) based on open source time series DB ClickHouse can be used to collect server logs.

Architecture

Every WCS server sends logging data to ClickHouse DB independently using JDBC-driver and HTTP connection. To optimize ClickHouse server load, the data are buffered and sent by time interval batch

Data table description

The logging data are collected to ClickHouse tables listed below. To speed up, an integer event identifiers are written to the tables. There is a textual string dictionary decribing all the events for each table to display human readable selection results.

Connections data (table ConnectionEvent)

Streams data (table StreamEvent)

CDN data (table CDNEvent)

Stream metrics data (table MediaSessionEvent)

Since build 5.2.1896 it is possible to collect a certain stream metrics data

HLS stream data (table HlsStreamEvent)

Since build 5.2.1917 it is possible to collect a certain HLS stream events data

HLS segments data (table HlsSegmenterEvent)

Since build 5.2.1917 it is possible to collect a certain HLS stream segments data

HLS subscribers data (table HlsClientEvent)

Since build 5.2.1929 it is possible to collect a certain HLS stream subscribers data. Since build 5.2.2041 some fields were added

Mixer metrics data (table MixerEvent)

Since build 5.2.1923 it is possible to collect a certain mixer metrics data

Incoming audio recovery data (table AudioRecoveryEvent)

Since build 5.2.1969 it is possible to collect a certain stream audio recovery data

RTMP incoming streams buffer metrics data (table RtmpInBufferEvent)

Since build 5.2.1978 it is possible to collect a certain incoming RTMP stream buffer metrics data

REST hooks data (table RestHooksEvent)

Since build 5.2.2005 it is possible to collect REST hooks data

RTP audio and video synchronization changing events (table RtpSynchronizationEvent)

Since build 5.2.2017 it is possible to collect RTP audio and video synchronization changing events data for stream published

Packets sent to a player events (table PlayerPacketEvent)

Since build 5.2.2024 it is possible to collect packets sent to a player events data

Configuration

ClickHouse installation and setup

Server requirement

• CPU from 4 physical cores, frequency from 3 GHz, for example Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
• RAM from 32 Gb
• HDD from 2 Tb

ClickHouse installation from rpm package (CentOS, Red Hat etc)

1. Add the official repository

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo

2. Install ClickHouse

sudo yum install -y clickhouse-server clickhouse-client

3. Launch ClickHouse

sudo systemctl enable clickhouse-server
sudo systemctl start clickhouse-server

ClickHouse installation from deb package (Debian, Ubuntu etc)

1. Add the official repository

sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update

2. Install ClickHouse

sudo apt-get install -y clickhouse-server clickhouse-client

3. Launch ClickHouse

sudo systemctl enable clickhouse-server
sudo systemctl start clickhouse-server

ClickHouse configuration for WCS builds before 5.2.1999

1. Uncomment the following string in /etc/clickhouse-server/config.xml file to listen all the server network interfaces

    <listen_host>::</listen_host>

2. Set the following parameter for default user in /etc/clickhouse-server/users.xml file to temporary allow users management

            <access_management>1</access_management>

3. Restart ClickHouse

systemctl restart clickhouse-server

4. Create wcs database and tables

cat wcs_clickhouse.sql | clickhouse-client -mn

CREATE DATABASE IF NOT EXISTS wcs;

DROP DICTIONARY IF EXISTS wcs.DictionaryStreamEvents;
 
DROP DICTIONARY IF EXISTS wcs.DictionaryConnectionEvents;
 
DROP DICTIONARY IF EXISTS wcs.DictionaryCDNEvents;

DROP DICTIONARY IF EXISTS wcs.DictionaryHlsStreamEventType;

DROP DICTIONARY IF EXISTS wcs.DictionaryHlsStreamEventSeverity;
 
DROP DICTIONARY IF EXISTS wcs.DictionaryMixerEvents;

DROP DICTIONARY IF EXISTS wcs.DictionaryBufferStateTypes;
 
DROP TABLE IF EXISTS wcs.StreamEvent;
 
DROP TABLE IF EXISTS wcs.ConnectionEvent;
 
DROP TABLE IF EXISTS wcs.CDNEvent;
 
DROP TABLE IF EXISTS wcs.StreamEventTypes;
 
DROP TABLE IF EXISTS wcs.ConnectionEventTypes;
 
DROP TABLE IF EXISTS wcs.CDNEventTypes;

DROP TABLE IF EXISTS wcs.MediaSessionEvents;

DROP TABLE IF EXISTS wcs.HlsStreamEvents;

DROP TABLE IF EXISTS wcs.HlsSegmenterEvents;

DROP TABLE IF EXISTS wcs.HlsStreamEventSeverity;

DROP TABLE IF EXISTS wcs.HlsStreamEventType;

DROP TABLE IF EXISTS wcs.HlsClientEvents;

DROP TABLE IF EXISTS wcs.MixerEvent;

DROP TABLE IF EXISTS wcs.MixerEventTypes;

DROP TABLE IF EXISTS wcs.RtmpInBufferEvent;

DROP TABLE IF EXISTS wcs.AudioRecoveryEvent;

DROP TABLE IF EXISTS wcs.BufferStateTypes;

CREATE TABLE wcs.ConnectionEventTypes
(
    `id` UInt32,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;
 
INSERT INTO wcs.ConnectionEventTypes VALUES (0, 'CONNECTED'), (1, 'DISCONNECTED');
 
CREATE TABLE wcs.StreamEventTypes
(
    `id` UInt32,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;
 
INSERT INTO wcs.StreamEventTypes VALUES (0,'CREATED'),(1,'LOCAL_SDP_CREATED'),(2,'REMOTE_SDP_RECEIVED'),(3,'ICE_STARTED'),(4,'ICE_COMPLETE'),(5,'DTLS_STARTED'),(6,'DTLS_COMPLETE'),(7,'INITIALIZED'),(8,'DISPOSING'),(9,'DISPOSED'),(10,'AUDIO_RECEIVED'),(11,'VIDEO_RECEIVED'),(12,'VIDEO_KFRAME_RECEIVED'),(13,'AUDIO_RTCP_RECEIVED'),(14,'VIDEO_RTCP_RECEIVED'),(15,'RESOLUTION_RECEIVED'),(16,'VIDEO_ENCODER_CREATED'),(17,'AUDIO_ENCODER_CREATED'),(18,'VIDEO_ENCODER_DISPOSED'),(19,'AUDIO_ENCODER_DISPOSED'),(20,'TERMINATED'),(21,'AUDIO_SENT'),(22,'VIDEO_SENT'),(23,'VIDEO_JITTER_BUFFER_STALL'),(24,'SENT_PLI'),(25,'RECEIVED_PLI'),(26,'SYNC_BUFFER_FULL'),(27,'SYNC_FORCE_FAILED'),(28,'SYNC_SHIFT'),(29,'SYNC_DEVIATION'),(30,'VIDEO_STATS'),(31,'RECORD');
 
CREATE TABLE wcs.CDNEventTypes
(
    `id` UInt32,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;
 
INSERT INTO wcs.CDNEventTypes VALUES (0, 'STATE'), (1, 'CDN_STATE'), (2, 'VERSION'), (3, 'ACL_REFRESH'), (4, 'ACL_UPDATE');
 
CREATE DICTIONARY wcs.DictionaryStreamEvents (
    `id` UInt16,
    `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'StreamEventTypes'
))   
LAYOUT(FLAT())
LIFETIME(300);
 
CREATE DICTIONARY wcs.DictionaryConnectionEvents (
    `id` UInt16,
    `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'ConnectionEventTypes'
))   
LAYOUT(FLAT())
LIFETIME(300);
 
CREATE DICTIONARY wcs.DictionaryCDNEvents (
    `id` UInt16,
    `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'CDNEventTypes'
))   
LAYOUT(FLAT())
LIFETIME(300);
 
CREATE TABLE wcs.StreamEvent
(
    `timestamp` UInt64,
    `ip` IPv4,
    `sessionId` String,
    `mediaSessionId` String,
    `streamName` String,
    `eventType` UInt64,
    `eventPayload` String
)
ENGINE = MergeTree()
ORDER BY (sessionId, mediaSessionId, streamName)
SETTINGS index_granularity = 8192;
 
CREATE TABLE wcs.ConnectionEvent
(
    `timestamp` UInt64,
    `ip` IPv4,
    `sessionId` String,
    `eventType` UInt64,
    `eventPayload` String
)
ENGINE = MergeTree()
ORDER BY (timestamp, sessionId)
SETTINGS index_granularity = 8192;
 
CREATE TABLE wcs.CDNEvent
(
    `timestamp` UInt64,
    `ip` IPv4,
    `nodeId` String,
    `eventType` UInt64,
    `eventPayload` String
)
ENGINE = MergeTree()
ORDER BY (nodeId, eventType)
SETTINGS index_granularity = 8192;

CREATE TABLE wcs.MediaSessionEvents
(
    `timestamp` UInt64,
    `ip` IPv4,
    `mediaSessionId` String,
    `streamName` String,
    `videoProfileId` UInt32,
    `videoWidth` UInt32,
    `videoHeight` UInt32,
    `videoFrameRate` UInt32,
    `videoKframes` UInt64,
    `videoPframes` UInt64,
    `videoBframes` UInt64,
    `videoRate` UInt64,
    `audioRate` UInt64,
    `videoSyncTime` UInt64,
    `audioSyncTime` UInt64,
    `videoTimestamp` UInt64,
    `audioTimestamp` UInt64,
    `lastKeyFrameSyncTime` UInt64,
    `sendNACK` UInt64,
    `recvNACK` UInt64,
    `videoFramesLost` UInt64,
    `audioPacketsLost` UInt64,
    `audioPlaybackSpeed` Float32,
    `videoPlaybackSpeed` Float32
)
ENGINE = MergeTree()
ORDER BY (mediaSessionId, streamName)
SETTINGS index_granularity = 8192;

CREATE TABLE wcs.HlsSegmenterEvents
(
    `timestamp` UInt64,
    `ip` IPv4,
    `streamId` String,
    `variantName` String,
    `segmentId` String,
    `segmentStartPts` UInt64,
    `videoStartPts` UInt64,
    `audioStartPts` UInt64,
    `videoWidth` UInt32,
    `videoHeight` UInt32,
    `videoFrameCount` UInt32,
    `audioPacketCount` UInt32,
    `segmentDuration` UInt64,
    `independent` Bool,
    `gap` Bool,
    `discontinuity` Bool,
    `segmentInterval` UInt64,
    `partial` Bool,
    `playbackSpeed` Float32
)
ENGINE = MergeTree()
ORDER BY (streamId)
SETTINGS index_granularity = 8192;

CREATE TABLE wcs.HlsStreamEventSeverity
(
    `id` UInt8,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;

INSERT INTO wcs.HlsStreamEventSeverity VALUES (0, 'INFO'), (1, 'WARNING'), (2, 'ERROR');

CREATE TABLE wcs.HlsStreamEventType
(
    `id` UInt16,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;

INSERT INTO wcs.HlsStreamEventType
VALUES (0, 'PLAYBACK_SPEED'), (1, 'FPS_CHANGED'), (2, 'GAP'), (3, 'RESOLUTION_CHANGED'), (4, 'DISCONTINUITY'), (5, 'TASK_SKIPPED'), (6, 'NO_KYE_FRAME'), (7, 'NO_VIDEO'), (8, 'NO_AUDIO'), (9, 'SEGMENT_INTERVAL'), (10, 'OTHER');

CREATE DICTIONARY wcs.DictionaryHlsStreamEventSeverity
(
   `id` UInt8,
   `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'HlsStreamEventSeverity'
))
LAYOUT(FLAT())
LIFETIME(300);

CREATE DICTIONARY wcs.DictionaryHlsStreamEventType
(
   `id` UInt16,
   `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'HlsStreamEventType'
))
LAYOUT(FLAT())
LIFETIME(300);

CREATE TABLE wcs.HlsStreamEvents
(
    `timestamp` UInt64,
    `ip` IPv4,
    `severity` UInt8,
    `messageType` UInt16,
    `streamId` String,
    `variantName` String,
    `segmentId` String,
    `message` String
)
ENGINE = MergeTree()
ORDER BY (streamId)
SETTINGS index_granularity = 8192; 

CREATE TABLE wcs.HlsClientEvents
(
    `creationTime` UInt64,
    `responseTime` UInt64,
    `streamId` String,
    `variantName` String,
    `uri` String,
    `localIp` IPv4,
    `remoteIp` IPv4,
    `remotePort` UInt32,
    `userAgent` String,
    `httpStatus` UInt32,
    `clientId` UInt64
)
ENGINE = MergeTree()
ORDER BY (creationTime)
SETTINGS index_granularity = 8192;

CREATE TABLE wcs.MixerEvent
(
    `timestamp` UInt64,
    `mixerMediaSessionId` String,
    `mixerStreamName` String,
    `mediaSessionId` String,
    `streamName` String,
    `mixerAverageTickTimeInMs` Int64,
    `audioMixerSync` Int64,
    `videoMixerSync` Int64,
    `nextAudioDataTime` Int64,
    `nextVideoDataTime` Int64,
    `audioBuffered` Int64,
    `videoBuffered` Int64,
    `audioDropsCounter` Int64,
    `audioDropsSizeInBytes` Int64,
    `videoDropsCounter` Int64,
    `videoDropsSizeInBytes` Int64,
    `videoFps` Int64,
    `audioRate` DOUBLE,
    `videoRate` DOUBLE,
    `eventType` UInt32,
    `eventPayload` String
)
ENGINE = MergeTree()
ORDER BY timestamp
SETTINGS index_granularity = 8192;

CREATE TABLE wcs.MixerEventTypes
(
    `id` UInt32,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;

INSERT INTO wcs.MixerEventTypes VALUES (0, 'nullEvent'), (1, 'dropBallastAudio'), (2, 'dropBallastVideo'), (3, 'audioNotBuffered'), (4, 'videoNotBuffered'), (5, 'audioBufferExhausted'), (6, 'videoBufferExhausted'), (7, 'alignStreamFailed'), (8, 'alignStreamDropAudio'), (9, 'alignStreamDropVideo'), (10, 'rateOutOfBoundsAudio'), (11, 'rateOutOfBoundsVideo');

CREATE DICTIONARY wcs.DictionaryMixerEvents (
    `id` UInt16,
    `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'MixerEventTypes'
))
LAYOUT(FLAT())
LIFETIME(300);

CREATE TABLE wcs.RtmpInBufferEvent
(
    `timestamp` UInt64,
    `streamClockTime` UInt64,
    `mediaSessionId` String,
    `streamName` String,
    `nextAudioDataTime` Int64,
    `nextVideoDataTime` Int64,
    `audioBuffered` Int64,
    `videoBuffered` Int64,
    `maximumAllowedBuffer` Int64,
    `bufferingCounter` Int64,
    `lastAudioDataTime` Int64,
    `lastVideoDataTime` Int64,
    `bufferState` UInt32
)
ENGINE = MergeTree()
ORDER BY timestamp
SETTINGS index_granularity = 8192;

CREATE TABLE wcs.BufferStateTypes
(
    `id` UInt32,
    `type` String
)
ENGINE = MergeTree()
ORDER BY id
SETTINGS index_granularity = 8192;

INSERT INTO wcs.BufferStateTypes VALUES (0, 'BUFFERING'), (1, 'HOLD'), (2, 'TERMINATED'), (3, 'OVERFLOW'), (4, 'PASSTHROUGH');

CREATE DICTIONARY wcs.DictionaryBufferStateTypes (
    `id` UInt16,
    `type` String DEFAULT ''
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(
 host 'localhost'
 port 9000
 user 'default'
 password ''
 db 'wcs'
 table 'BufferStateTypes'
))
LAYOUT(FLAT())
LIFETIME(300);

CREATE TABLE wcs.AudioRecoveryEvent
(
    `timestamp` UInt64,
    `mediaSessionId` String,
    `type` UInt32,
    `rtpTimestamp` UInt64
)
ENGINE = MergeTree()
ORDER BY rtpTimestamp
SETTINGS index_granularity = 8192;

5. Create wcs user and grant permissions to all the tables in wcs database

cat wcs_clickhouse_users.sql | clickhouse-client -mn

CREATE USER IF NOT EXISTS wcs IDENTIFIED BY 'wcs';
GRANT ALL ON wcs.* TO wcs WITH GRANT OPTION;

6. Disable users management for default user by setting the following parameter in /etc/clickhouse-server/users.xml file

            <access_management>0</access_management>

7. Restart ClickHouse

systemctl restart clickhouse-server

ClickHouse configuration for WCS builds 5.2.1999 and newer

Since WCS build 5.2.1999 DB is created automatically when a certain WCS build is connection fron a certain IP address. All the datadases metadata are stored in the table WcsMetadata.metadata

SELECT *
FROM WcsMetadata.metadata

┌─dbName───────────────────────────────┬─ip────────────┬─hostname──────────────┬─versionNumber─┬─versionHash──────────────────────────────┐
│ test1flashphonercom_192168065_521999 │ 192.168.0.65  │ test1.flashphoner.com │ 5.2.1999      │ 93e39647113e0121dabc4283ef700814c355568f │
│ test2flashphonercom_192168039_521999 │ 192.168.0.39  │ test2.flashphoner.com │ 5.2.1999      │ 93e39647113e0121dabc4283ef700814c355568f │
└──────────────────────────────────────┴───────────────┴───────────────────────┴───────────────┴──────────────────────────────────────────┘

Therefore only the user should be created and privileges to create databases, tables and to insert data to the tables should be granted to set up ClickHouse.

1. Uncomment the following string in /etc/clickhouse-server/config.xml file to listen all the server network interfaces 

    <listen_host>::</listen_host>

2. Set the following parameter for default user in /etc/clickhouse-server/users.xml file to temporary allow users management

            <access_management>1</access_management>
            <named_collection_control>1</named_collection_control>
            <show_named_collections>1</show_named_collections>
            <show_named_collections_secrets>1</show_named_collections_secrets>

3. Restart ClickHouse

systemctl restart clickhouse-server

4. Create wcs user and grant the privileges needed

cat wcs_clickhouse_users.sql | clickhouse-client -mn

CREATE USER IF NOT EXISTS wcs IDENTIFIED BY 'wcs';
SET allow_introspection_functions = 1;
GRANT ALL ON *.* TO wcs WITH GRANT OPTION;

5. Disable users management for default user by setting the following parameter in /etc/clickhouse-server/users.xml file

            <access_management>0</access_management>

6. Restart ClickHouse

systemctl restart clickhouse-server

WCS configuration

Data logging to ClickHouse is enabled by the following list of the data to collect

rels_enabled=CONNECTION,STREAM,CDN,MEDIA_SESSION

The following data types are available:

ClickHouse connection setup in WCS builds before 5.2.1999

ClickHouse server, database address and protocol are set by the following parameters

rels_client_type=HTTP
rels_database_address=http://clickhouseserver:8123/wcs?user=wcs&password=wcs

HTTP protocol is recommended and used by default. But it can be switched to JDBC driver if needed

rels_client_type=JDBC
rels_database_address=jdbc:clickhouse://clickhouseserver:8123/wcs?user=wcs&password=wcs

ClickHouse connection setup in WCS builds 5.2.1999 and newer

ClickHouse server, user and protocol are set by the following parameters

rels_client_type=HTTP
rels_database_address=clickhouseserver:8123
rels_database_properties=user=wcs&password=wcs

HTTP protocol is recommended and used by default. But it can be switched to JDBC driver if needed

rels_client_type=JDBC

Stop data logging without WCS server restart

Data logging can be stopped without WCS restart if necessary. To do this:

1. Disable data logging in server settings

rels_enabled=

2. Reload settings using CLI command

reload node-settings

ClickHouse server address changing without WCS server restart

ClickHouse server address can be changed without WCS restart. To do this:

1. Change address in server settings

rels_database_address=jdbc:clickhouse://newclickhouseserver:8123/wcs?user=wcs&password=wcs

2. Disable data logging in server settings

#rels_enabled=CONNECTION,STREAM,CDN,MEDIA_SESSION
rels_enabled=

3. Reload settings using CLI command

reload node-settings

4. Enable data logging in server settings

rels_enabled=CONNECTION,STREAM,CDN,MEDIA_SESSION

5. Reload settings using CLI command

reload node-settings

ClickHouse client thread pool size

Since build 5.2.2005 ClickHouse thread pool size may be set up. A dynamic pool of 2 threads is used by default

rels_database_thread_pool_size=2

If the parameter is set to 0

rels_database_thread_pool_size=0

ClickHouse client will use the static pool size depending on CPU cores count

(CPU count * 2) + 1

Outgoing data compression

Since build 5.2.2005 outgoing data may be compressed. By default, data compression is disabled

rels_enable_compression=false

Data compression may be enabled for huge data amount to reduce a traffic between WCS and ClickHouse servers, but in this case a ClickHouse client issue may occur, and some large string data may be lost.

Data collection management using REST API

The data of CONNECTION, STREAM, CDN, HLS_STREAM, REST_HOOKS types are always collected, for all the client sessions and all the streams. All the other types are collected on demand only because a data amount may be too big.

The data collection for a certain stream is enabled by REST API.

A REST-query should be HTTP/HTTPS POST request as follows:

• HTTP: http://streaming.flashphoner.com:8081/rest-api/rels/startup
• HTTPS: https://streaming.flashphoner.com:8444/rest-api/rels/startup

Where:

• streaming.flashphoner.com - WCS server address
• 8081 - the standard REST / HTTP port of the WCS server
• 8444 - the standard HTTPS port
• rest-api - the required part of the URL
• /rels/startup - REST-method to use

REST methods and responses

/rels/startup

Start data collectio of a certain type for the certain streams

Request example

POST /rest-api/rels/startup HTTP/1.1
Host: localhost:8081
Content-Type: application/json

{
    "mediaSession": {
        "frequency":100,
        "ids": [
            "d7d6b6e4-b137-461a-8a32-d6abf2b8666e",
            "39cbf770-128a-11ef-b839-d1a1f53f8bd2"
        ] 
    }
}

Response example

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json

Return codes

/rels/find_all

Get the sent data statistics

Request example

POST /rest-api/rels/find_all HTTP/1.1
Host: localhost:8081
Content-Type: application/json

Response example

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json

[
  {
    "type": "CONNECTION",
    "sentBytes": 1989,
    "bitrateKbps": 0,
    "sentEvents": 6,
    "queueEvents": 0
  },
  {
    "type": "STREAM",
    "sentBytes": 70766,
    "bitrateKbps": 0,
    "sentEvents": 73,
    "queueEvents": 1
  },
  {
    "type": "CDN",
    "sentBytes": 0,
    "bitrateKbps": 0,
    "sentEvents": 0,
    "queueEvents": 0
  },
  {
    "type": "MEDIA_SESSION",
    "ids": [
      "d7d6b6e4-b137-461a-8a32-d6abf2b8666e",
      "39cbf770-128a-11ef-b839-d1a1f53f8bd2"
    ],
    "sentBytes": 205794,
    "bitrateKbps": 143,
    "sentEvents": 999,
    "queueEvents": 119
  }
]

Return codes

/rels/terminate

Stop data collection of the certain type for the certain streams

Request example

POST /rest-api/rels/terminate HTTP/1.1
Host: localhost:8081
Content-Type: application/json

{
    "mediaSession": {
        "ids": [
            "d7d6b6e4-b137-461a-8a32-d6abf2b8666e"
        ] 
    }
}

Response example

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json

Return codes

/rels/terminate_all

Stop the certain types data collection for all the streams

Request example

POST /rest-api/rels/terminate_all HTTP/1.1
Host: localhost:8081
Content-Type: application/json

{
    "types": [
        "MEDIA_SESSION"
    ]
}

Response example

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json

{
  "MEDIA_SESSION": [
    "39cbf770-128a-11ef-b839-d1a1f53f8bd2"
  ]
}

Return codes

Parameters

Automatic conditional data collection

Since build 5.2.2005 it is possible to collect some data types automatically by a certain conditions. If a media stream published on the server conforms the conditions, a data of type defined will be collected for the stream.

Conditional collection of the following data types is supported:

• MEDIA_SESSION
• MIXER
• HLS_SEGMENTER

• HLS_CLIENT (since build 5.2.2091)

• RTMP_IN_BUFFER (since build 5.2.2091)
• RTP_SYNCHRONIZATION (since build 5.2.2091)

The conditions should be defined in /usr/local/FlashphonerWebCallServer/conf/rels_trap.json file

{
    "mediaSession":
    [
        {
            "ips": ["127.0.0.1"],
            "frequency": "1000ms"
        },
        {
            "ips": ["192.168.0.0/24", "192.168.2.0/24"],
            "streamName": ".*\\-screen"
        },
        {
            "ips": ["192.168.0.101/32"],
            "streamName": "test.*",
            "frequency": "100th"
        },
        {
            "streamName": "test.*",
            "mediaType": ["publish", "play"]
        }
    ],

    "mixer":
    [
        {
            "name": "conference\\-.*",
            "streamName": "user.*",
            "frequency": "1000ms"
        }
    ],

    "hlsSegmenter":
    [
        {
            "streamId": "test.*"
        }
    ],
 
    "hlsClient":[
        {
            "streamId": "test.*"
        }
    ],
    
    "rtpSynchronization":[
        {
            "ips":["192.168.0.0/24"],
            "streamName": "test.*"
        }
    ],

    "rtmpInBuffer":[
        {
            "ips":["192.168.0.0/24"],
            "streamName": "test.*"
        }
    ]
}

Where:

• mediaSession - filter block to collect MEDIA_SESSION data
  • ips - client IP addresses list
  • streamName - stream name mask regular expression
  • type - session type: "publish", "play"
  • frequency - data collection frequency for the stream conforming the filter
• mixer - filter block to collect MIXER data
  • name - mixer name mask regular expression
  • streamName - incoming stream name mask regular expression
  • frequency - data collection frequency for the mixer conforming the filter
• hlsSegmenter - filter block to collect HLS_SEGMENTER data
  
  • streamId - HLS stream name mask regular expression
• hlsClient - filter block to collect HLS_CLIENT data
  • streamId - HLS stream name mask regular expression
• rtpSynchronization - filter block to collect RTP_SYNCHRONIZATION data
  • ips - client IP addresses list
  • streamName - stream name mask regular expression
• rtmpInBuffer - filter block to collect RTMP_IN_BUFFER data
  • ips - client IP addresses list
  • streamName - stream name mask regular expression

Publishers IP address list to filter stream publishers may contain an exact address or address mask in CIDR form

{
    "mediaSession":
    [
        {
            "ips": ["127.0.0.1", "192.168.0.0/24"],
            ...
        },
        ...
    ],
    ...
}

If both IP addresses list and stream name mask are set, then the data will be collected only for the stream with name and client address conforming the conditions

{
    "mediaSession":
    [
        ...,
        {
            "ips": ["192.168.2.0/24"],
            "streamName": ".*\\-screen",
            ...
        },
        ...
    ],
    ...
}

If both mixer name and incoming stream name masks are set in MIXER filter, then the data will be collected for the mixer only after the stream with the name conforming the stream name mask set in the conditions

{
    ...,

    "mixer":
    [
        {
            "name": "conference\\-.*",
            "streamName": "user.*",
            "frequency": "1000ms"
        }
    ],

    ...
}

The frequency parameter sets data collection frequency for the cyclically collected types:

• "frequency": "100ms" - data are collected every 100 ms
• "frequency": "100th" - data are collected when every 100 media packets received

A changes in the file rels_trap.json will be applied without WCS restarting but affects only new published streams, mixers and HLS streams.

Data retrieving from DB

Logging data can be retrieved using SQL queries in CliskHouse client

SQL queries examples for WCS builds before 5.2.1999

select timestamp,ip,sessionId,mediaSessionId,streamName,dictGetString('wcs.DictionaryStreamEvents','type', eventType) as eventType from wcs.StreamEvent where streamName = 'test'

select timestamp,ip,sessionId,dictGetString('wcs.DictionaryConnectionEvents','type', eventType) as eventType from wcs.ConnectionEvent

select timestamp,ip,nodeId,dictGetString('wcs.DictionaryCDNEvents','type', eventType) as eventType,eventPayload from wcs.CDNEvent

SQL queries examples for WCS builds 5.2.1999 and newer

select timestamp,ip,sessionId,mediaSessionId,streamName,dictGetString('wcs.DictionaryStreamEvents','type', eventType) as eventType from test1flashphonercom_192168065_521999.StreamEvent where streamName = 'test'

select timestamp,ip,sessionId,dictGetString('wcs.DictionaryConnectionEvents','type', eventType) as eventType from test1flashphonercom_192168065_521999.ConnectionEvent

select timestamp,ip,nodeId,dictGetString('wcs.DictionaryCDNEvents','type', eventType) as eventType,eventPayload from test1flashphonercom_192168065_521999.CDNEvent

Data recording to a files

It may not be possible to deploy a ClickHouse server in a production environment. That's why since build 5.2.2058 a collected data may be recorded to a text files which may be imported to ClickHouse for further analysis.

Files recording is enabled by default with the following parameter

rels_client_type=FILE

The data are recorded to the /usr/local/FlashphonerWebCallServer/logs/rels folder. A subfolder named by host name, server IP address and WCS version is created in the folder. In their turn, a text files per every data table are placed to the subfolder

/usr/local/FlashphonerWebCallServer/logs/rels/
└── testflashphonercom_192168039_522060
    ├── ConnectionEvent.csv
    ├── ConnectionEvent.json
    ├── StreamEvent.csv
    └── StreamEvent.json

Every data table consists of two text files:

• json file contains a data types description to import the table to ClickHouse DB
• csv file contains a data in comma separated form

Importing the data recorded to ClickHouse DB

WCS installation is required to import the data recorded to ClickHouse DB (a staging server for example). There is the tool for data importing shipped with WCS which may be launched as follows

/usr/local/FlashphonerWebCallServer/tools/rels_import.sh --dbpath=/usr/local/FlashphonerWebCallServer/logs/rels/testflashphonercom_192168039_522060

Where

• dbpath - path to a data files folder to import

The tool creates a database named as data files folder, testflashphonercom_192168039_522060 for the example above.

The tool uses the following parameters from /usr/local/FlashphonerWebCallServer/conf/flashphoner.properties file to connect to ClickHouse instance

rels_client_type=HTTP
rels_database_address=database.flashphoner.com:8123
rels_database_properties=user=wcs&password=wcs

For Clickhouse 24 and newer, data deduplication token feature will be used to prevent an excessive import of identical data rows if the tool is called repeatedly for the same files.