Overview
It may be necessary to collect stream, client connections and CDN events data while managing a big number of WCS servers, to debug a streaming problems. In fact, the information that logged on every server, should be collected at one point. Note that logging itself is minimized in production use, to prevent server disk excessive load.
To collect such big amount of data, time series databases are good choice. Since build 5.2.774, the Remote Event Logging System (RELS) based on open source time series DB ClickHouse can be used to collect server logs.
Architecture
Every WCS server sends logging data to ClickHouse DB independently using JDBC-driver and HTTP connection. To optimize ClickHouse server load, the data are buffered and sent by time interval batch
Data table description
The logging data are collected to ClickHouse tables listed below. To speed up, an integer event identifiers are written to the tables. There is a textual string dictionary decribing all the events for each table to display human readable selection results.
Connections data (table ConnectionEvent)
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
sessionId | String | Session Id |
eventType | UInt64 | Event type Id |
eventPayload | String | Event payload |
Event types (table ConnectionEventTypes)
Field | Type | Description |
---|---|---|
id | UInt32 | Event type id |
type | String | Event type description |
Events dictionary (table DictionaryConnectionEvents)
Field | Type | Description |
---|---|---|
id | UInt64 | Event type id |
type | String | Event type description |
Streams data (table StreamEvent)
Field | Type | Description |
---|---|---|
timestamp | UInt64 | Time stamp |
ip | IPv4 | Server address |
sessionId | String | Session Id |
mediaSessionId | String | Media session Id |
streamName | String | Stream name |
eventType | UInt64 | Event type Id |
eventPayload | String | Event payload |
Event types (table StreamEventTypes)
Field | Type | Description |
---|---|---|
id | UInt32 | Event type id |
type | String | Event type description |
Events dictionary (table DictionaryStreamEvents)
Field | Type | Description |
---|---|---|
id | UInt64 | Event type id |
type | String | Event type description |
CDN data (table CDNEvent)
timestamp | UInt64 | Time stamp |
---|---|---|
ip | IPv4 | Server address |
ip | IPv4 | Адрес сервера |
nodeId | String | Node Id (IP address string) |
eventType | UInt64 | Event type Id |
eventPayload | String | Event payload |
Event types (table CDNEventTypes)
Field | Type | Description |
---|---|---|
id | UInt32 | Event type id |
type | String | Event type description |
Events dictionary (table DictionaryCDNEvents)
Field | Type | Description |
---|---|---|
id | UInt64 | Event type id |
type | String | Event type description |
Configuration
ClickHouse installation and setup
Server requirement
- CPU from 4 physical cores, frequency from 3 GHz, for example Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
- RAM from 32 Gb
- HDD from 2 Tb
ClickHouse installation in CentOS 7
1. Create repository file altinity_clickhouse.repo in /etc/yum.repos.d folder
sudo cat <<EOF > /etc/yum.repos.d/altinity_clickhouse.repo [altinity_clickhouse] name=altinity_clickhouse baseurl=https://packagecloud.io/altinity/clickhouse/el/7/$basearch repo_gpgcheck=1 gpgcheck=0 enabled=1 gpgkey=https://packagecloud.io/altinity/clickhouse/gpgkey sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=300 [altinity_clickhouse-source] name=altinity_clickhouse-source baseurl=https://packagecloud.io/altinity/clickhouse/el/7/SRPMS repo_gpgcheck=1 gpgcheck=0 enabled=1 gpgkey=https://packagecloud.io/altinity/clickhouse/gpgkey sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=300 EOF
2. Add repository
sudo yum -q makecache -y --enablerepo='altinity_clickhouse'
3. Install ClickHouse
sudo yum install -y clickhouse-server clickhouse-client
4. Launch ClickHouse
systemctl start clickhouse-server
ClickHouse configuration
1. Uncomment the following string in /etc/clickhouse-server/config.xml file to listen all the server network interfaces
<listen_host>::</listen_host>
2. Set the following parameter for default user in /etc/clickhouse-server/users.xml file to temporary allow users managent
<access_management>1</access_management>
3. Restart ClickHouse
systemctl restart clickhouse-server
4. Create wcs database and tables
cat wcs_clickhouse.sql | clickhouse-client -mn
5. Create wcs user and grant permissions to all the tables in wcs database
cat wcs_clickhouse_users.sql | clickhouse-client -mn
6. Disable users management for default user by setting the following parameter in /etc/clickhouse-server/users.xml file
<access_management>0</access_management>
7. Restart ClickHouse
systemctl restart clickhouse-server
WCS configuration
Data logging to ClickHouse is enabled by the following parameter
rels_enabled=true
ClickHouse server and database address is set by the following parameter
rels_database_address=jdbc:clickhouse://clickhouseserver:8123/wcs?user=wcs&password=wcs
Stop data logging without WCS server restart
Data logging can be stopped without WCS restart if necessary. To do this:
1. Disable data logging in server settings
rels_enabled=false
2. Reload settings using CLI command
reload node-settings
ClickHouse server address changing without WCS server restart
ClickHouse server address can be changed without WCS restart. To do this:
1. Change address in server settings
rels_database_address=jdbc:clickhouse://newclickhouseserver:8123/wcs?user=wcs&password=wcs
2. Disable data logging in server settings
rels_enabled=false
3. Reload settings using CLI command
reload node-settings
4. Enable data logging in server settings
rels_enabled=true
5. Reload settings using CLI command
reload node-settings
Data retrieving from DB
Logging data can be retrieved usin SQL queries in CliskHouse client
select timestamp,ip,sessionId,mediaSessionId,streamName,dictGetString('wcs.DictionaryStreamEvents','type', eventType) as eventType from wcs.StreamEvent where streamName = 'test'
select timestamp,ip,sessionId,dictGetString('wcs.DictionaryConnectionEvents','type', eventType) as eventType from wcs.ConnectionEvent
select timestamp,ip,nodeId,dictGetString('wcs.DictionaryCDNEvents','type', eventType) as eventType,eventPayload from wcs.CDNEvent