Metrics collection configuration and management

Configuration

WCS clients configuration

The metrics description to collect and ingest point configuration for a clients are in JSON file /usr/local/FlashphonerWebCallServer/conf/webrtc_metrics.json at WCS server side. When a client connects to the server, it sends this configuration to the client.

The default configuration file contains a setup for Websocket ingest point, i.e. the WCS server accepts the metrics batches:

{
 "collect": "off",
 "sampling": 1000,
 "batchSize": 30,
 "compression": "none,gzip",
 "types": {
    ...
 }
}

Where

• collect sets the client to send (on) or not to send (off) metrics to the ingest point. Metrics are not sent by default.
• sampling is a minimal period to collect metrics in milliseconds. Usually, the default one collection per second ("sampling": 1000) is good enough for metrics consistency and client performance.
• batchSize is a minimal metrics batches quantity to send to the server. By default, client should try to send every 30 metrics batches, so metrics are sent to the ingest point every 30 seconds
• compression is a comma separated compression types list. By default, client should try to compress metrics batch using gzip, or should send uncompressed metrics batch if compression fails or is not supported.
• types is a metrics description object (see below)

HTTP/HTTPS ingest point configuration requires an additional fields

{
 "ingestPoint": "https://demo.flashphoner.com:8481",
 "keyStore": {
  "path": "/usr/local/FlashphonerWebCallServer/conf/jwt-hs256.p12",
  "type": "PKCS12",
  "password": "storepass",
  "alias": "sqajwt",
  "accessExpiresIn": 60
 },
 "collect": "on",
 ...
}

Where

• ingestPoint is the ingest point URL. Usually, should be HTTPS because the most browsers do not allow to send HTTP requests from HTTPS page, and WebRTC works on HTTPS only page in the most browsers
• keyStore is the authentication key store parameters. This key is used to generate a bearer authentication header for the client to send every metrics batch

The metrics description object contains all the possible WebRTC statistics metrics arranged by groups. A client should check if all the metrics from the description object are available and send an actual supported metrics list to the server. Then, client should send metrics values periodically, grouped in the batches. Server, in its turn, writes all the data received to Clickhouse DB. If some digital metric value is not available, server writes 0. If some string metric value is not available, server writes empty string "".

Metrics description object

{
    ...,
    "types": {
        "codec": {
            "metrics": "timestamp,payloadType,mimeType,clockRate,channels"
        },
        "inbound-rtp": {
            "metrics": "timestamp,ssrc,kind,packetsReceived,packetsReceivedWithEct1,packetsReceivedWithCe,packetsReportedAsLost,packetsReportedAsLostButRecovered,packetsLost,jitter,trackIdentifier,mid,remoteId,framesDecoded,keyFramesDecoded,framesRendered,framesDropped,frameWidth,frameHeight,framesPerSecond,qpSum,totalDecodeTime,totalInterFrameDelay,totalSquaredInterFrameDelay,pauseCount,totalPausesDuration,freezeCount,totalFreezesDuration,lastPacketReceivedTimestamp,headerBytesReceived,packetsDiscarded,fecBytesReceived,fecPacketsReceived,fecPacketsDiscarded,bytesReceived,nackCount,firCount,pliCount,totalProcessingDelay,estimatedPlayoutTimestamp,jitterBufferDelay,jitterBufferTargetDelay,jitterBufferEmittedCount,jitterBufferMinimumDelay,totalSamplesReceived,concealedSamples,silentConcealedSamples,concealmentEvents,insertedSamplesForDeceleration,removedSamplesForAcceleration,audioLevel,totalAudioEnergy,totalSamplesDuration,framesReceived,decoderImplementation,playoutId,powerEfficientDecoder,framesAssembledFromMultiplePackets,totalAssemblyTime,retransmittedPacketsReceived,retransmittedBytesReceived,rtxSsrc,fecSsrc,totalCorruptionProbability,totalSquaredCorruptionProbability,corruptionMeasurements"
        },
        "outbound-rtp": {
            "metrics": "timestamp,ssrc,kind,packetsSent,bytesSent,mid,mediaSourceId,remoteId,rid,encodingIndex,headerBytesSent,retransmittedPacketsSent,retransmittedBytesSent,rtxSsrc,targetBitrate,totalEncodedBytesTarget,frameWidth,frameHeight,framesPerSecond,framesSent,hugeFramesSent,framesEncoded,keyFramesEncoded,qpSum,psnrSum,psnrMeasurements,totalEncodeTime,totalPacketSendDelay,qualityLimitationReason,qualityLimitationDurations,qualityLimitationResolutionChanges,nackCount,firCount,pliCount,encoderImplementation,powerEfficientEncoder,active,scalabilityMode,packetsSentWithEct1"
        },
        "remote-inbound-rtp": {
            "metrics": "timestamp,ssrc,kind,packetsReceived,packetsReceivedWithEct1,packetsReceivedWithCe,packetsReportedAsLost,packetsReportedAsLostButRecovered,packetsLost,jitter,localId,roundTripTime,totalRoundTripTime,fractionLost,roundTripTimeMeasurements,packetsWithBleachedEct1Marking"
        },
        "remote-outbound-rtp": {
            "metrics": "timestamp,ssrc,kind,packetsSent,bytesSent,localId,remoteTimestamp,reportsSent,roundTripTime,totalRoundTripTime,roundTripTimeMeasurements"
        },
        "media-source": {
            "metrics": "timestamp,trackIdentifier,kind,audioLevel,totalAudioEnergy,totalSamplesDuration,echoReturnLoss,echoReturnLossEnhancement,width,height,frames,framesPerSecond"
        }
    }
}

WCS ingest point configuration

WCS may be the an ingestion point for metrics. This extends the centralized server data logging to ClickHouse DB feature:

rels_enabled=RTC_CODEC_EVENT,RTC_INBOUND_RTP_EVENT,RTC_OUTBOUND_RTP_EVENT,RTC_REMOTE_INBOUND_RTP_EVENT,RTC_REMOTE_OUTBOUND_RTP_EVENT,RTC_MEDIA_SOURCE_EVENT,RTC_MEDIA_PLAYOUT_EVENT,RTC_PEER_CONNECTION_EVENT,RTC_DATA_CHANNEL_EVENT,RTC_TRANSPORT_EVENT,RTC_CANDIDATE_PAIR_EVENT,RTC_LOCAL_CANDIDATE_EVENT,RTC_REMOTE_CANDIDATE_EVENT,RTC_CERTIFICATE_EVENT,SESSION_METADATA
rels_client_type=JDBC
rels_database_address=clickhouseserver:8123
rels_database_properties=user=wcs&password=wcs

Additional settings are required for HTTP/HTTPS ingest point

rtc_metrics_ingest_enabled=true
rtc_metrics_ingest_ip=0.0.0.0
rtc_metrics_ingest_http_port=8480
rtc_metrics_ingest_https_port=8481
rtc_metrics_resend_token_interval=30
rtc_metrics_resend_token_enabled=true

The access token refresh interval in seconds rtc_metrics_resend_token_interval must be less or equal to accessExpiresIn value to prevent early token expiration.

Centralized metrics collection from multiple WCS nodes

A multiple WCS nodes may send metrics data to the same Clickhouse DB server. This allows to balance HTTPS metrics collection, but in this case a client sessions metadata should also be stored centrally. The Redis database is used to store sessions metadata.

Redis connection setup

The following parameter enables session metadata storing in Redis database:

rtc_metrics_ingest_meta_repository=REDIS

Redis connection should be set up in /usr/local/FlashphonerWebCallServer/conf/redis-client.yaml configuration file, for example

# FLASHPHONER COMPANY CONFIDENTIAL
# __________________________________________
#
# [2009] - [2026] Flashphoner company
# All Rights Reserved.
#
# NOTICE:  All information contained herein is, and remains
# the property of Flashphoner company and its suppliers,
# if any.  The intellectual and technical concepts contained
# herein are proprietary to Flashphoner company
# and its suppliers and may be covered by U.S. and Foreign Patents,
# patents in process, and are protected by trade secret or copyright law.
# Dissemination of this information or reproduction of this material
# is strictly forbidden unless prior written permission is obtained
# from Flashphoner company.

########################################################################################################################################################
# This configuration is used to establish a connection and work with the Redis-Server, which stores the necessary metadata for processing RTC metrics.##
########################################################################################################################################################
# WARNING! if the connection fails, the server stops accepting metrics. See server logs to ensure a successful connection ##############################
########################################################################################################################################################
# WARNING! To apply the new configuration, the server must be restarted.################################################################################
########################################################################################################################################################

## Use this block to configure connection to Redis-Server
connection:
  # Host of Redis-Server (required)
  host: 127.0.0.1
  # Port of Redis-Server (required)
  port: 6379

  # Password of Redis-Server
  #password: admin

  ## Connection timeout in seconds. This value is used for:
  ## 1. Maximum wait time when establishing initial connection
  ## 2. Delay between reconnection
  ## The default is `30` seconds.
  #timeout: 30

## Use this block to configure key behaviour
#key:
  ## Responsible for the key's lifetime. The lifetime is reset to its initial value after each use to receive metrics.
  ## If it is not specified, the lifetime is unlimited or is regulated by the Redis-Server configuration.
  #ttl: 10000

## Use this block to configure SSL
#ssl:
  ## SSL Provider. By default, the value is `JDK` and only it is supported
  #provider: JDK

  ## Truststore credentials
  #truststore:
    ## The path can be specified as absolute, otherwise it is relative ./conf
    ## Supported formats: PKCS12 or JKS
    #path: redis-certs/truststore.jks

    ## Truststore password
    #password: admin123

  ## Keystore credentials
  #keystore:

    ## The path can be specified as absolute, otherwise it is relative ./conf
    ## Supported formats: PKCS12
    #path: redis-certs/client.p12

    ## Keystore password
    #password: admin123

## Use this block to configure work with commands such as `set`, `expire`, `get`, `del` and other.
#commands:
  ## Maximum wait time in seconds when sending a command to the Redis-Server.
  ## By default its `30`
  #timeout: 30

  ## Try to reconnect if the command fails to complete and complete it
  ## By default its `true`
  #autoReconnect: true

  ## Cancel commands on reconnect failure
  ## By default its `false`
  #cancelCommandsOnReconnectFailure: false

Redis itself configuration file /etc/redis/redis.conf looks like this

bind 0.0.0.0
notify-keyspace-events KEA
port 6379
tls-port 6380
tls-cert-file /etc/ssl/domain/domain.crt
tls-key-file /etc/ssl/domain/priv.key
tls-ca-cert-file /etc/ssl/domain/ca.crt
tls-auth-clients yes

Clickhouse central database setup

By default, metrics are send to a separate databases, one per each WCS server. The following parameter sets the database name explicitly

rels_database=rels_db_central

This setting should be equal on all WCS servers.

The following parameter sets the Clickhouse database name for WCS handling metrics analyze requests /rest-api/stream_metrics_analyzer

sqa_database=rels_db_central

Ingest point logging configuration

WCS ingest point logs are written to the /usr/local/FlashphonerWebCallServer/logs/ingest_logs/ingest.log. The logging is set up in /usr/local/FlashphonerWebCallServer/conf/log4j.properties file:

log4j.logger.io.lettuce=INFO,ingestAppender
log4j.additivity.io.lettuce=false

log4j.logger.com.flashphoner.media.webrtc.metrics=INFO,ingestAppender
log4j.additivity.com.flashphoner.media.webrtc.metrics=false

log4j.logger.com.flashphoner.server.metrics=INFO, ingestAppender
log4j.additivity.com.flashphoner.server.metrics=false

log4j.appender.ingestAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.ingestAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.ingestAppender.File=${com.flashphoner.fms.AppHome}/logs/ingest_logs/ingest.log
log4j.appender.ingestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.ingestAppender.layout.ConversionPattern=%d{HH:mm:ss,SSS} %-5p %25.25c{1} - %t %m%n

Management

Metrics collection may be enabled or disabled for all the clients by REST API query /rest-api/v3/stream/rtc_metrics/update_all or for a certain client by REST API query /rest-api/v3/stream/rtc_metrics/update_batch. SFU metrics collection is managed by REST API /rest-api/v3/sfu/rtc_metrics/update_all and /rest-api/v3/sfu/rtc_metrics/update_batch queries respectively.