...

Servers in a CDN can be grouped together by geographical (location), technical (GPU using for transcoding) and anoter another basis. CDN group can be assigned to the server with the following parameter in flashphoner.properties file

...

profiles:
 
 -webrtc-144:
  audio:
    codec : opus
    rate : 48000
    channels : 2
  video:
    width : 256
    height : 144
    codecImpl : OPENH264

...

If picture width is not set or equal to 0, video will be scaled by height with aspect ratio preserving.

...

FFmpeg parameters management

When using FFmpeg FF codec implementation, encoding preset, profile and level can be set. For example, the following parameters

...

The settings above should be applyed applied to Transcoder nodes.

Transcoding profile management using REST API

The specific REST API is used to manage transcoding profiles on the fly, without server restart

REST query should be HTTP/HTTPS POST request as follows:

• HTTP: http://test.flashphoner.com:8081/rest-api/cdn/profile/print
• HTTPS: https://test.flashphoner.com:8444/rest-api/cdn/profile/print

Where:

• test.flashphoner.com - WCS server address
• 8081 - standard REST / HTTP port of WCS server
• 8444 - standard HTTPS port
• rest-api - URL mandatory part
• /cdn/profile/print - REST query used

REST queries should be sent to Edge server. Any profile changes are stored in memory and applied immediately, also they are written to cdn_profiles.yml file to use after server restart.

REST queries and response states

...

REST query

...

REST query body example

...

Response body example

...

Response states

...

Description

...

/cdn/profile/add

...

...

Multi channels audio support

Since build 5.2.773, audio transcoding by profile with certain number of channels (1 - mono, 2 - stereo) is supported

profiles:
 -240p:
  audio:
    codec : mpeg4-generic
    rate : 48000
    channels: 1
  video:
    height : 240
    bitrate : 300
    gop : 50
    codec : h264

A channels number in a profile should be equal to channels number in original audio track for subscriber on Edge server to play stereo sound when stream with stereo sound is published, i.e.

profiles:
 -240p:
  audio:
    codec : mpeg4-generic
    rate : 48000
    channels: 2

If profile codec, samplerate and channels number are equal to original audio track parameters, the track will be passed to Edge server without transcoding

Transcoding profile management using REST API

The specific REST API is used to manage transcoding profiles on the fly, without server restart

REST query should be HTTP/HTTPS POST request as follows:

• HTTP: http://test.flashphoner.com:8081/rest-api/cdn/profile/print
• HTTPS: https://test.flashphoner.com:8444/rest-api/cdn/profile/print

Where:

• test.flashphoner.com - WCS server address
• 8081 - standard REST / HTTP port of WCS server
• 8444 - standard HTTPS port
• rest-api - URL mandatory part
• /cdn/profile/print - REST query used

REST queries should be sent to Edge server. Any profile changes are stored in memory and applied immediately, also they are written to cdn_profiles.yml file to use after server restart.

REST queries and response states

{
"name": "-profile1",
"profile": {
        "audio": {
            "bitratetype": 0"audio", 
            "channelsbitrate": 0, 
            "codec": "opus", 
            "rate": 48000 
        }, 
        "video": {
            "type": "video", 
            "bitrate": 0, 
            "codec": "h264", 
            "codecImpl": "FF", 
            "fps": 30, 
            "gop": 90, 
            "height": 360, 
            "quality": 0, 
            "width": 640
        }
}
}

{
"name": "-profile1",
"profile": {
        "video": {
            "bitrate": 400, 
            "quality": 25
        }
}
}

    "-240p": {
        "audio": {
            "bitrate": 0, 
            "channels": 0, 
            "codec": "opus", 
            "rate": 48000, 
            "type": "audio"
        }, 
        "video": {
            "bitrate": 0, 
            "codec": "h264", 
            "codecImpl": "FF", 
            "fps": 25, 
            "gop": 50, 
            "height": 240, 
            "preset": "veryfast", 
            "quality": 0, 
            "type": "video", 
            "width": 0
        }
    }, 
    "-480p": {
        "audio": {
            "bitrate": 0, 
            "channels": 0, 
            "codec": "opus", 
            "rate": 48000, 
            "type": "audio"
        }, 
        "video": {
            "bitrate": 0, 
            "codec": "h264", 
            "codecImpl": "FF", 
            "fps": 25, 
            "gop": 50, 
            "height": 480, 
            "preset": "veryfast", 
            "quality": 0, 
            "type": "video", 
            "width": 0
        }
    }

{
"name": "-profile1"
}

...

-profile1

{
"audio": {
"type": "audio"
"bitrate": 0, 
"channels": 0, 
"codec": "opus", 
"rate": 48000, 
}, 
"video": {
"type": "video", 
"bitrate": 0, 
"codec": "h264", 
"codecImpl": "FF", 
"fps": 30, 
"gop": 90, 
"height": 360, 
"quality": 0, 
"width": 640
}

cdn_group_origin_to_transcoder_relation=true

Profile parameters should contain audio or video sections.

The mandatory profile parameters are:

• audio:
  • bitrate
  • codec
  • ratechannels
• video:
  • bitrate
  • codec
  • width
  • height
  • quality

...

 profile3:
  audio:
    codec : opus
    rate : 48000
  video:
    height : 180

Choosing a route for stream playback

CDN routes are based on the following periodic data sendings between CDN nodes:

• Origin sends published streams data to Transcoder and Edge;
• Transcoder sends pulled streams data to Edge in the same group;
• Edge sends nothing and is always an end point of a route.

A route for stream playback on Edge server is chosen as follows:

Image Removed

1. If transcoding profile is set on Edge server:

1.1. If there is the stream with such name on Transcoder node in the same group with Edge:

1.1.1. If the stream is already transcoded by this profile, Edge pulls the stream from Transcoder

1.1.2. If the stream is transcoded by another profile:

1.1.2.1. Stream will be transcoded by profile set

1.1.2.2. Edge will pull the stream from Transcoder

1.2. If Transcoder that belongs to the same group with Edge can pull the stream with such name from Origin:

1.2.1. Transcoder will pull the stream from Origin

1.2.2. The stream will be transcoded by the profile set.

1.2.3. Edge will pull the stream from Transcoder

1.3. In other cases, Edge pulls the stream, from Origin

2. If transcoding profile is not set on Edge server, Edge pulls the stream Origin

An example of streams translation via CDN with Transcoder nodes

Image Removed

Where

• stream1, stream2, stream3 – streams published to Origin server
• stream1-P1 – stream1 stream transcoded by settings profile P1
• stream2-P1 – stream1 stream transcoded by settings profile P2

Node groups and transcoding

By defaut, Edge will choose Transcoder in the same group to transcode the stream by profile. For example, if CDN has two segments grouped by location: in Europe and USA, then stream translation for the stream by one profile from every group looks as follows:

Image Removed

In this case, only one stream will be broadcast from Origin to each group, this decreases intergroup channels load, but stream will be transcoded in every group independantly.

This behaviour can be changes with the following parameter

cdn_group_origin_to_transcoder_relation=true

Then Transcoder in the same group with Origin will be chosen promarily to transcode the stream, and stream translatiion will look as follows:

Image Removed

In this case, the stream is transcoded one time for all location groups, but as many streams will be broadcast to each group as requested by Edges for playback. This inscreases intergroup channels load.

Getting stream routes information with REST API

To get an information about CDN stream routes REST API query /cdn/stream/show_routes is used

REST query should be HTTP/HTTPS POST request as follows:

• HTTP: http://test.flashphoner.com:8081/rest-api/cdn/stream/show_routes
• HTTPS: https://test.flashphoner.com:8444/rest-api/cdn/stream/show_routes

Where:

• test.flashphoner.com - WCS server address
• 8081 - a standard WCS REST / HTTP port
• 8444 - a standard WCS HTTPS port
• rest-api - mandatory part of URL
• /cdn/stream/show_routes - REST query used

REST queries and responses

...

REST query

...

REST query body example

...

REST response example

...

Response states

...

Description

...

/cdn/stream/show_routes

...

{
 "streamName":"test-webrtc-144"
}

...

...

Transcoder node tuning to work under high load

The server CPU load increases while a large number of streams are transcoded. If the server CPU does not manage to process all the streams frames, frame encoding queues take a much of server RAM (not JVM heap!). As a result, memory leak may occur and, therefore, server can stop working depending on memory allocation libraries used by server operating system.

To prevent this, the following tune settings are recommended to be used together:

• suspend incoming streams decoding when encoding queues are full
• use jemalloc memory allocation library

Incoming streams decoding suspending

Incoming streams decoding suspending allows to smooth (but not fully prevent) peak server loads, if this is set for transcoder group. The feature is enabled with the following parameter

streaming_video_decoder_wait_for_distributors=true

Encoding queue size to suspend streams decoding is set in frames with the following parameter

streaming_video_decoder_wait_for_distributors_max_queue_size=3

Timeout to wait for queue freeing is set in milliseconds

streaming_video_decoder_wait_for_distributors_timeout=33

Using jemalloc memory allocation library

By default, glibc memory allocation library is used in most cases. To prevent memory leaks under high load it is recommended to set Transcoder nodes to use jemalloc library as follows (Centos 7 for example):

1. Install build prerequisites

yum -y install autoconf libtool pkg-config g++ make cmake bzip

2. Download library source code

wget https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2

3. Unpack the archive

tar -xvjf jemalloc-5.2.1.tar.bz2

4. Build the library

cd jemalloc-5.2.1
./configure && make && make install

5. Comment the following line in /usr/local/FlashphonerWebCallServer/bin/setenv.sh file

export MALLOC_ARENA_MAX=4

and add the following line

export LD_PRELOAD=/usr/local/lib/libjemalloc.so

6. Switch decoder to OpenH264 in flashphoner.properties file

decoder_priority=OPENH264,FF

7. Restart WCS.

Lowering multithreaded encoding threshold

By default, multithreaded encoding is enabled for 720p and higher profiles. Sometimes it is necessary to lower this threshold to encode 480p streams also in two and more threads

video_encoder_second_thread_threshold=408950

Reducing memory allocations for decoding

To reduce memory allocations for decoding, since build 5.2.559 the ability to use buffer pools for decoded pictures is added. The feature can be enabled with the following parameter

decoder_buffer_pool=true

In this case, a pool of buffers will be allocated for every incoming picture resolution. When buffer is not needed any more, it is not deleted, but returned to pool. To get memory usage information, statistic collection should be enabled with the following parameter

decoder_buffer_pool_stats=true

Statistic is available by the following query (JSON format only)

curl -s 'http://localhost:8081/?action=stat&format=json&groups=decoder_buffer_pool_stats'

and grouped by pools

{
  "decoder_buffer_pool_stats": {
    "decoder_buffer_pool_info": {
      "decoder_buffer_pool_info_pools": {
        "

...

3072x1536": {
          "decoder_buffer_pool_width": 3072,
          "

...

decoder_buffer_pool_height": 1536,
          "decoder_buffer_pool_size_bytes": 84934656,
        

...

  "decoder_buffer_pool_leased": 6,
          "decoder_buffer_pool_allocated": 12
    

...

...

},
        "1280x720"

...

: {
          

...

"decoder_buffer_pool_width": 1280,
          "decoder_buffer_pool_height": 720,
          "decoder_buffer_pool_size_bytes": 688584704,
        

...

  "decoder_buffer_pool_leased": 0,
          "decoder_buffer_pool_allocated": 3605
        }
      },
  

...

200 – OK

500 – Internal Server Error

...

Show CDN stream routes

Parameters

...

Description

...

Example

...

Stream name (including transcoding profile if necessary)

...

test-webrtc-144

...

REQUESTED-PROFILE

...

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}

...

VIDEO{width=256, height=144, gop=null, fps=null, bitrate=0, codec='h264', codecImpl='OPENH264', quality=null}

...

1-PROFILE-192.168.1.220

...

2-STREAM-192.168.1.220

...

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}

...

VIDEO{width=320, height=180, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='FF', quality=null}

...

3-NEW-TRANSCODER-192.168.1.220

...

4-PROXY-192.168.1.219

...

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}

...

VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}

CDN routes checking examples

Let's look how to check some CDN routes build for stream playback.

For example we use three nodes CDN:

• 192.168.1.219 - Origin
• 192.168.1.220 - Transcoder
• 192.168.1.221 - Edge

WebRTC H264+opus (48 kHz, stereo) stream named test is published to Origin

Stream pulling from Origin without transcoding if publishing and playback profiles are equal

Stream named test is played on Edge by profile

 -webrtc-opus-video-proxy:
  audio:
    codec : opus
    rate : 48000
    channels : 2
  video:
    "decoder_buffer_pool_total_size_bytes": 773519360
    }
  }
}

Where

• decoder_buffer_pool_total_size_bytes - total memory size allocated to all the pools in bytes
• decoder_buffer_pool_width - pool picture width
• decoder_buffer_pool_height - pool picture height
• decoder_buffer_pool_size_bytes - memory size allocated to the pool in bytes
• decoder_buffer_pool_leased - buffers used count in the pool
• decoder_buffer_pool_allocated - buffers allocated ciunt in the pool

The memory allocated to buffers unused can be freed by /debug/decoder/shrink_buffer_provider_pools REST API query

Freeing the memory allocated to buffers unused

REST query should be HTTP/HTTPS POST request as follows:

• HTTP: http://test.flashphoner.com:8081/rest-api/debug/decoder/shrink_buffer_provider_pools
• HTTPS: https://test.flashphoner.com:8444/rest-api/debug/decoder/shrink_buffer_provider_pools

Where:

• test.flashphoner.com - WCS server address
• 8081 - standard REST / HTTP port of WCS server
• 8444 - standard HTTPS port
• rest-api - URL mandatory part
• /cdn/profile/print - REST query used

REST queries should be sent to Edge server.

Stream transcoding by two or more profiles with the same video parameters

A few transcoding profiles with the same video but different audio parametesr can be defined on Edge nodes, for example, to play a stream via HLS and WebRTC:

  -240p-HLS:
    audio:
      codec: mpeg4-generic
    video:
      height: 240
      bitrate: 300
      codec: h264
  -240p-WebRTC:
    audio:
      codec: opus
    video:
      height: 240
      bitrate: 300
      codec: h264

The response to this query

http://192.168.1.221:8081/rest-api/cdn/stream/show_routes
{
    "streamName": "test-webrtc-opus-video-proxy"
}

should be interpreted as follows:

1. Profile requested parameters:

    "REQUESTED-PROFILE": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='h264', codecImpl='null', quality=null}"
    ],

2. Stream is pulled from Origin:

    "1-PROXY-PROFILE-192.168.1.219": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}"
    ],

3. Stream can be pulled from Transcoder

    "2-NEW-TRANSCODER-192.168.1.220": [],

4. But is pulled from Origin because stream publishing parameters are equal to requested playback parameters:

    "3-PROXY-192.168.1.219": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}"
    ]

Stream pulling from Transcoder with audio transcoding

Stream named test is played on Edge by profile

 -webrtc-pcma-video-proxy:
  audio:
    codec : pcma
    rate : 8000
    channels : 1

The response to this query

http://192.168.1.221:8081/rest-api/cdn/stream/show_routes
{
    "streamName": "test-webrtc-pcma-video-proxy"
}

should be interpreted as follows:

1. Profile requested parameters:

    "REQUESTED-PROFILE": [
        "AUDIO{bitrate=0, codec='pcma', rate=8000, channels=1}"
    ],

2. Transcoding profile is created on Transcoder:

    "1-PROFILE-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='null', quality=null}",
        "AUDIO{bitrate=0, codec='PCMA', rate=8000, channels=1}"
    ],

3. Stream is pulled from Transcoder

    "2-STREAM-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='null', quality=null}",
        "AUDIO{bitrate=0, codec='PCMA', rate=8000, channels=1}"
    ],
    "3-NEW-TRANSCODER-192.168.1.220": [],

4. Stream publishing to Origin parameters:In this case, if the stream is transcoded at the same Transcoder node by those two profiles, only one video encoder and two audio encoders will be used.

Transcoding to higher resolutions prevention

Since build 5.2.607, stream transcoding to higher resolutions can be escaped. The feature can be enabled with the following parameter

cdn_strict_transcoding_boundaries=true

In this case, if a stream is published to Origin with resolution 640x480, and Edge server tries to request playback by 720p profile, Edge will receive an original stream directly from Origin without transcoding. Resolutions are compared by height.

Stream requesting by higher resolution profiles can be strictly forbidden with the following parameter

cdn_strict_transcoding_throws_exception=true

When, if a stream is published to Origin with resolution 640x480, and Edge server tries to request playback by 720p profile, Edge can not choose stream playback route and will return error to a client.

These parameters should be set on Edge server.

Known limits

1. If stream transcoding to higher resolutions is disabled, a stream height must be set for every profile.

Choosing a route for stream playback

CDN routes are based on the following periodic data sendings between CDN nodes:

• Origin sends published streams data to Transcoder and Edge;
• Transcoder sends pulled streams data to Edge in the same group;
• Edge sends nothing and is always an end point of a route.

A route for stream playback on Edge server is chosen as follows:

Image Added

1. If transcoding profile is set on Edge server:

1.1. If there is the stream with such name on Transcoder node in the same group with Edge:

1.1.1. If the stream is already transcoded by this profile, Edge pulls the stream from Transcoder

1.1.2. If the stream is transcoded by another profile:

1.1.2.1. Stream will be transcoded by profile set

1.1.2.2. Edge will pull the stream from Transcoder

1.2. If Transcoder that belongs to the same group with Edge can pull the stream with such name from Origin:

1.2.1. Transcoder will pull the stream from Origin

1.2.2. The stream will be transcoded by the profile set.

1.2.3. Edge will pull the stream from Transcoder

1.3. In other cases, Edge pulls the stream, from Origin

2. If transcoding profile is not set on Edge server, Edge pulls the stream Origin

An example of streams translation via CDN with Transcoder nodes

Image Added

Where

• stream1, stream2, stream3 – streams published to Origin server
• stream1-P1 – stream1 stream transcoded by settings profile P1
• stream2-P1 – stream1 stream transcoded by settings profile P2

Node groups and transcoding

By default, Edge will choose Transcoder in the same group to transcode a stream by profile. For example, if CDN has two segments grouped by location - in Europe and USA, then stream distribution to every group when transcoding by one profile looks as follows:

Image Added

In this case, only one stream will be broadcast from Origin to each group, this decreases intergroup channels load, but stream will be transcoded in every group independantly.

This behaviour can be changed with the following server configuration parameter

cdn_group_origin_to_transcoder_relation=true

This setting should be enabled on Transcoders and Edges.

Then primarily Transcoder in the same group as Origin will be chosen to transcode a stream, and stream distribution will look as follows:

Image Added

In this case, the stream is transcoded one time for all location groups, but as many streams will be broadcast to each group as requested by Edges for playback. This inscreases intergroup channels load.

Starting from version 5.2.418, in case Transcoder from the same group as Origin is unavailable, Transcoder from the group Edge is in will be chosen, or any other available Transcoder if Transcoder from the group of the Edge is also unavailable. So, Edge will be able to get transcoded stream even when there are no available transcoders in the groups of the Origin and the Edge.

The priority of CDN groups when choosing available Transcoder for transcoding of a new steam will be as follows:

1. First group specified on Origin (other groups Origin is in are not taken into account - priority goes to the group Edge is in)
2. Groups specified on Edge (if Transcoder from the first group is unavailable , then priority goes to the second etc.)
3. Other groups (equal priority)

However, if the requested stream is already being transcoded on another Transcoder - not the priority Transcoder for this Edge, - then the Edge will get the stream from that Transcoder, which is currently performing transcoding.

Image Added

Getting stream routes information with REST API

To get an information about CDN stream routes REST API query /cdn/stream/show_routes is used

REST query should be HTTP/HTTPS POST request as follows:

• HTTP: http://test.flashphoner.com:8081/rest-api/cdn/stream/show_routes
• HTTPS: https://test.flashphoner.com:8444/rest-api/cdn/stream/show_routes

Where:

• test.flashphoner.com - WCS server address
• 8081 - a standard WCS REST / HTTP port
• 8444 - a standard WCS HTTPS port
• rest-api - mandatory part of URL
• /cdn/stream/show_routes - REST query used

REST queries and responses

{
 "streamName":"test-webrtc-144"
}

{
    "

...

REQUESTED-PROFILE": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=

...

256, height=

...

144, gop=null, fps=null, bitrate=0, codec='

...

h264', codecImpl='OPENH264', quality=null}"
    ]

...

Stream pulling from Transcoder with video transcoding

Stream named test is played on Edge by profile

...

...

,
    "1-PROFILE-192.168.1.220": [
    

...

...

...

  "AUDIO{bitrate=0, codec='opus', rate

...

=48000, channels=2}",
  

...

...

...

 "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='null', quality=null}",
       

...

The response to this query

http://192.168.1.221:8081/rest-api/cdn/stream/show_routes
{
    "streamName": "test-webrtc-opus-vp8"
}

should be interpreted as follows:

1. Profile requested parameters:

...

...

 "VIDEO{width=320, height=180, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='FF', quality=null}",
        "VIDEO{width=256, height=144, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='OPENH264', quality=null}"
    ],
    "2-STREAM-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=

...

0, height=

...

0, gop=

...

null, fps=

...

null, bitrate=0, codec='

...

H264', codecImpl='

...

', quality=null

...

2. Transcoding profile is created on Transcoder:

...

...

}",
        "VIDEO{width=

...

320, height=

...

180, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='FF', quality=null}",
        "VIDEO{width=

...

256, height=

...

144, gop=

...

null, fps=

...

null, bitrate=0, codec='

...

H264', codecImpl='

...

OPENH264', quality=null}"
    ],

...

3. Stream is pulled from Transcoder

...

    "3-NEW-TRANSCODER-192.168.1.220": [],
    "

...

4-

...

PROXY-192.168.1.

...

219": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264

...

'

...

, codecImpl='

...

', quality=null}"
    ]

...

4. Stream publishing to Origin parameters:

...

...

}

Parameters

test-webrtc-144

REQUESTED-PROFILE

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}

...

VIDEO{width=

...

256, height=

...

144, gop=null, fps=null, bitrate=0, codec='

...

h264', codecImpl='OPENH264', quality=null}

...

...

Node load detection by CPU load

If there are some Transcoder nodes with the same profiles in CDN, those nodes load can be balanced  by CPU load. Maximum allowed load for certain Transcoder can be defined with the following parameter

cdn_node_load_average_threshold=1.0

This value sets average CPU load (JVM SystemLoadAverage parameter) to CPU cores avalable ratio, including hyperthreading cores. When this value is exceeded, the Transcoder node is excluded from route elections for new stteams and profiles. In this case the Transcoder node is displayed in the stream route PROFILE section for profiles the stream is already transcoded by.

Node current state (if node participates in new stream route elections) can be checked with REST API. A node in NEW_STREAMS_ALLOWED state can pull new streams to transcode, a node in GROUP_CONNECTION_ALLOWED state can only push the streams already transcoded.

This parameter should be set on Transcoder nodes. Also this parameter can be set on Origin nodes, in this case their load can be checked via REST API, and if the Origin node become GROUP_CONNECTION_ALLOWED then new streams can be published to another Origin server.

Transcoder node video encoders limiting

If necessary, maximum video encoders simultaneously used amount can be lemited for Transcoder nodes with the following parameter

cdn_transcoder_video_encoders_threshold=10000

When this amount is reached, the Transcoder node becomes GROUP_CONNECTION_ALLOWED. All the video encoders on server are included, for example, if one stream is transcoding by two profiles, 3 video encoders are used, including PNG encoder.

This parameter should be set on Transcoder nodes.

Even Transcoder node load distribution

To distribute streams published to Transcoder nodes evenly, a next available node will be chosen to transcode new stream (Round Robiun algorithm). Stream name caching is used to prevent the same stream requested from couple of clients simultaneously to be spread over a number of Transcoder nodes. Stream cache timeout is set in milliseconds with the following parameter

cdn_transcoder_for_new_connects_expire=10000

In this case cache timeout is 10 seconds.

This parameter should be set on Transcoder nodes.

Origin transcoding overload prevention

The following setting is used to prevent Origin nodes from transcoding streams if no Transcoders available

cdn_origin_allowed_to_transcode=false

In this case (by default), if no Trascoders available for new stream by certain profile, this stream will not be played with No available transcoders error.

Transcoding on Origin nodes can be enabled if necessary

cdn_origin_allowed_to_transcode=true

This parameter should be set on Edge nodes.

Node state management with REST API

REST query should be HTTP/HTTPS POST request as follows:

...

1-PROFILE-192.168.1.220

2-STREAM-192.168.1.220

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}

VIDEO{width=320, height=180, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='FF', quality=null}

3-NEW-TRANSCODER-192.168.1.220

4-PROXY-192.168.1.219

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}

VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}

CDN routes checking examples

Let's look how to check some CDN routes build for stream playback.

For example we use three nodes CDN:

• 192.168.1.219 - Origin
• 192.168.1.220 - Transcoder
• 192.168.1.221 - Edge

WebRTC H264+opus (48 kHz, stereo) stream named test is published to Origin

Stream pulling from Origin without transcoding if publishing and playback profiles are equal

Stream named test is played on Edge by profile

 -webrtc-opus-video-proxy:
  audio:
    codec : opus
    rate : 48000
    channels : 2
  video:
    codec: h264

The response to this query

http://192.168.1.221:8081/rest-api/cdn/stream/show_

...

Where:

• test.flashphoner.com - WCS server address
• 8081 - a standard WCS REST / HTTP port
• 8444 - a standard WCS HTTPS port
• rest-api - mandatory part of URL
• /cdn/show_nodes - REST query used

REST queries and responses

...

REST query

...

REST query body example

...

REST response

...

Response states

...

Description

...

routes
{
    "streamName": "test-webrtc-opus-video-proxy"
}

should be interpreted as follows:

1. Profile requested parameters:

...

...

"REQUESTED-PROFILE": [
        "

...

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "

...

VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='h264', codecImpl='null', quality=null}"
    ],

2. Stream is pulled from Origin:

    "1-PROXY-PROFILE-192.168.1.219": [
    

...

...

...

"AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "

...

VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}"
    ],

3. Stream can be pulled from Transcoder

    "

...

2-NEW-TRANSCODER-192.168.1.

...

220"

...

: [],

4. But is pulled from Origin because stream publishing parameters are equal to requested playback parameters:

    "3-PROXY-192.168.1.219": [
        "

...

AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "

...

200 – OK

500 – Internal Server Error

...

Show CDN nodes state

...

VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}"
    ]

Stream pulling from Transcoder with audio transcoding

Stream named test is played on Edge by profile

...

...

NEW_STREAMS_ALLOWED

...

200 – OK

500 – Internal Server Error

...

 -webrtc-pcma-video-proxy:
  audio:
    codec : pcma
    rate : 8000
    channels : 1

The response to this query

http://192.168.1.221:8081/rest-api/cdn/stream/show_routes
{
    "

...

streamName": "

...

test-webrtc-pcma-video-proxy"
}

...

200 – OK

500 – Internal Server Error

...

Parameters

...

Parameter name

...

Description

...

Example

...

globalState

...

Node state: ACTIVE or PASSIVE

...

ACTIVE

...

id

...

Node address

...

192.168.1.64

...

If this node participates in stream playback route elections:

NEW_STREAMS_ALLOWED - participates

GROUP_CONNECTION_ALLOWED - not participates

...

NEW_STREAMS_ALLOWED

...

ORIGIN

REST query /cdn/show_nodes may be sent to certain node, the node responds with all visible CDN nodes state excluding itself.

REST query /cdn/enforce_state allows forcefully change certain node state, for example, exclude the Transcoder node from route elections. Node can be pushed out of CDN with query

...

...

should be interpreted as follows:

1. Profile requested parameters:

    "REQUESTED-PROFILE": [
        "AUDIO{bitrate=0, codec='pcma', rate=8000, channels=1}"
    ],

2. Transcoding profile is created on Transcoder:

    "1-PROFILE-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='null', quality=null}",
         
Content-Type: application/json           "AUDIO{bitrate=0, codec='PCMA', rate=8000, channels=1}"
    ],

3. Stream is pulled from Transcoder

    "2-STREAM-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='null', quality=null}",
        "AUDIO{bitrate=0, codec='PCMA', rate=8000, channels=1}"
    ],
{    "3-NEW-TRANSCODER-192.168.1.220": [],

4. Stream publishing to Origin parameters:

    "3-PROXY-192.168.1.219": [
        "AUDIO{bitrate=0,    codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}"
    ]

Stream pulling from Transcoder with video transcoding

Stream named test is played on Edge by profile

 -opus-vp8:
   "state"audio:
 "PASSIVE"   codec : opus
    rate : 48000
    channels : 2
  video:
}    width : 320
    height : 240
    gop : 60
    fps : 30
    codec : vp8
    codecImpl :     

...

FF

The response to this query

POST http://192.168.1.221:8081/rest-api/cdn/stream/enforce_state HTTP/1.1
Content-Length: 19show_routes
{
                        
Content-Type: application/json           "streamName": "test-webrtc-opus-vp8"
}

should be interpreted as follows:

1. Profile requested parameters:

    "REQUESTED-PROFILE": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=320, height=240, gop=60, fps=30, bitrate=0, codec='vp8', codecImpl='FF', quality=null}"
     ],

2. Transcoding profile is created on Transcoder:

          
{"1-PROFILE-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}",
              
    "state": "ACTIVE"        "VIDEO{width=320, height=240, gop=60, fps=30, bitrate=0, codec='VP8', codecImpl='FF', quality=null}"
    ],

3. Stream is pulled from Transcoder

    "2-STREAM-192.168.1.220": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
}                "VIDEO{width=0, height=0, gop=null, fps=null, bitrate=0, codec='H264', codecImpl='', quality=null}",
        "VIDEO{width=320, height=240, gop=60, fps=30, bitrate=0, codec='VP8', codecImpl='FF', quality=null}"
    ],
    "3-NEW-TRANSCODER-192.168.1.220": 

...

[],

4. Stream publishing to Origin parameters:

POST /rest-api/cdn/enforce_state HTTP/1.1
Content-Length: 4    "3-PROXY-192.168.1.219": [
        "AUDIO{bitrate=0, codec='opus', rate=48000, channels=2}",
        "VIDEO{width=0, height=0, gop=null, fps=null,  
Content-Type: application/json bitrate=0, codec='H264', codecImpl='', quality=null}"
          
                                         
{}                                       

Node state definition and broadcasting

Current CDN node state is defined as follows in priority order:
1. If there is a value enforced by /cdn/enforce_state REST query, state will be set to this value.
2. If CPU load threshold is reached, state will be set to GROUP_CONNECTIONS_ALLOWED.
3. If video encoders threshold is reached, state will be set to GROUP_CONNECTIONS_ALLOWED.
4. If no limits are reached, state will be set to NEW_STREAMS_ALLOWED.

CDN node broadcasts its state to another nodes periodically, using the interval set in milliseconds with the folowing parameter

cdn_nodes_state_refresh_interval=60000

...

]

CDN stream advertising by keyframe receiving

By default, all the streams published on Origin nodes are advertised to CDN and become available to select route and play right after publishing. In this case, a video stream picture resolution and aspect ratio are unknown until a publisher sends the first keyframe.

In some cases, it is necessary to know the stream picture resolution and aspect ratio on Edge server when the stream becomes available in CDN. To do this, Origin server can advertise streams after keyframe receiving since build 5.2.404. This behaviour can be enabled using the following parameter

cdn_advertise_streams_by_kframe=true

Note that audio only streams (without video track) will not be available in CDN with this setting, because they contain no keyframes.

Transcoder node load balancing

Node load detection by CPU load

If there are some Transcoder nodes with the same profiles in CDN, those nodes load can be balanced  by CPU load. Maximum allowed load for certain Transcoder can be defined with the following parameter

cdn_node_load_average_threshold=1.0

This value sets average CPU load (JVM SystemLoadAverage parameter) to CPU cores avalable ratio, including hyperthreading cores. When this value is exceeded, the Transcoder node is excluded from route elections for new stteams and profiles. In this case the Transcoder node is displayed in the stream route PROFILE section for profiles the stream is already transcoded by.

Node current state (if node participates in new stream route elections) can be checked with REST API. A node in NEW_STREAMS_ALLOWED state can pull new streams to transcode, a node in GROUP_CONNECTION_ALLOWED state can only push the streams already transcoded.

This parameter should be set on Transcoder nodes. Also this parameter can be set on Origin nodes, in this case their load can be checked via REST API, and if the Origin node become GROUP_CONNECTION_ALLOWED then new streams can be published to another Origin server.

Transcoder node video encoders limiting

If necessary, maximum video encoders simultaneously used amount can be lemited for Transcoder nodes with the following parameter

cdn_transcoder_video_encoders_threshold=10000

When this amount is reached, the Transcoder node becomes GROUP_CONNECTION_ALLOWED. All the video encoders on server are included, for example, if one stream is transcoding by two profiles, 3 video encoders are used, including PNG encoder.

This parameter should be set on Transcoder nodes.

Decoders load limiting

Since build 5.2.594, an integral decoders load value can be limited. For example, to allow no more then 20 streams 1080p with 30 fps to be decoded, the following parameter should be set

cdn_transcoder_video_decoders_load_threshold=1244160000

which is the result of multiplication

1920 * 1080 * 30 * 20

When this value is reached, the Transcoder node becomes GROUP_CONNECTION_ALLOWED.

This parameter should be set on Transcoder nodes.

Encoders load limiting

Since build 5.2.594, an integral decoders load value can be limited. For example, to allow no more then 20 streams to be encoded to 720p, 480p and 240p with 25 fps, the following parameter should be set

cdn_transcoder_video_encoders_load_threshold=716400000

which is the result of summation

1280 * 720 * 25 * 20 + 852 * 480 * 25 * 20 + 426 * 240 * 25 * 20

When this value is reached, the Transcoder node becomes GROUP_CONNECTION_ALLOWED.

This parameter should be set on Transcoder nodes.

Degraded streams percent limiting

Since build 5.2.594, a degrading streams percent can be limited. For example, to stop new streams transcoding when 10% of streams is degraded, the following parameter should be set

cdn_transcoder_degraded_streams_threshold=10

When this value is reached, the Transcoder node becomes GROUP_CONNECTION_ALLOWED. Note that streams usually start to degrade due to lack of decoder or encoder perfomance, and decoder/encoder picture queues are grown to maximum values to the moment of degradation, that leads to RAM waste. Therefore, this parameter must be used in addition to encoder/decoder load thresholds only.

This parameter should be set on Transcoder nodes.

Even Transcoder node load distribution

To distribute streams published to Transcoder nodes evenly, a next available node will be chosen to transcode new stream (Round Robiun algorithm). Stream name caching is used to prevent the same stream requested from couple of clients simultaneously to be spread over a number of Transcoder nodes. Stream cache timeout is set in milliseconds with the following parameter

cdn_transcoder_for_new_connects_expire=10000

In this case cache timeout is 10 seconds.

This parameter should be set on Transcoder nodes.

Origin transcoding overload prevention

The following setting is used to prevent Origin nodes from transcoding streams if no Transcoders available

cdn_origin_allowed_to_transcode=false

In this case (by default), if no Trascoders available for new stream by certain profile, this stream will not be played with No available transcoders error.

Transcoding on Origin nodes can be enabled if necessary

cdn_origin_allowed_to_transcode=true

This parameter should be set on Edge nodes.

Node state management with REST API

REST query should be HTTP/HTTPS POST request as follows:

• HTTP: http://test.flashphoner.com:8081/rest-api/cdn/show_nodes
• HTTPS: https://test.flashphoner.com:8444/rest-api/cdn/show_nodes

Where:

• test.flashphoner.com - WCS server address
• 8081 - a standard WCS REST / HTTP port
• 8444 - a standard WCS HTTPS port
• rest-api - mandatory part of URL
• /cdn/show_nodes - REST query used

REST queries and responses

[
    {
        "globalState": "ACTIVE", 
        "id": "192.168.1.64", 
        "processingState": "NEW_STREAMS_ALLOWED", 
        "role": "TRANSCODER"
    }, 
    {
        "globalState": "ACTIVE", 
        "id": "192.168.1.39", 
        "processingState": "NEW_STREAMS_ALLOWED", 
        "role": "ORIGIN"
    }
]

NEW_STREAMS_ALLOWED

{
"state": "GROUP_CONNECTIONS_ALLOWED"
}

Parameters

ACTIVE

192.168.1.64

NEW_STREAMS_ALLOWED

ORIGIN

REST query /cdn/show_nodes may be sent to certain node, the node responds with all visible CDN nodes state excluding itself.

REST query /cdn/enforce_state allows forcefully change certain node state, for example, exclude the Transcoder node from route elections. Node can be pushed out of CDN with query

POST /rest-api/cdn/enforce_state HTTP/1.1
Content-Length: 20                       
Content-Type: application/json           
                                         
{                                        
    "state": "PASSIVE"                   
}                                        

and then can be pulled back

POST /rest-api/cdn/enforce_state HTTP/1.1
Content-Length: 19                       
Content-Type: application/json           
                                         
{                                        
    "state": "ACTIVE"                   
}                                        

To drop enforced state, an emty query body should be passed

POST /rest-api/cdn/enforce_state HTTP/1.1
Content-Length: 4                        
Content-Type: application/json           
                                         
{}                                       

Node state definition and broadcasting

Current CDN node state is defined as follows in priority order:
1. If there is a value enforced by /cdn/enforce_state REST query, state will be set to this value.
2. If CPU load threshold is reached, state will be set to GROUP_CONNECTIONS_ALLOWED.
3. If video encoders threshold is reached, state will be set to GROUP_CONNECTIONS_ALLOWED.
4. If no limits are reached, state will be set to NEW_STREAMS_ALLOWED.

CDN node broadcasts its state to another nodes periodically, using the interval set in milliseconds with the folowing parameter

cdn_nodes_state_refresh_interval=60000

By default, this interval is 60 seconds. It is recommended to reduce this parameter to 1 second for Transcoder nodes to distribute the streams to another nodes.

Transcoder threshold reached state setup

Since build 5.2.640 Transcoder node threshold reached state can be set with the following parameter, GROUP_CONNECTIONS_ALLOWED by default

cdn_transcoder_threshold_state=GROUP_CONNECTIONS_ALLOWED

To prevent stream spreading over CDN, Transcoder node can be switched to CONNECTIONS_ALLOWED state if necessary

cdn_transcoder_threshold_state=CONNECTIONS_ALLOWED

In this state, Transcoder node allows to pull streams which are already decoded on, including new encoding profiles. For example, if some Transcoder node decodes test stream by 360p profile, and som of Edge servers requests test240p stream, playback route will be built via the same Transcoder node.

This parameter should be set on Transcoder nodes.

CDN nodes authentication

...

In these cases transcoding works according to codecs and SDP setup as defined for CDN 2.0.

Known limits

1. It is strongly not recommended to publish streams with same name to two Origin servers in the same CDN.

...

cdn_force_version=2.2

Known issues

1. When FFmpeg based encoder FF encoder imp;lementation is used, there can be distortions (macroblocks) in first frames of stream transcoded

...

profiles:
 -240p:
  audio:
    codec : opus
    rate : 48000
  video:
    height : 240
    gop : 50
    fps : 25
    codec : h264
    preset : veryfast
    quality : 25
    codecImpl : FF

b) use OpenH264 OPENH264 encoder implementation if it is necessary to keep specified video bitrate

profiles:
 -240p:
  audio:
    codec : opus
    rate : 48000
  video:
    height : 240
    bitrate : 400
    gop : 50
    fps : 25
    codec : h264
    codecImpl : OPENH264

2. Encoding quality settings cannot be applied if OpenH264 is used

Symptoms: picture quality is not changing when using different quality values in transcoding profile

Solution: do not use OpenH264 encoder because it does not support CRF

profiles:
 -240p:
  audio:
    ...
  video:
    ...
    quality: 10
    codecImpl : FF