Streaming Auto Restore¶

A streamer example with publishing/playback automatic restore¶

This example shows how to restore stream publishing/playback automatically:

changing publishing codec to VP8 when H264 publishing fails
republish a stream if browser stops sending media packets (this is detected by video bitrate drop to 0)
when network is changing (form Wi-Fi to LTE and vice versa)
when streaming fails due to server connection breaking (including server restart) or due to stream publishing stopping by other side

Bitrate checking parameters

Check bitrate - check if bitrate drops to 0
Change codec - change H264 codec to VP8 if bitrate drop is detected
Bitrate check interval - publishing bitrate checking interval
Max tries - maximium number of subsequent bitrate drops to 0

Connection restoring parameters

Restore connection - restore connection if session breaks or publishing/playback is failed
Timeout - connection restore tries interval
Max tries - maximum number of connection restore tries
Missing pings - maximum number of subsequent missing pings from server (0 disables ping checking)
Pings check period - pings checking interval (0 disables ping checking)

Code of the example¶

The example code is available on WCS server by the following path:

/usr/local/FlashphonerWebCallServer/client2/examples/demo/streaming/stream-auto-restore

stream-auto-restore.css - styles file
stream-auto-restore.html - client page
stream-auto-restore.js - main script to work

The example can be tested by the following URL:

https://host:8888/client2/examples/demo/streaming/stream_filter/stream-auto-restore.html

Where host - WCS server address.

Analyzing the code¶

To analyze the code take the file stream-auto-restore.js version with hash 2035db9 which is available here and can be downloaded with SDK build 2.0.209.

1. Page loading actions¶

1.1. API initialization¶

Flashphoner.init() code

Flashphoner.init();

1.2. Session and publishing/playing streams state objects initialization¶

code

currentSession = sessionState();
streamPublishing = streamState();
streamPlaying = streamState();

1.3. Bitrate checking object initialization¶

code

h264PublishFailureDetector = codecPublishingFailureDetector();

1.4. Connection restore object initialization¶

code

The function should be passed to the object to execute it when restore connection timer is fired

streamingRestarter = streamRestarter(function() {
  if (streamPublishing.wasActive) {
      onPublishRestart();
  }
  if (streamPlaying.wasActive && streamPlaying.name != streamPublishing.name) {
      onPlayRestart();
  }
});

1.5. Network change detector start¶

code

networkChangeDetector();

2. Server connection/disconnection actions¶

2.1. Connecting to the server¶

Flashphoner.createSession() code

The following parameters are passed when session is created:

url - server Websocket URL
receiveProbes - maximum number of subsequent missing pings from server (0 disables ping checking)
probesInterval - pings checking interval (0 disables ping checking)

Flashphoner.createSession({
    urlServer: url,
    receiveProbes: receiveProbes,
    probesInterval: probesInterval
}).on(SESSION_STATUS.ESTABLISHED, function (session) {
    ...
}).on(SESSION_STATUS.DISCONNECTED, function () {
    ...
}).on(SESSION_STATUS.FAILED, function () {
    ...
});

2.2. Receiving the event comfirming successful connection¶

ConnectionStatusEvent ESTABLISHED code

When xconnection is established:

session parameters are stored in session state object
stream publishing/playback is restarted if stream was published/played in previous session

Flashphoner.createSession({
    urlServer: url,
    receiveProbes: receiveProbes,
    probesInterval: probesInterval
}).on(SESSION_STATUS.ESTABLISHED, function (session) {
    setStatus("#connectStatus", session.status());
    currentSession.set(url, session);
    onConnected(session);
    if(restoreConnection) {
        if(streamPublishing.wasActive) {
            console.log("A stream was published before disconnection, restart publishing");
            onPublishRestart();
            return;
        }
        if(streamPlaying.wasActive) {
            console.log("A stream was played before disconnection, restart playback");
            onPlayRestart();
        }
    }
}).on(SESSION_STATUS.DISCONNECTED, function () {
    ...
}).on(SESSION_STATUS.FAILED, function () {
    ...
});

2.3. Connection closing by clicking `Disconnect` button¶

Session.disconnect() code

function onConnected(session) {
    $("#connectBtn").text("Disconnect").off('click').click(function () {
        $(this).prop('disabled', true);
        currentSession.isManuallyDisconnected = true;
        session.disconnect();
    }).prop('disabled', false);
    ...
}

2.4. Receiving the connection closing event¶

ConnectionStatusEvent DISCONNECTED code

If connection is closed manually by clicking Disconnect:

state objects are cleared
connection restore timer is stopped

Flashphoner.createSession({
    urlServer: url,
    receiveProbes: receiveProbes,
    probesInterval: probesInterval
}).on(SESSION_STATUS.ESTABLISHED, function (session) {
    ...
}).on(SESSION_STATUS.DISCONNECTED, function () {
    setStatus("#connectStatus", SESSION_STATUS.DISCONNECTED);
    onDisconnected();
    // Prevent streaming restart if session is manually disconnected
    if (currentSession.isManuallyDisconnected) {
        streamPublishing.clear();
        streamPlaying.clear();
        streamingRestarter.reset();
        currentSession.clear();
    }
}).on(SESSION_STATUS.FAILED, function () {
    ...
});

2.5. Receiving the connection failure event¶

ConnectionStatusEvent FAILED code

Connection restore timer is starting if a stream was published or played befor connection is failed

Flashphoner.createSession({
    urlServer: url,
    receiveProbes: receiveProbes,
    probesInterval: probesInterval
}).on(SESSION_STATUS.ESTABLISHED, function (session) {
    ...
}).on(SESSION_STATUS.DISCONNECTED, function () {
    ...
}).on(SESSION_STATUS.FAILED, function () {
    setStatus("#connectStatus", SESSION_STATUS.FAILED);
    onDisconnected();
    if(restoreConnection
       && (streamPublishing.wasActive || streamPlaying.wasActive)) {
        streamingRestarter.restart($("#restoreTimeout").val(), $("#restoreMaxTries").val());
    }
});

3. Stream publishing actions¶

3.1 Stream publishing¶

Session.createStream(), Stream.publish() code

The following parameters are passed while stream creation:

streamName - stream name to publish
localVideo - div element to display local video
stripCodecs - codec to exclude if codec changing option is active

session.createStream({
    name: streamName,
    display: localVideo,
    cacheLocalResources: true,
    receiveVideo: false,
    receiveAudio: false,
    stripCodecs: stripCodecs
    ...
}).publish();

3.2. Receiving the stream publishing event¶

StreamStatusEvent PUBLISHING code

When stream is publishing successfully:

bitrate checking timer starts
the stream parameters are stored in publishing stream state object
connection restore timer stops
stream playback starts if stream was played previously

session.createStream({
    ...
}).on(STREAM_STATUS.PUBLISHING, function (stream) {
    setStatus("#publishStatus", STREAM_STATUS.PUBLISHING);
    onPublishing(stream);
    streamPublishing.set(streamName, stream);
    streamingRestarter.reset();
    if ($("#restoreConnection").is(':checked')
       && streamPlaying.wasActive) {
        console.log("A stream was played before, restart playback");
        onPlayRestart();
    }
}).on(STREAM_STATUS.UNPUBLISHED, function () {
    ...
}).on(STREAM_STATUS.FAILED, function (stream) {
    ...
}).publish();

3.3. Bitrate checking timer startup¶

code

function onPublishing(stream) {
    ...
    // Start publish failure detector by bitrate #WCS-3382
    if($("#checkBitrate").is(':checked')) {
        h264PublishFailureDetector.startDetection(stream, $("#bitrateInteval").val(), $("#bitrateMaxTries").val());
    }
}

3.4. Publishing stopping by clicking `Stop` button¶

Stream.stop() code

function onPublishing(stream) {
    $("#publishBtn").text("Stop").off('click').click(function () {
        $(this).prop('disabled', true);
        streamPublishing.isManuallyStopped = true;
        stream.stop();
    }).prop('disabled', false);
    ...
}

3.5. Receiving stream publishing stoppin event¶

StreamStatusEvent UNPUBLISHED code

When stream is successfully stopped:

bitrate checking timer stops
connection restore timer stops
publishing stream state object is cleared

session.createStream({
    ...
}).on(STREAM_STATUS.PUBLISHING, function (stream) {
    ...
}).on(STREAM_STATUS.UNPUBLISHED, function () {
    setStatus("#publishStatus", STREAM_STATUS.UNPUBLISHED);
    onUnpublished();
    if (!streamPlaying.wasActive) {
        // No stream playback< we don't need restart any more
        streamingRestarter.reset();
    } else if (streamPlaying.wasActive && streamPlaying.name == streamPublishing.name) {
        // Prevent playback restart for the same stream
        streamingRestarter.reset();
    }
    streamPublishing.clear();
}).on(STREAM_STATUS.FAILED, function (stream) {
    ...
}).publish();

3.6. Receiving stream publishing failure event¶

StreamStatusEvent FAILED code

When stream publishing fails:

bitrate checking timer stops
connection restore timer starts unless local browser error is detected (media devices unavailable for example)

session.createStream({
    ...
}).on(STREAM_STATUS.PUBLISHING, function (stream) {
    ...
}).on(STREAM_STATUS.UNPUBLISHED, function () {
   ...
}).on(STREAM_STATUS.FAILED, function (stream) {
    setStatus("#publishStatus", STREAM_STATUS.FAILED, stream);
    onUnpublished();
    if ($("#restoreConnection").is(':checked') && stream.getInfo() != ERROR_INFO.LOCAL_ERROR) {
        streamingRestarter.restart($("#restoreTimeout").val(), $("#restoreMaxTries").val());
    }
}).publish();

3.7. Bitrate checking timer stopping¶

code

function onUnpublished() {
    ...
    h264PublishFailureDetector.stopDetection(streamPublishing.isManuallyStopped || currentSession.isManuallyDisconnected);
    ...
}

4. Stream playback actions¶

4.1. Stream playback¶

Session.createStream(), Stream.play() code

The following parameters are passed while stream creation:

streamName - stream name to play
remoteVideo - div element to display remote video

session.createStream({
    name: streamName,
    display: remoteVideo
    ...
}).play();

4.2. Receiving the stream playback event¶

StreamStatusEvent PLAYING code

When stream is successfully playing:

the stream parameters are stored in playing stream state object
connection restore timer stops

session.createStream({
    name: streamName,
    display: remoteVideo
}).on(STREAM_STATUS.PENDING, function (stream) {
    ...
}).on(STREAM_STATUS.PLAYING, function (stream) {
    setStatus("#playStatus", stream.status());
    onPlaying(stream);
    streamingRestarter.reset();
    streamPlaying.set(streamName, stream);
}).on(STREAM_STATUS.STOPPED, function () {
    ...
}).on(STREAM_STATUS.FAILED, function (stream) {
    ...
}).play();

4.3 Stream playback stopping by clicking `Stop` button¶

Stream.stop() code

function onPlaying(stream) {
    $("#playBtn").text("Stop").off('click').click(function(){
        $(this).prop('disabled', true);
        stream.stop();
    }).prop('disabled', false);
    $("#playInfo").text("");
}

4.4. Receiving the stream playback stopping event¶

StreamStatusEvent STOPPED code

When stream playback is successfully stopped:

connection restore timer stops
playing stream state object is cleared

session.createStream({
    name: streamName,
    display: remoteVideo
}).on(STREAM_STATUS.PENDING, function (stream) {
    ...
}).on(STREAM_STATUS.PLAYING, function (stream) {
    ...
}).on(STREAM_STATUS.STOPPED, function () {
    setStatus("#playStatus", STREAM_STATUS.STOPPED);
    onStopped();
    streamingRestarter.reset();
    streamPlaying.clear();
}).on(STREAM_STATUS.FAILED, function (stream) {
    ...
}).play();

4.5. Receiving the stream playback failure event¶

StreamStatusEvent FAILED code

Connection restore timer starts is stream playback fails

    session.createStream({
        name: streamName,
        display: remoteVideo
    }).on(STREAM_STATUS.PENDING, function (stream) {
        ...
    }).on(STREAM_STATUS.PLAYING, function (stream) {
        ...
    }).on(STREAM_STATUS.STOPPED, function () {
        ...
    }).on(STREAM_STATUS.FAILED, function (stream) {
        setStatus("#playStatus", STREAM_STATUS.FAILED, stream);
        onStopped();
        if ($("#restoreConnection").is(':checked')) {
            streamingRestarter.restart($("#restoreTimeout").val(), $("#restoreMaxTries").val());
        }
    }).play();

5. Bitrate checking and stream republishing if bitrate drops to 0¶

5.1. Receiving browser WebRTC statistics, codec and bitrate detection, publishing stopping if bitrate drops to 0¶

code

stream.getStats(function(stat) {
    let videoStats = stat.outboundStream.video;
    if(!videoStats) {
        return;
    }
    let stats_codec = videoStats.codec;
    let bytesSent = videoStats.bytesSent;
    let bitrate = (bytesSent - detector.lastBytesSent) * 8;
    if (bitrate == 0) {
        detector.counter.inc();
        console.log("Bitrate is 0 (" + detector.counter.getCurrent() + ")");
        if (detector.counter.exceeded()) {
            detector.failed = true;
            console.log("Publishing seems to be failed, stop the stream");
            stream.stop();
        }
    } else {
        detector.counter.reset();
    }
    detector.lastBytesSent = bytesSent;
    detector.codec = stats_codec;
    $("#publishInfo").text(detector.codec);
});

5.2. Bitrate checking timer stopping¶

code

if (detector.publishFailureIntervalID) {
    clearInterval(detector.publishFailureIntervalID);
    detector.publishFailureIntervalID = null;
}

5.3. Stream republishing¶

code

if (detector.failed) {
    $("#publishInfo").text("Failed to publish " + detector.codec);
    if($("#changeCodec").is(':checked')) {
        // Try to change codec from H264 to VP8 #WCS-3382
        if (detector.codec == "H264") {
            console.log("H264 publishing seems to be failed, trying VP8 by stripping H264");
            let stripCodecs = "H264";
            publishBtnClick(stripCodecs);
        } else if (detector.codec == "VP8") {
            console.log("VP8 publishing seems to be failed, giving up");
        }
    } else {
        // Try to republish with the same codec #WCS-3410
        publishBtnClick();
    }
}

6. Connection restoration¶

6.1. Connection restore timer launching¶

code

The timer invokes a function to perform an ctions needed

restarter.restartTimerId = setInterval(function(){
    if (restarter.counter.exceeded()) {
        logger.info("Tried to restart for " + restartMaxTimes + " times with " +restartTimeout + " ms interval, cancelled");
        restarter.reset();
        return;
    }
    onRestart();
    restarter.counter.inc();
}, restartTimeout);

6.2. Connection restore timer stopping¶

code

if (restarter.restartTimerId) {
    clearInterval(restarter.restartTimerId);
    logger.info("Timer " + restarter.restartTimerId + " stopped");
    restarter.restartTimerId = null;
}
restarter.counter.reset();

6.3. New session creation if previous session is failed or disconnected¶

code

let sessions = Flashphoner.getSessions();
if (!sessions.length || sessions[0].status() == SESSION_STATUS.FAILED) {
    logger.info("Restart session to publish");
    click("connectBtn");
} else {
    ...
}

6.4. Republishing¶

code

let streams = sessions[0].getStreams();
let stream = null;
let clickButton = false;
if (streams.length == 0) {
    // No streams in session, try to restart publishing
    logger.info("No streams in session, restart publishing");
    clickButton = true;
} else {
    // If there is already a stream, check its state and restart publishing if needed
    for (let i = 0; i < streams.length; i++) {
        if (streams[i].name() === $('#publishStream').val()) {
            stream = streams[i];
            if (!isStreamPublishing(stream)) {
                logger.info("Restart stream " + stream.name() + " publishing");
                clickButton = true;
            }
            break;
        }
    }
    if (!stream) {
        logger.info("Restart stream publishing");
        clickButton = true;
    }
}
if (clickButton) {
    click("publishBtn");
}

6.5. Replaying¶

code

let streams = sessions[0].getStreams();
let stream = null;
let clickButton = false;
if (streams.length == 0) {
    // No streams in session, try to restart playing
    logger.info("No streams in session, restart playback");
    clickButton = true;
} else {
    // If there is already a stream, check its state and restart playing if needed
    for (let i = 0; i < streams.length; i++) {
        if (streams[i].name() === $('#playStream').val()) {
            stream = streams[i];
            if (!isStreamPlaying(stream)) {
                logger.info("Restart stream " + stream.name() + " playback");
                clickButton = true;
            }
            break;
        }
    }
    if (!stream) {
        logger.info("Restart stream playback");
        clickButton = true;
    }
}
if (clickButton) {
    click("playBtn");
}

7. Network change actions¶

7.1. Network change event handling¶

NetworkInformation.onchange code

if (Browser.isChrome() || (Browser.isFirefox() && Browser.isAndroid())) {
    connection = navigator.connection || navigator.mozConnection || navigator.webkitConnection;
    if (connection) {
        connectionType = connection.type;
        if (Browser.isFirefox()) {
            connection.ontypechange = onNetworkChange;
        } else {
            connection.onchange = onNetworkChange;
        }
    }
}

7.2. Closing the connection if network is changed¶

code

if (isNetworkConnected() && connection.type != connectionType) {
    if (currentSession.getStatus() == SESSION_STATUS.ESTABLISHED) {
        let logger = Flashphoner.getLogger();
        logger.info("Close session due to network change from " + connectionType + " to " + connection.type);
        currentSession.sdkSession.disconnect();
    }
}

Streaming Auto Restore¶

A streamer example with publishing/playback automatic restore¶

Code of the example¶

Analyzing the code¶

1. Page loading actions¶

1.1. API initialization¶

1.2. Session and publishing/playing streams state objects initialization¶

1.3. Bitrate checking object initialization¶

1.4. Connection restore object initialization¶

1.5. Network change detector start¶

2. Server connection/disconnection actions¶

2.1. Connecting to the server¶

2.2. Receiving the event comfirming successful connection¶

2.3. Connection closing by clicking Disconnect button¶

2.4. Receiving the connection closing event¶

2.5. Receiving the connection failure event¶

3. Stream publishing actions¶

3.1 Stream publishing¶

3.2. Receiving the stream publishing event¶

3.3. Bitrate checking timer startup¶

3.4. Publishing stopping by clicking Stop button¶

3.5. Receiving stream publishing stoppin event¶

3.6. Receiving stream publishing failure event¶

3.7. Bitrate checking timer stopping¶

4. Stream playback actions¶

4.1. Stream playback¶

4.2. Receiving the stream playback event¶

4.3 Stream playback stopping by clicking Stop button¶

4.4. Receiving the stream playback stopping event¶

4.5. Receiving the stream playback failure event¶

5. Bitrate checking and stream republishing if bitrate drops to 0¶

5.1. Receiving browser WebRTC statistics, codec and bitrate detection, publishing stopping if bitrate drops to 0¶

5.2. Bitrate checking timer stopping¶

5.3. Stream republishing¶

6. Connection restoration¶

6.1. Connection restore timer launching¶

6.2. Connection restore timer stopping¶

6.3. New session creation if previous session is failed or disconnected¶

6.4. Republishing¶

6.5. Replaying¶

7. Network change actions¶

7.1. Network change event handling¶

7.2. Closing the connection if network is changed¶

2.3. Connection closing by clicking `Disconnect` button¶

3.4. Publishing stopping by clicking `Stop` button¶

4.3 Stream playback stopping by clicking `Stop` button¶