Overview

Since build 5.2.795 it is possible to extract raw audio data from published stream in PCM 16 bit format with a following processing on server side. For example, stream sound can be recorded to a file.

Note that the stream published should have at least one subscriber to decode and extract audio from it.

Audio processing implementation

To intercept raw audio data, a Java class implementing IDecodedPcmInterceptor interface should be developed. The method pcmDecoded() of this class will receive decoded audio packets in PCM format, as byte array. Let's take a look to class implementation example to record raw audio from a stream into a WAV file:

// Package name should be strictly defined as com.flashphoner.sdk.media
package com.flashphoner.sdk.media;

// Import Flashphoner SDK packages as needed
import com.flashphoner.media.rtp.recorder.OutputFileType;
import com.flashphoner.media.utils.WaveUtil;
import com.flashphoner.sdk.setting.Settings;
import com.flashphoner.media.utils.FileNameUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
// Import standard Java packages as needed
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.Timer;
import java.util.TimerTask;

/**
 * Custom decoded audio interceptor implementation example
 * The example records first 10 seconds of audio track from a stream published to WAV file
 */
public class DecodedPcmInterceptorTest implements IDecodedPcmInterceptor{

    // Will log errors to server log.
    private static final Logger log = LoggerFactory.getLogger("DecodedPcmInterceptorTest");

    // File object to write data
    protected RandomAccessFile incomingRecorder;
    // Last audio packet timestamp
    private volatile long lastTs;
    // Sampling rate to write to a file header
    private int samplingRate;
    // Number of cahnnels to write to a file header
    private int numChannels;
    // Timer to stop recording
    private Timer cancelTimer;
    // Stream name to form file name
    private String streamName;

    public DecodedPcmInterceptorTest() {
    }

    /**
     * Method to handle decoded audio
     * @param streamName - stream name
     * @param pcm - decoded audio data from packet as byte array
     * @param samplingRate - audio track sampling rate
     * @param numChannels - audio track number of channels
     * @param timestamp - audio packet timestamp
     */
    @Override
    public void pcmDecoded(String streamName, byte[] pcm, int samplingRate, int numChannels, long timestamp) {
        // Remember data to write to the file header
        updateSr(samplingRate);
        updateNumChannels(numChannels);
        // Remember the stream name and create the file to write
        updateStreamName(streamName);
        // Start timeout to stop recording
        startCloseTimer();
        try {
            // Write audio data to the file
            recordIncoming(pcm, timestamp);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    
    /**
     * Method to start timer task which should stop recording after 10 seconds
     */
    public void startCloseTimer() {
        if (cancelTimer == null) {
            cancelTimer = new Timer();
            cancelTimer.schedule(new TimerTask() {
                @Override
                public void run() {
                    try {
                        close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
            }, 10000);
        }
    }

    /**
     * Method to store audio sampling rate to write it to the file header
     */
    public void updateSr(int samplingRate) {
        if (this.samplingRate == 0) {
            this.samplingRate = samplingRate;
        }
    }

    /**
     * Method to store audio channels number to write it to the file header
     */
    public void updateNumChannels(int numChannels) {
        if (this.numChannels == 0) {
            this.numChannels = numChannels;
        }
    }

    /**
     * Method to store the stream name and create the file to write
     */
    public void updateStreamName(String streamName) {
        if (this.streamName == null || this.streamName.isEmpty()) {
            // Create the file name
            this.streamName = FileNameUtils.adaptRecordName(streamName+".wav");
            try {
                // Create the file and reserve header space
                incomingRecorder = new RandomAccessFile(Settings.RECORD.getValue()+"/"+this.streamName, "rw");
                incomingRecorder.write(new byte[OutputFileType.WAV_HEADER_OFFSET]);
            } catch (IOException e) {
                log.error("Can't create DecodedPcmInterceptorTest, " + e.getMessage());
            }
            log.info("Create DecodedPcmInterceptorTest");
        }
    }

    /**
     * Method to write audio data to the file
     */
    public void recordIncoming(byte[] data, long ts) throws IOException {
        incomingRecorder.write(data);
        lastTs = ts;
    }

    /**
     * Method to close file
     */
    private void close() throws IOException {
        // Write header to the file
        writeHeader();
        try {
            // Close the file
            incomingRecorder.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
        log.info("Close DecodedPcmInterceptorTest");
    }

    /**
     * Method to write header to the beginning of the file
     */
    protected void writeHeader() throws IOException {
        // Get the file size
        int size = (int) incomingRecorder.length();
        // Form the header
        byte[] header = WaveUtil.getPcmWaveHeader((size - OutputFileType.WAV_HEADER_OFFSET), samplingRate, numChannels);
        // Write the header to the beginning of the file
        incomingRecorder.seek(0);
        incomingRecorder.write(header);
    }
}

Then the class should be complied into byte code. To do this, create folder tree accordind to class package name

mkdir -p com/flashphoner/sdk/media

and execute the command

javac -cp /usr/local/FlashphonerWebCallServer/lib/wcs-core.jar:/usr/local/FlashphonerWebCallServer/lib/slf4j-api-1.6.4.jar ./com/flashphoner/sdk/media/DecodedPcmInterceptorTest.java

Now, pack the code compiled to jar file

jar -cf testPcmInterceptor.jar ./com/flashphoner/sdk/media/*.class

and copy this file to WCS libraries folder

cp testPcmInterceptor.jar /usr/local/FlashphonerWebCallServer/lib

To use custom frames interceptor class, set its package name and folder to record WAV files to the following parameters in flashphoner.properties file

decoded_pcm_interceptor=com.flashphoner.sdk.media.DecodedPcmInterceptorTest
record=/usr/local/FlashphonerWebCallServer/records

and restart WCS.

A separate folder for custom Java libraries

Since build 5.2.1512, custom layout Java libraries (jar files) should be placed to the folder /usr/local/FlashphonerWebCallServer/lib/custom 

cp testlayout.jar /usr/local/FlashphonerWebCallServer/lib/custom

This folder is kept while updating WCS to a newer builds. A jar files do not need to be copied again after updating.

Testing

1. Publish a test stream in Two Way Streaming example https://test1.flashphoner.com:8444/client2/examples/demo/streaming/two_way_streaming/two_way_streaming.html, where test1.flashphoner.com is WCS server address

2. Check if WAV file exists in  /usr/local/FlashphonerWebCallServer/records/ folder