Skip to content

Decoded frames interception and hangling with OpenCV

Overview

The decoded frames interception and handling feature allows to use a powerful images handling library OpenCV. The feature may be useful, for example, in case of AR implementatioin at server side and other similar cases, and is available since WCS build 5.2.1914.

OpenCV library should be built before implementing a custom Java class to integrate it.

Building OpenCV

Let's describe OpenCV 4.9.0 building on Centos 7, to provide glibc 2.17 compatibility.

  1. Install JDK 8 and ANT

    yum install openjdk-8-jdk ant

  2. Detect JDK installation folder path and set the environment variable JAVA_HOME

    export JAVA_HOME=$(echo $(readlink -f $(which javac)) | sed 's/\/bin\/javac//g')

  3. Install CMake 3.6.2 or newer

    wget https://cmake.org/files/v3.6/cmake-3.6.2.tar.gz
tar -zxf cmake-3.6.2.tar.gz
cd cmake-3.6.2
./bootstrap --prefix=/usr/local
make -j$(nproc)
make install

  4. Install GCC 11 and switch to it to build OpenCV

    yum install centos-release-scl
yum install devtoolset-11-gcc devtoolset-11-gcc-c++
scl enable devtoolset-11 -- bash

  5. Download and unpack OpenCV source code

    wget -O opencv-490.zip https://github.com/opencv/opencv/archive/4.9.0.zip
unzip opencv-490.zip

  6. Set up the build

    cd opencv-490
mkdir build
cd build
cmake ..

  7. Build OpenCV

    make -j $(nproc)

  8. Copy native and Java modules to WCS installation folder

    cp bin/opencv-490.jar /usr/local/FlashphonerWebCallServer/lib/custom
cp -r lib/* /usr/local/FlashphonerWebCallServer/lib/so

Interceptor implementation using OpenCV

A Java class implementing Java IDecodedFrameInterceptorinterface should be developed. The function frameDecoded() of this class will receive decoded frames in YUV I420 format, create OpenCV Matrix for each YUV420 data array and apply blur effect

Previous transcoding generation implementation example

transcoding_gen3=false

TestInterceptor.java

// Package name should be strictly defined as com.flashphoner.frameInterceptor
package com.flashphoner.frameInterceptor;

import com.flashphoner.sdk.media.IDecodedFrameInterceptor;
import com.flashphoner.sdk.media.YUVFrame;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.Size;
import org.opencv.imgproc.Imgproc;

import java.nio.ByteBuffer;

public class OpenCvInterceptor implements IDecodedFrameInterceptor {

   // Load OpenCv java wrapper
   static {
       System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
   }

   @Override
   public void frameDecoded(String streamName, YUVFrame frame) {
       ByteBuffer[] data = frame.getData();

       // WCS handles YUV pointers separately. Create Matrix for each YUV420 plane
       Mat yMatrix = new Mat(frame.getHeight(),        frame.getStrideY(),      CvType.CV_8UC1, data[YUVFrame.Y_POINTER_IDX]);
       Mat uMatrix = new Mat(frame.getChromaHeight(),  frame.getStrideChroma(), CvType.CV_8UC1, data[YUVFrame.U_POINTER_IDX]);
       Mat vMatrix = new Mat(frame.getChromaHeight(),  frame.getStrideChroma(), CvType.CV_8UC1, data[YUVFrame.V_POINTER_IDX]);

       Size gaussianKernelSize = new Size(33, 33);
       // Apply Gaussian Blue. Passed kernel size should be odd
       Imgproc.GaussianBlur(yMatrix, yMatrix, gaussianKernelSize, 0);
       Imgproc.GaussianBlur(uMatrix, uMatrix, gaussianKernelSize, 0);
       Imgproc.GaussianBlur(vMatrix, vMatrix, gaussianKernelSize, 0);
   }

}

Transcoding generation 3 implementation example

transcoding_gen3=true

TestInterceptor.java

// Package name should be strictly defined as com.flashphoner.frameInterceptor
package com.flashphoner.frameInterceptor;

import com.flashphoner.sdk.media.IDecodedFrameInterceptor;
import com.flashphoner.sdk.media.YUVFrame;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.Size;
import org.opencv.imgproc.Imgproc;

import java.nio.ByteBuffer;

public class OpenCvInterceptor implements IDecodedFrameInterceptor {

   // Load OpenCv java wrapper
   static {
       System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
   }

   @Override
   public void frameDecoded(String streamName, YUVFrame frame) {
       if (!frame.isNewGen()) {
           // skip only GEN3 supported
           return;
       }
       ByteBuffer[] data = frame.getData();

       // WCS handles YUV pointers separately. Create Matrix for each YUV420 plane
       Mat yMatrix = new Mat(frame.getHeight(),        frame.getStrideY(),      CvType.CV_8UC1, data[YUVFrame.Y_POINTER_IDX]);
       Mat uMatrix = new Mat(frame.getChromaHeight(),  frame.getStrideChroma(), CvType.CV_8UC1, data[YUVFrame.U_POINTER_IDX]);
       Mat vMatrix = new Mat(frame.getChromaHeight(),  frame.getStrideChroma(), CvType.CV_8UC1, data[YUVFrame.V_POINTER_IDX]);

       Size gaussianKernelSize = new Size(33, 33);
       // Apply Gaussian Blue. Passed kernel size should be odd
       Imgproc.GaussianBlur(yMatrix, yMatrix, gaussianKernelSize, 0);
       Imgproc.GaussianBlur(uMatrix, uMatrix, gaussianKernelSize, 0);
       Imgproc.GaussianBlur(vMatrix, vMatrix, gaussianKernelSize, 0);
   }

}

Building and using the example

Then the class should be complied into byte code. To do this, create folder tree accordind to TestInterceptor class package name

mkdir -p com/flashphoner/frameInterceptor

and execute the command

javac -cp /usr/local/FlashphonerWebCallServer/lib/wcs-core.jar:/usr/local/FlashphonerWebCallServer/lib/custom/opencv-490.jar ./com/flashphoner/frameInterceptor/TestInterceptor.java

Now, pack the code compiled to jar file

jar -cf testinterceptor.jar ./com/flashphoner/frameInterceptor/TestInterceptor.class

and copy this file to WCS custom libraries folder

cp testinterceptor.jar /usr/local/FlashphonerWebCallServer/lib/custom

To use custom frames interceptor class, set its package name to the following parameter in flashphoner.properties

decoded_frame_interceptor=com.flashphoner.frameInterceptor.TestInterceptor

and restart WCS.

Testing

  1. Publish a test stream in Two Way Streaming example

  2. Play the stream in Player example with explicit resolution setting to enable transcoding, for example
    https://test1.flashphoner.com:8444/client2/examples/demo/streaming/player/player.html?resolution=320x240,
    where test1.flashphoner.com is WCS server address

The picture will be blurred.