OpenCV and Processing 13 – Magic & Love Interactive

In this example, we move on to the Video module of OpenCV 3.0.0. The first function we test is the Dense Optical Flow. It demonstrates the use of the calcOpticalFlowFarneback function. Again it makes use of the previous CVImage object to bridge between the Processing PImage and OpenCV Mat. The example also reduces the size of the video (using the variable factor) before sending it for the optical flow processing; otherwise, the process can be lengthy.

The optical flow process will basically compare two consecutive frames (the Mat last and grey) from the live webcam video. It will try to compute where the current pixels move to in the new frame.

Here are a number of screen shots from the sample run.

import processing.video.*;
import org.opencv.video.Video;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
 
Capture cap;
CVImage img;
Mat last;
int w, h;
int factor;
 
void setup() {
  size(640, 480, P3D);
  background(0);
  factor = 8;
  w = width/factor;
  h = height/factor;
  System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
  cap = new Capture(this, width, height);
  cap.start();
  img = new CVImage(w, h);
  last = new Mat(h, w, CvType.CV_8UC1);
  fill(255);
}
 
void draw() {
  img.copy(cap, 0, 0, cap.width, cap.height, 0, 0, img.width, img.height);
  img.toCV();
  Mat grey = img.getGrey();
  Mat flow = new Mat(last.size(), CvType.CV_32FC2);
  Video.calcOpticalFlowFarneback(last, grey, flow, 0.5, 3, 10, 2, 7, 1.5, Video.OPTFLOW_FARNEBACK_GAUSSIAN);
  grey.copyTo(last);
  background(0);
  drawFlow(flow);
  text("Frame rate: " + nf(round(frameRate), 2), 10, 20, 0);
  grey.release();
  flow.release();
}
 
void drawFlow(Mat _m) {
  for (int y=0; y<_m.rows (); y++) {
    for (int x=0; x<_m.cols (); x++) {
      double [] pt = _m.get(y, x);
      double dy = pt[1];
      double dx = pt[0];
      if (dx == 0 && dy == 0) 
        continue;
      int x1 = x*factor;
      int y1 = y*factor;
      int x2 = round((x+(float)dx) * factor);
      int y2 = round((y+(float)dy) * factor);
      x1 = constrain(x1, 0, cap.width-1);
      y1 = constrain(y1, 0, cap.height-1);
      x2 = constrain(x2, 0, cap.width-1);
      y2 = constrain(y2, 0, cap.height-1);
      color col = cap.pixels[y1*cap.width+x1];
      stroke(col);
      line(x1, y1, 0, x2, y2, 0);
    }
  }
}
 
void captureEvent(Capture _c) {
  _c.read();
}

The technical reference for the algorithm is:

Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Image Analysis, pages 363–370. Springer, 2003.