如何在处理中使用网络摄像头确定上唇和下唇之间的距离？

时间：2016-04-27 07:28:29

标签： opencv image-processing processing video-processing face-detection

我应该从哪里开始？我可以使用Python，Java脚本看到大量的人脸识别和分析，但是如何处理？

我想通过网络摄像头在最高点和最低点之间使用2个点来确定距离，以便在进一步的项目中使用它。

任何帮助将不胜感激

2 个答案:

答案 0 :(得分：2)

如果您想单独处理，可以使用Greg Borenstein's OpenCV for Processing library：

  1. 您可以从Face Detection example
    2. 开始
  2. 检测到脸部后，您可以使用OpenCV.CASCADE_MOUTH检测脸部矩形内的嘴巴。
  3. 一旦你检测到嘴巴，也许你可以使用嘴巴边界框高度逃脱。有关更多详细信息，请使用OpenCV来阈值该矩形。希望张开的嘴巴可以从皮肤的其他部分很好地分割。 Finding contours应该为您提供可以使用的积分列表。

    4. 对于更精确的事情，你可以使用Jason Saragih的CLM FaceTracker, which is available as an OpenFrameworks addon。 OpenFrameworks与Processing有相似之处。如果您在处理中确实需要这种准确性，则可以在后台运行FaceOSC并使用oscP5

    读取处理中的口部坐标

    <强>更新

    对于第一个选项，使用HAAR级联分类器，结果存在一些问题：

    1. OpenCV Processing库可以加载一个级联，第二个实例将覆盖第一个。
    2. OpenCV.CASCADE_MOUTH似乎对闭口效果更好，但开口不太好

      3. 要完成第一个问题，您可以直接使用OpenCV Java API，绕过OpenCV Processing进行多级联检测。

      有几个参数可以帮助检测，例如将手前的边界框的想法作为提示传递给分类器。 我已经在笔记本电脑上使用网络摄像头进行了基本测试，并在不同距离测量面部和嘴部的边界框。这是一个例子：

      import gab.opencv.*;
import org.opencv.core.*;
import org.opencv.objdetect.*;

import processing.video.*;

Capture video;
OpenCV opencv;

CascadeClassifier faceDetector,mouthDetector;
MatOfRect faceDetections,mouthDetections;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
Size minFeatureSizeFace = new Size(50,60);
Size maxFeatureSizeFace = new Size(125,150);
Size minFeatureSizeMouth = new Size(30,10);
Size maxFeatureSizeMouth = new Size(120,60);

// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 4;
//laptop webcam face rectangle
//far, small scale, ~50,60px
//typing distance, ~83,91px
//really close, ~125,150
//laptop webcam mouth rectangle
//far, small scale, ~30,10
//typing distance, ~50,25px
//really close, ~120,60

int mouthHeightHistory = 30;
int[] mouthHeights = new int[mouthHeightHistory]; 

void setup() {
  opencv = new OpenCV(this,320,240);
  size(opencv.width, opencv.height);
  noFill();
  frameRate(30);

  video = new Capture(this,width,height);
  video.start();

  faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_alt2.xml"));
  mouthDetector = new CascadeClassifier(dataPath("haarcascade_mcs_mouth.xml"));

}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);
  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  //detect face using raw Java OpenCV API
  Mat equalizedImg = opencv.getGray();
  faceDetections = new MatOfRect();
  faceDetector.detectMultiScale(equalizedImg, faceDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeFace, maxFeatureSizeFace);
  Rect[] faceDetectionResults = faceDetections.toArray();
  int faces = faceDetectionResults.length;
  text("detected faces: "+faces,5,15);
  if(faces >= 1){
    Rect face = faceDetectionResults[0];
    stroke(0,192,0);
    rect(face.x,face.y,face.width,face.height);
    //detect mouth - only within face rectangle, not the whole frame
    Rect faceLower = face.clone();
    faceLower.height = (int) (face.height * 0.65);
    faceLower.y = face.y + faceLower.height; 
    Mat faceROI = equalizedImg.submat(faceLower);
    //debug view of ROI
    PImage faceImg = createImage(faceLower.width,faceLower.height,RGB);
    opencv.toPImage(faceROI,faceImg);
    image(faceImg,width-faceImg.width,0);

    mouthDetections = new MatOfRect();
    mouthDetector.detectMultiScale(faceROI, mouthDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeMouth, maxFeatureSizeMouth);
    Rect[] mouthDetectionResults = mouthDetections.toArray();
    int mouths = mouthDetectionResults.length;
    text("detected mouths: "+mouths,5,25);
    if(mouths >= 1){
      Rect mouth = mouthDetectionResults[0];
      stroke(192,0,0);
      rect(faceLower.x + mouth.x,faceLower.y + mouth.y,mouth.width,mouth.height);
      text("mouth height:"+mouth.height+"~px",5,35);
      updateAndPlotMouthHistory(mouth.height);
    }
  }
}
void updateAndPlotMouthHistory(int newHeight){
  //shift older values by 1
  for(int i = mouthHeightHistory-1; i > 0; i--){
    mouthHeights[i] = mouthHeights[i-1];
  } 
  //add new value at the front
  mouthHeights[0] = newHeight;
  //plot
  float graphWidth = 100.0;
  float elementWidth = graphWidth / mouthHeightHistory;  
  for(int i = 0; i < mouthHeightHistory; i++){
    rect(elementWidth * i,45,elementWidth,mouthHeights[i]);
  }
}
void captureEvent(Capture c) {
  c.read();
}

      要做的一个非常重要的注意事项：我已将来自OpenCV Processing库文件夹（~/Documents/Processing/libraries/opencv_processing/library/cascade-files）的级联xml文件复制到草图的数据文件夹中。我的草图是OpenCVMouthOpen，因此文件夹结构如下所示：

      OpenCVMouthOpen
├── OpenCVMouthOpen.pde
└── data
    ├── haarcascade_frontalface_alt.xml
    ├── haarcascade_frontalface_alt2.xml
    ├── haarcascade_frontalface_alt_tree.xml
    ├── haarcascade_frontalface_default.xml
    ├── haarcascade_mcs_mouth.xml
    └── lbpcascade_frontalface.xml

      如果您不复制级联文件并按原样使用代码，则不会出现任何错误，但检测不起作用。如果你想检查，你可以做

      println(faceDetector.empty())

      在setup()函数的末尾，如果得到false，则已加载级联，如果得到true，则尚未加载级联。

      您可能需要使用面部和嘴部的minFeatureSize和maxFeatureSize值进行设置。第二个问题，级联不能很好地检测大开口是棘手的。可能有一个已经训练过的开口级联，但你需要找到它。否则，使用这种方法你可能需要自己训练一个，这可能有点单调乏味。

      然而，请注意，当检测到嘴巴时，左侧绘制了一个倒置的情节。在我的测试中，我注意到高度不是非常准确，但图表中有明显的变化。您可能无法获得稳定的嘴高，但通过比较当前与平均的先前高度值，您应该看到一些峰值（值从正到负，反之亦然），这可以让您了解嘴巴张开/近距离变化

      尽管在整个图像中搜索嘴而不是面部可能会稍微慢一些，但准确度较低，这是一个更简单的设置。你可以通过更少的准确性和更多的误报来逃避你的项目，这可能更简单：

      import gab.opencv.*;
import java.awt.Rectangle;
import org.opencv.objdetect.Objdetect;
import processing.video.*;

Capture video;
OpenCV opencv;
Rectangle[] faces,mouths;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
int minFeatureSize = 20;
int maxFeatureSize = 150;
// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 6;

void setup() {
  size(320, 240);
  noFill();
  stroke(0, 192, 0);
  strokeWeight(3);

  video = new Capture(this,width,height);
  video.start();

  opencv  = new OpenCV(this,320,240);
  opencv.loadCascade(OpenCV.CASCADE_MOUTH);
}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);
  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  Rectangle[] mouths = opencv.detect(searchScaleFactor,minNeighbors,flags,minFeatureSize, maxFeatureSize);
  for (int i = 0; i < mouths.length; i++) {
    text(mouths[i].x + "," + mouths[i].y + "," + mouths[i].width + "," + mouths[i].height,mouths[i].x, mouths[i].y);
    rect(mouths[i].x, mouths[i].y, mouths[i].width, mouths[i].height);
  }
}
void captureEvent(Capture c) {
  c.read();
}

      我也提到了分段/阈值处理。这是一个粗略的例子，使用检测到的面部的下半部分只是一个基本阈值，然后是一些基本的形态滤波器（侵蚀/扩张）来清理阈值图像：

      import gab.opencv.*;
import org.opencv.core.*;
import org.opencv.objdetect.*;
import org.opencv.imgproc.Imgproc;
import java.awt.Rectangle;
import java.util.*;

import processing.video.*;

Capture video;
OpenCV opencv;

CascadeClassifier faceDetector,mouthDetector;
MatOfRect faceDetections,mouthDetections;

//cascade detections parameters - explanations from Mastering OpenCV with Practical Computer Vision Projects
int flags = Objdetect.CASCADE_FIND_BIGGEST_OBJECT;
// Smallest object size.
Size minFeatureSizeFace = new Size(50,60);
Size maxFeatureSizeFace = new Size(125,150);

// How detailed should the search be. Must be larger than 1.0.
float searchScaleFactor = 1.1f;
// How much the detections should be filtered out. This should depend on how bad false detections are to your system.
// minNeighbors=2 means lots of good+bad detections, and minNeighbors=6 means only good detections are given but some are missed.
int minNeighbors = 4;
//laptop webcam face rectangle
//far, small scale, ~50,60px
//typing distance, ~83,91px
//really close, ~125,150

float threshold = 160;
int erodeAmt = 1;
int dilateAmt = 5;

void setup() {
  opencv = new OpenCV(this,320,240);
  size(opencv.width, opencv.height);
  noFill();

  video = new Capture(this,width,height);
  video.start();

  faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_alt2.xml"));
  mouthDetector = new CascadeClassifier(dataPath("haarcascade_mcs_mouth.xml"));

}

void draw() {
  //feed cam image to OpenCV, it turns it to grayscale
  opencv.loadImage(video);

  opencv.equalizeHistogram();
  image(opencv.getOutput(), 0, 0 );

  //detect face using raw Java OpenCV API
  Mat equalizedImg = opencv.getGray();
  faceDetections = new MatOfRect();
  faceDetector.detectMultiScale(equalizedImg, faceDetections, searchScaleFactor, minNeighbors, flags, minFeatureSizeFace, maxFeatureSizeFace);
  Rect[] faceDetectionResults = faceDetections.toArray();
  int faces = faceDetectionResults.length;
  text("detected faces: "+faces,5,15);
  if(faces > 0){
    Rect face = faceDetectionResults[0];
    stroke(0,192,0);
    rect(face.x,face.y,face.width,face.height);
    //detect mouth - only within face rectangle, not the whole frame
    Rect faceLower = face.clone();
    faceLower.height = (int) (face.height * 0.55);
    faceLower.y = face.y + faceLower.height; 
    //submat grabs a portion of the image (submatrix) = our region of interest (ROI)
    Mat faceROI = equalizedImg.submat(faceLower);
    Mat faceROIThresh = faceROI.clone();
    //threshold
    Imgproc.threshold(faceROI, faceROIThresh, threshold, width, Imgproc.THRESH_BINARY_INV);
    Imgproc.erode(faceROIThresh, faceROIThresh, new Mat(), new Point(-1,-1), erodeAmt);
    Imgproc.dilate(faceROIThresh, faceROIThresh, new Mat(), new Point(-1,-1), dilateAmt);
    //find contours
    Mat faceContours = faceROIThresh.clone();
    List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
    Imgproc.findContours(faceContours, contours, new Mat(), Imgproc.RETR_EXTERNAL , Imgproc.CHAIN_APPROX_SIMPLE);
    //draw contours
    for(int i = 0 ; i < contours.size(); i++){
      MatOfPoint contour = contours.get(i);
      Point[] points = contour.toArray();
      stroke(map(i,0,contours.size()-1,32,255),0,0);
      beginShape();
      for(Point p : points){
        vertex((float)p.x,(float)p.y);
      }
      endShape();
    }

    //debug view of ROI
    PImage faceImg = createImage(faceLower.width,faceLower.height,RGB);
    opencv.toPImage(faceROIThresh,faceImg);
    image(faceImg,width-faceImg.width,0);
  }
  text("Drag mouseX to control threshold: " + threshold+
      "\nHold 'e' and drag mouseX to control erodeAmt: " + erodeAmt+
      "\nHold 'd' and drag mouseX to control dilateAmt: " + dilateAmt,5,210);
}
void mouseDragged(){
  if(keyPressed){
    if(key == 'e') erodeAmt = (int)map(mouseX,0,width,1,6);
    if(key == 'd') dilateAmt = (int)map(mouseX,0,width,1,10);
  }else{
    threshold = mouseX;
  }
}
void captureEvent(Capture c) {
  c.read();
}

      通过使用YCrCb色彩空间来更好地分割皮肤，可以稍微改善一下，但总的来说，你会注意到有很多变量可以做到，这不会使这个设置变得非常灵活。

      您将使用FaceOSC 更好的结果，并通过oscP5读取处理所需的值。以下是FaceOSCReceiver处理示例的略微简化版本，主要侧重于口：

      import oscP5.*;
OscP5 oscP5;

// num faces found
int found;

// pose
float poseScale;
PVector posePosition = new PVector();


// gesture
float mouthHeight;
float mouthWidth;

void setup() {
  size(640, 480);
  frameRate(30);

  oscP5 = new OscP5(this, 8338);
  oscP5.plug(this, "found", "/found");
  oscP5.plug(this, "poseScale", "/pose/scale");
  oscP5.plug(this, "posePosition", "/pose/position");
  oscP5.plug(this, "mouthWidthReceived", "/gesture/mouth/width");
  oscP5.plug(this, "mouthHeightReceived", "/gesture/mouth/height");
}

void draw() {  
  background(255);
  stroke(0);

  if(found > 0) {
    translate(posePosition.x, posePosition.y);
    scale(poseScale);
    noFill();
    ellipse(0, 20, mouthWidth* 3, mouthHeight * 3);
  }
}

// OSC CALLBACK FUNCTIONS

public void found(int i) {
  println("found: " + i);
  found = i;
}

public void poseScale(float s) {
  println("scale: " + s);
  poseScale = s;
}

public void posePosition(float x, float y) {
  println("pose position\tX: " + x + " Y: " + y );
  posePosition.set(x, y, 0);
}

public void mouthWidthReceived(float w) {
  println("mouth Width: " + w);
  mouthWidth = w;
}

public void mouthHeightReceived(float h) {
  println("mouth height: " + h);
  mouthHeight = h;
}


// all other OSC messages end up here
void oscEvent(OscMessage m) {
  if(m.isPlugged() == false) {
    println("UNPLUGGED: " + m);
  }
}

      在OSX上，您只需下载compiled FaceOSC app即可。 在其他操作系统上，您可能需要setup OpenFrameworks，下载ofxFaceTracker并自行编译FaceOSC。

答案 1 :(得分：-1)

很难回答一般的“我该怎么做”这类问题。 Stack Overflow专为特定的“我试过X，期望Y，但得到Z而不是”类型问题而设计。但我会试着回答一下：

你需要将问题分解成更小的部分。

第1步：您是否可以在草图中显示网络摄像头？一秒钟不要担心计算机视觉的东西。只需连接相机。做一些研究并试一试。

第2步：您能检测该视频中的面部特征吗？您可以尝试自己动手，也可以使用the Processing libraries page的视频和视觉部分中列出的众多库中的一个。

第3步：阅读有关这些库的文档。试试看吧。您可能必须使用每个库制作一堆小示例草图，直到找到您喜欢的一个。 我们无法为您执行此操作，哪一个适合您取决于您。如果您对某些特定内容感到困惑，我们可以尝试为您提供帮助，但我们无法真正帮助您挑选图书馆。

第4步：完成一系列示例程序并选择了一个库后，开始朝着目标努力。你能用图书馆检测面部特征吗？ 只是那部分正常工作。一旦你有了这个工作，你能看到开口或闭嘴等变化吗？

一次完成一小步。如果您遇到问题，请发布一个MCVE以及一个特定的技术问题，我们将从那里开始。祝你好运。

相关问题
最新问题