OpenCV Viola & Jones object detection in MATLAB

In image processing, one of the most successful object detectors devised is the Viola and Jones detector, proposed in their seminal CVPR paper in 2001. A popular implementation used by image processing researchers and implementers is provided by the OpenCV library. In this post, I’ll show you how run the OpenCV object detector in MATLAB for Windows. You should have some familiarity with OpenCV and with the Viola and Jones detector to work through this tutorial.

Steps in the object detector

MATLAB is able to call functions in shared libraries. This means that, using the compiled OpenCV DLLs, we are able to directly call various OpenCV functions from within MATLAB. The flow of our MATLAB program, including the required OpenCV external function calls (based on this example), will go something like this:

1. cvLoadHaarClassifierCascade: Load object detector cascade
2. cvCreateMemStorage: Allocate memory for detector
3. cvLoadImage: Load image from disk
4. cvHaarDetectObjects: Perform object detection
5. For each detected object:
   1. cvGetSeqElem: Get next detected object of type cvRect
   2. Display this detection result in MATLAB
6. cvReleaseImage: Unload the image from memory
7. cvReleaseMemStorage: De-allocate memory for detector
8. cvReleaseHaarClassifierCascade: Unload the cascade from memory

Loading shared libraries

The first step is to load the OpenCV shared libraries using MATLAB’s loadlibrary() function. To use the functions listed in the object detector steps above, we need to load the OpenCV libraries cxcore100.dll, cv100.dll and highgui100.dll. Assuming that OpenCV has been installed to "C:\Program Files\OpenCV", the libraries are loaded like this:

opencvPath = 'C:\Program Files\OpenCV';
includePath = fullfile(opencvPath, 'cxcore\include');

loadlibrary(...
	fullfile(opencvPath, 'bin\cxcore100.dll'), ...
	fullfile(opencvPath, 'cxcore\include\cxcore.h'), ...
		'alias', 'cxcore100', 'includepath', includePath);
loadlibrary(...
	fullfile(opencvPath, 'bin\cv100.dll'), ...
	fullfile(opencvPath, 'cv\include\cv.h'), ...
		'alias', 'cv100', 'includepath', includePath);
loadlibrary(...
	fullfile(opencvPath, 'bin\highgui100.dll'), ...
	fullfile(opencvPath, 'otherlibs\highgui\highgui.h'), ...
		'alias', 'highgui100', 'includepath', includePath);

You will get some warnings; these can be ignored for our purposes. You can display the list of functions that a particular shared library exports with the libfunctions() command in MATLAB For example, to list the functions exported by the highgui library:

>> libfunctions('highgui100')

Functions in library highgui100:

cvConvertImage             cvQueryFrame
cvCreateCameraCapture      cvReleaseCapture
cvCreateFileCapture        cvReleaseVideoWriter
cvCreateTrackbar           cvResizeWindow
cvCreateVideoWriter        cvRetrieveFrame
cvDestroyAllWindows        cvSaveImage
cvDestroyWindow            cvSetCaptureProperty
cvGetCaptureProperty       cvSetMouseCallback
cvGetTrackbarPos           cvSetPostprocessFuncWin32
cvGetWindowHandle          cvSetPreprocessFuncWin32
cvGetWindowName            cvSetTrackbarPos
cvGrabFrame                cvShowImage
cvInitSystem               cvStartWindowThread
cvLoadImage                cvWaitKey
cvLoadImageM               cvWriteFrame
cvMoveWindow
cvNamedWindow

The first step in our object detector is to load a detector cascade. We are going to load one of the frontal face detector cascades that is provided with a normal OpenCV installation:

classifierFilename = 'C:/Program Files/OpenCV/data/haarcascades/haarcascade_frontalface_alt.xml';
cvCascade = calllib('cv100', 'cvLoadHaarClassifierCascade', classifierFilename, ...
	libstruct('CvSize',struct('width',int16(100),'height',int16(100))));

The function calllib() returns a libpointer structure containing two fairly self-explanatory fields, DataType and Value. To display the return value from cvLoadHaarClassifierCascade(), we can run:

>> cvCascade.Value

ans =

               flags: 1.1125e+009
               count: 22
    orig_window_size: [1x1 struct]
    real_window_size: [1x1 struct]
               scale: 0
    stage_classifier: [1x1 struct]
         hid_cascade: []

The above output shows that MATLAB has successfully loaded the cascade file and returned a pointer to an OpenCV CvHaarClassifierCascade object.

Prototype M-files

We could now continue implementing all of our OpenCV function calls from the object detector steps like this, however we will run into a problem when cvGetSeqElem is called. To see why, try this:

libfunctions('cxcore100', '-full')

The -full option lists the signatures for each imported function. The signature for the function cvGetSeqElem() is listed as:

[cstring, CvSeqPtr] cvGetSeqElem(CvSeqPtr, int32)

This shows that the return value for the imported cvGetSeqElem() function will be a pointer to a character (cstring). This is based on the function declaration in the cxcore.h header file:

CVAPI(char*)  cvGetSeqElem( const CvSeq* seq, int index );

However, in step 5.1 of our object detector steps, we require a CvRect object. Normally in C++ you would simply cast the character pointer return value to a CvRect object, but MATLAB does not support casting of return values from calllib(), so there is no way we can cast this to a CvRect.

The solution is what is referred to as a prototype M-file. By constructing a prototype M-file, we can define our own signatures for the imported functions rather than using the declarations from the C++ header file.

Let’s generate the prototype M-file now:

loadlibrary(...
	fullfile(opencvPath, 'bin\cxcore100.dll'), ...
	fullfile(opencvPath, 'cxcore\include\cxcore.h'), ...
		'mfilename', 'proto_cxcore');

This will automatically generate a prototype M-file named proto_cxcore.m based on the C++ header file. Open this file up and find the function signature for cvGetSeqElem and replace it with the following:

% char * cvGetSeqElem ( const CvSeq * seq , int index );

fcns.name{fcnNum}='cvGetSeqElem'; fcns.calltype{fcnNum}='cdecl'; fcns.LHS{fcnNum}='CvRectPtr'; fcns.RHS{fcnNum}={'CvSeqPtr', 'int32'};fcnNum=fcnNum+1;

This changes the return type for cvGetSeqElem() from a char pointer to a CvRect pointer.

We can now load the library using the new prototype:

loadlibrary(...
	fullfile(opencvPath, 'bin\cxcore100.dll'), ...
		@proto_cxcore);

An example face detector

We now have all the pieces ready to write a complete object detector. The code listing below implements the object detector steps listed above to perform face detection on an image. Additionally, the image is displayed in MATLAB and a box is drawn around any detected faces.

opencvPath = 'C:\Program Files\OpenCV';
includePath = fullfile(opencvPath, 'cxcore\include');
inputImage = 'lenna.jpg';

%% Load the required libraries

if libisloaded('highgui100'), unloadlibrary highgui100, end
if libisloaded('cv100'), unloadlibrary cv100, end
if libisloaded('cxcore100'), unloadlibrary cxcore100, end

loadlibrary(...
	fullfile(opencvPath, 'bin\cxcore100.dll'), @proto_cxcore);
loadlibrary(...
	fullfile(opencvPath, 'bin\cv100.dll'), ...
	fullfile(opencvPath, 'cv\include\cv.h'), ...
		'alias', 'cv100', 'includepath', includePath);
loadlibrary(...
	fullfile(opencvPath, 'bin\highgui100.dll'), ...
	fullfile(opencvPath, 'otherlibs\highgui\highgui.h'), ...
		'alias', 'highgui100', 'includepath', includePath);

%% Load the cascade

classifierFilename = 'C:/Program Files/OpenCV/data/haarcascades/haarcascade_frontalface_alt.xml';
cvCascade = calllib('cv100', 'cvLoadHaarClassifierCascade', classifierFilename, ...
	libstruct('CvSize',struct('width',int16(100),'height',int16(100))));

%% Create memory storage

cvStorage = calllib('cxcore100', 'cvCreateMemStorage', 0);

%% Load the input image

cvImage = calllib('highgui100', ...
	'cvLoadImage', inputImage, int16(1));
if ~cvImage.Value.nSize
	error('Image could not be loaded');
end

%% Perform object detection

cvSeq = calllib('cv100', ...
	'cvHaarDetectObjects', cvImage, cvCascade, cvStorage, 1.1, 2, 0, ...
	libstruct('CvSize',struct('width',int16(40),'height',int16(40))));

%% Loop through the detections and display bounding boxes

imshow(imread(inputImage)); %load and display image in MATLAB
for n = 1:cvSeq.Value.total
	cvRect = calllib('cxcore100', ...
		'cvGetSeqElem', cvSeq, int16(n));
	rectangle('Position', ...
		[cvRect.Value.x cvRect.Value.y ...
		cvRect.Value.width cvRect.Value.height], ...
		'EdgeColor', 'r', 'LineWidth', 3);
end

%% Release resources

calllib('cxcore100', 'cvReleaseImage', cvImage);
calllib('cxcore100', 'cvReleaseMemStorage', cvStorage);
calllib('cv100', 'cvReleaseHaarClassifierCascade', cvCascade);

As an example, the following is the output after running the detector above on a greyscale version of the Lenna test image:

Note: If you get a segmentation fault attempting to run the code above, try evaluating the cells one-by-one (e.g. by pressing Ctrl-Enter) – it seems to fix the problem.