mathnathan

Alpha Blending is a technique to merge/blend two images or ROI’s (Region Of Interest) together. It is done by taking two arrays and applying weights to each. With this, the AddWeighted() function calculates the new pixel values in the blended ROI. Here is a simple porgam I used to implement an alpha blend.

# include <cv.h>
# include <highgui.h>

int main( int argc, char** argv ) {
	IplImage *src1, *src2;
	if( argc == 9 && ((src1=cvLoadImage(argv[1],1)) != 0
		)&&((src2=cvLoadImage(argv[2],1)) != 0))
		{
			int x = atoi(argv[3]);
			int y = atoi(argv[4]);
			int width = atoi(argv[5]);
			int height = atoi(argv[6]);
			double alpha = (double)atof(argv[7]);
			double beta = (double)atof(argv[8]);
			cvSetImageROI(src1, cvRect(x,y,width,height));
			cvSetImageROI(src2, cvRect(x,y,width,height));
			cvAddWeighted(src1, alpha, src2, beta, 0.0, src1);
			cvResetImageROI(src1);
			cvNamedWindow( "Alpha_blend", 1 );
			cvShowImage( "Alpha_blend", src1 );
			cvWaitKey();
		}
	return 0;
}

Get the code from my Git Repository
If you need help with Git, follow this quick introduction – Getting Started With Git

This program takes in eight arguments…

argv[1] - The destination image.
argv[2] - Source image to be blended onto the destination image.
argv[3], argv[4] - The (x,y) coordinate of the ROI in both the
                   destination image and the source image to be blended.
argv[5], argv[6] - Width and Height from the point (x,y) of the ROI.
argv[7] - Alpha, the weight of argv[1].
argv[8] - Beta, the weight of argv[2] to be blended onto argv[1].

For this simple example I didn’t have any applications in mind, I just wanted to see some images blended together. So the (x,y) coordinate and the width and height for the ROI are the same for both images.

The first new function that I used here was the cvSetImageROI…

void cvSetImageROI( IplImage* image, CvRect rect );

image - Image header.
rect - ROI rectangle.

This function sets the ROI (Region Of Interest) for a particular image. This happens inside the IplImage structure. Recall there was a data member…

 struct _IplROI *roi;

in the IplImage which is a pointer to the _IplROI struct and looks like this…

typedef struct _IplROI
{
     int coi;
     int xOffset;
     int yOffset;
     int width;
     int height;
} IplROI;

Notice this is just information for a subregion of the original image, where coi is channel of interest. To set that subregion I needed a rectangular region of the original image, and for this I used a CvRect structure…

typedef struct CvRect
{
     int x;
     int y;
     int width;
     int height;
} CvRect;

This is pretty self explanatory…

The way it is initialized is even easier!

CvRect  cvRect( int x, int y, int width, int height )
{
    CvRect r;

    r.x = x;
    r.y = y;
    r.width = width;
    r.height = height;

    return r;
}

There is kind of a lot going on here in this one line.

cvSetImageROI(src1, cvRect(x,y,width,height));

First calling cvRect as we can see returns a cvRect structure. The cvSetImageROI function expects a cvRect struct to be its second parameter, so it wouldn’t matter if I defined the cvRect in or out of the function call, as long as I’m sure to pass it in there. Then, cvSetImageROI takes the image passed in and sets its roi pointer to the region defined by the cvRect structure.

For example, say I create a pointer to an IplImage structure called img, then I set the roi pointer to some rectangular subregion of the image. Under these circumstances if I pass around the img pointer whenever I reference it’s picture, it will not refer to the original image, but instead it will reference the roi that was set.

The IplImage structure doesn’t loose the original image though, to set it so that it references the original again just call cvResetImageROI() which points the roi at NULL again.

void cvResetImageROI( IplImage* image );

image - Image header

Back to the program, once both of the roi’s are set for each image that is when the blending happens. I’ll have it noted that with the roi’s set to NULL you can still pass in both images and, provided they are the same size, both images in their entirety will be blended. It’s a pretty simple concept, the destination image is transformed following this simple calculation…

dst = src1*alpha + src2*beta + gamma

So I know alpha, beta, and gamma are doubles, what are src1, and src2? Well they are arrays, or matrices to be more precise. In each of their entries they store the RGB values of each pixel, and as our handy dandy linear algebra taught us, multiplying a matrix by a scalar will result in the multiplication of each entry with that scalar. This means I’m just changing the value of each pixel to be a mix (to the certain alpha and beta values) of the two images! Here is it’s declaration.

void  cvAddWeighted( const CvArr* src1, double alpha,
                     const CvArr* src2, double beta,
                     double gamma, CvArr* dst );

src1 - The first source array.
alpha - Weight of the first array elements.
src2 - The second source array.
beta - Weight of the second array elements.
gamma - Scalar, added to each sum.
dst - The destination array.

CvArr is a peculiar data type. I went rummaging through the source and this is what I found out about it.

typedef void CvArr;
/* CvArr* is used to pass arbitrary
 * array-like data structures
 * into functions where the particular
 * array type is recognized at runtime:
 */

Ya, so take that however you’d like.

Alright so how did it work? Well I’ll show ya!

There it is! It’s no longer a secret…. Yes, I am Dr. Manhattan!

References

I’m studying from Gary Bradski’s Learning OpenCV

From my perspective one of the most intriguing applications of all this simple image and file handling so far is reading in data from a webcam. I was pleasantly surprised at how easy it was using OpenCV. Below is the code I used to read in information from a webcam and save it to a file.

# include "highgui.h"
# include "cv.h"

int main( int argc, char** argv ) {
	CvCapture* capture;

	capture = cvCreateCameraCapture(0);

	assert( capture != NULL );

	IplImage* bgr_frame = cvQueryFrame( capture );

	CvSize size = cvSize(
			(int)cvGetCaptureProperty( capture,
                            CV_CAP_PROP_FRAME_WIDTH),
			(int)cvGetCaptureProperty( capture,
                            CV_CAP_PROP_FRAME_HEIGHT)
			);

	cvNamedWindow( "Webcam", CV_WINDOW_AUTOSIZE );

	CvVideoWriter *writer = cvCreateVideoWriter( argv[ 1 ],
                            CV_FOURCC('D','I','V','X'),
                            30,
                            size
                            );

	while( (bgr_frame = cvQueryFrame( capture )) != NULL ) {
		cvWriteFrame( writer, bgr_frame );
		cvShowImage( "Webcam", bgr_frame );
		char c = cvWaitKey( 33 );
		if( c == 27 ) break;
	}
	cvReleaseVideoWriter( &writer );
	cvReleaseCapture( &capture );
	cvDestroyWindow( "Webcam" );
	return( 0 );
}

Get the code from my Git Repository
If you need help with Git, follow this quick introduction – Getting Started With Git

So the idea behind the program is this… First I wanted to make a pointer to a CvCapture structure, because I’m going to need somehwere to store the webcam information and capture feed. When I initialized the new capture structure, I used a new function CreateCameraCapture…

CvCapture* cvCreateCameraCapture( int index )

index - The camera device number
        Index   Device
          0     /dev/video0
          1     /dev/video1
          2     /dev/video2
          3     /dev/video3
          ...
          7     /dev/video7
          with
          -1    /dev/video

So I initialized the structure with my webcam /dev/video0, any number can be used, and the -1 will choose one arbitrarily.

Next I needed an IplImage structure pointer for displaying each frame captured by my webcam, but I wanted to do more than just display it. I wanted to write each frame down into a file as the camera was running. For the CreateVideoWriter function, which we’ll see momentarily, I needed the size of the images being captured by my webcam. There is another structure for storing the sizes, CvSize. It’s declared inside cxtypes.h as simply as it sounds.

typedef struct
{
    int width;
    int height;
}
CvSize;

After declaring a variable of type CvSize, I instantiated it with the cvSize function.

CvSize cvSize( int width, int height )
{
      CvSize s;
      s.width = width;
      s.height = height;
      return s;
}

It’s as easy as that, I don’t think that needs much explanation.

What does deserve an explanation is the super handy dandy cvGetCaptureProperty function. Notice I used it to get the height and width of my camera’s captured images and set that as the height and width of the new CvSize structure. Here is how it works.

double cvGetCaptureProperty(CvCapture* capture, int property_id);

capture - This is the structure containing all the information about
           it's video/camera/movie
property_id - The property information desried. 

Can be any of the following: CV_CAP_PROP_POS_MSEC       0
                             CV_CAP_PROP_POS_FRAMES     1
                             CV_CAP_PROP_POS_AVI_RATIO  2
                             CV_CAP_PROP_FRAME_WIDTH    3
                             CV_CAP_PROP_FRAME_HEIGHT   4
                             CV_CAP_PROP_FPS            5
                             CV_CAP_PROP_FOURCC         6
                             CV_CAP_PROP_FRAME_COUNT    7
                             CV_CAP_PROP_FORMAT         8
                             CV_CAP_PROP_MODE           9
                             CV_CAP_PROP_BRIGHTNESS    10
                             CV_CAP_PROP_CONTRAST      11
                             CV_CAP_PROP_SATURATION    12
                             CV_CAP_PROP_HUE           13
                             CV_CAP_PROP_GAIN          14
                             CV_CAP_PROP_EXPOSURE      15
                             CV_CAP_PROP_CONVERT_RGB   16
                             CV_CAP_PROP_WHITE_BALANCE 17
                             CV_CAP_PROP_RECTIFICATION 18

I find this to be a great function, one worth memorizing. Take note that it returns a double, so I needed to static cast them to integers for the cvSize function to accept them.

Inside the loop I’m thinking, I want to be writing the frames from the video feed before I display them. This is from remembering that IplImage structures instantiated to a CvCapture’s frames are released immediately after the function ends. So I’d have to write each frame down before they were released, and to write an image I needed a writer…

typedef struct CvVideoWriter CvVideoWriter;
CvVideoWriter* cvCreateVideoWriter(
                           const char* filename,
                           int fourcc,
                           double fps,
                           CvSize frame_size
                           );

filename - Name of the output video file.
fourcc - 4-character code of codec used to compress the frames.
         For example,
             CV_FOURCC('P','I','M','1') is MPEG-1 codec,
             CV_FOURCC('M','J','P','G') is motion-jpeg codec etc.
         Under Win32 it is possible to pass -1 in order to choose
         compression method and additional compression
         parameters from dialog.
fps - Framerate of the created video stream.
frame_size - Size of video frames.

I had some trouble with this function for a bit. The codec you choose to compress the frames must be on your computer. I put in a codec I believed was on my system only to find it not writing the file, after a while of searching and reading through the source, I found the problem which I recorded in my OpenCV Errors section.

I know what you’re thinking. “Hey! You put the fps in manually to 30, but I remember seeing a CV_CAP_PROP_FPS, why didn’t you use the cvGetCaptureProperty function and store it in a double and put it in that way?” Well I tried that. For a reason I’m not sure of yet, my webcam doesn’t seem to be sharing that information nicely with my computer, so I was receiving an error telling me it couldn’t read the fps from the CvCapture structure. Oh well. 30 fps seems to be working fine.

So I’m ready for the loop. I get it going with some new pizazz, but the logic is the same. As planned I write the frame first, then display it.

int cvWriteFrame( CvVideoWriter* writer, const IplImage* image );

writer - Video writer structure.
image - The frame to be written.

It does precisely what it says, takes in an image and writes it to the file that was given to the CreateVideoWriter function. Then show the image, and do that standard clean up, including releasing the new CvVideoWriter with the usual approach.

void cvReleaseVideoWriter( CvVideoWriter** writer );

writer - Pointer to the video file writer structure.

That’s that. Now by running this program a window will pop up displaying everything the webcam sees and save it to a file given as the first parameter to the main function. You can use the video player that was written in the previous post Basic Video Manipulation to test out the newly recorded video.

References

I’m studying from Gary Bradski’s Learning OpenCV

Video Handling Basics

Understanding how to manipulate images was critical to learning video manipulation. This is because a video is just a loop of images! So toying with videos was a piece a pie after getting comfortable with image handling. Since I learned how to load images and display them, the only thing I needed to play videos was a loop. The next program should look very similar to the first two…

# include "stdio.h"
# include "highgui.h"

int main( int argc, char** argv ) {
	IplImage* frame = 0;

	if( argc < 2 ) {
		printf( "Usage: Accepts one video as argument\n" );
		exit( 0 );
	}

	cvNamedWindow( "Example2", CV_WINDOW_AUTOSIZE );
	cvMoveWindow( "Example2", 100, 100);
	CvCapture* capture = cvCreateFileCapture( argv[1] );

	while(1) {
	    frame = cvQueryFrame( capture );
	    if( !frame) break;
	    cvShowImage( "Example2", frame );
	    char c = cvWaitKey(33);
	    if( c == 27 ) break;
	}

	cvReleaseCapture( &capture );
	cvDestroyWindow( "Example2" );
}

Get the code from my Git Repository
If you need help with Git, follow this quick introduction – Getting Started With Git

First I created another window “Example2″ with my usual approach, no problems here.

Dealing with videos seemed simple enough in theory, but I was slightly taken aback at it’s complexity once I delved into the source files on my computer. CvCapture is another data structure, which has a complicated definition. All of it’s members are declared virtually and has different capture objects it uses for different situations. (of which, I don’t know yet…)

The way I loaded a video’s data into the CvCapture data structure is through the CreateFileCapture function. After looking through the source, this function calls the other functions with various usages, they were of the form

CvCapture* cvCreateFileCapture_XXX( const char* filename );

filename - Filename of the video you want to pass in.

where the XXX was different names. I just took note to remember this function takes a pointer to the filename of the video I’m trying to load.

Now remembering that a video is just a loop of images, I could be sure I was going to need an IplImage structure to store each one in. After creating the image variable, there was really nothing left but to jump into the loop.

while(1) {
   frame = cvQueryFrame( capture );
   if( !frame) break;
   cvShowImage( "Example2", frame );
   char c = cvWaitKey(33);
   if( c == 27 ) break;
}

Now inside the loop, my thinking process is this. I want to take each frame from the CvCapture data structure and display them one at a time, quickly. The first step was extracting each frame. I accomplished this using the QueryFrame function.

 IplImage* cvQueryFrame( CvCapture* capture );

capture - Here you pass the video structure.

It takes a pointer to a CvCapture structure, parses through the data and returns a pointer to each frame which is an IplImage structure. This function is just a combination of cvGrabFrame and cvRetrieveFrame in one call. Pretty handy, so I just stored it in frame, my IplImage pointer.

I then displayed the image in my “Example2″ window using my old friend cvShowImage. My question was “Okay, how long do I let this image sit here?”. After a little reading a good time to assume without knowledge of the captures frame rate it 30ms. the other 3 milliseconds are waiting for input (the ecs key == 27 in ASCII). There is a way to extract the correct frame rate from the CvCapture structure, but I’ve yet to glean this procedure…

char c = cvWaitKey(33);
if( c == 27 ) break;

Then if the esc key is pressed I could exit the loop. The loop only has two exit methods.

if( !frame ) break;
if( c == 27 ) break;

Then the usual clean ups were done, minus one. Notice I didn’t kill any of the IplImage structures? Apparently the CvCapture structure took care of that for me, how kind. Anyway, trying to release information that isn’t there is clearly not a good idea!

I saved the file, edited my CMakeLists.txt to accommodate the extra program and built it. I tested it on the sample video tree.avi and some other vids on my hard drive. You may need to finagle with the frame rate a bit to get a good flow for now.

It would be wise to put more exit procedures and provide more error checking. However, this was boring… I wanted to get to the fun stuff.

The Fun Stuff

As with the images, once I got my video basics down I wanted to try some more simple manipulations. My thought, play a video I normally enjoy with the previous procedure, then play another one next to it with each frame being blurred.

Alright so here it is.

# include "stdio.h"
# include "cv.h"
# include "highgui.h"

int main( int argc, char** argv ) {
	IplImage* frame = 0;
	IplImage* augframe = 0;

	if( argc < 2 ) {
		printf( "Usage: Accepts one video as argument\n" );
		exit( 0 );
	}

	cvNamedWindow( "Example2", CV_WINDOW_AUTOSIZE );
	cvMoveWindow( "Example2", 100, 100);
	cvNamedWindow( "output", CV_WINDOW_AUTOSIZE );
	cvMoveWindow( "output", 500, 100);
	CvCapture* capture = cvCreateFileCapture( argv[1] );

	while(1) {
	    frame = cvQueryFrame( capture );
	    if( !frame) break;
	    augframe = cvCreateImage( cvGetSize( frame ),
                                 IPL_DEPTH_8U, 3 );
		cvSmooth( frame, augframe, CV_BLUR, 3, 3 );
	    cvShowImage( "Example2", frame );
	    cvShowImage( "output", augframe );
	    char c = cvWaitKey(73);
	    if( c == 27 ) break;
	}

	cvReleaseCapture( &capture );
	cvDestroyWindow( "Example2" );
}

Get the code from my Git Repository
If you need help with Git, follow my quick introduction – Getting Started With Git

Everything highlighted I’ve used before, so there is nothing new to talk about. I used the same cvSmooth function from the previous post, I just chose to use another one of the smoothing procedures that was listed in the table. CV_BLUR was my choice, I don’t see much of a difference between it and the CV_GAUSSIAN. I suppose it’s just a different algorithm.

I did want to note one thing though. Recall that the CvCapture structure is taking care of releasing the frames for me now, this means that as soon as the call cvShowImage terminated, the image stored in frame was released… This meant that in order to create the augframe from the original using the cvCreateImage function it had to be done before frame was displayed!

I say try this out with some videos on your computer. Then perhaps take a look at some of the other filters, and what they do.

References

I’m studying from Gary Bradski’s Learning OpenCV