I finished this project last year and it’s been ‘dying’ to be posted. My goal was to take point clouds obtained from two Kinects and register them into one coordinate system. After the registration process was completed, the new point cloud would be much more robust with many of the obstructed blank spots filled in.
The project follows a simple algorithm.
- Use libfreenect to obtain point clouds from both Kinects
- Use OpenGL to display the point clouds in an interactice virtual environment
- Use OpenCV to display the RGB streams for the user to select correspondences
- Calculate the transformations using Procrustes Analysis and the correspondence matrices
- Apply the translation and rotation to the point clouds visualized in OpenGL
The implementation of this algorithm can be found here.
After a few weeks of stagnant development with the project, I made a bet with my friends and advisor that I could finish the project in one weekend before I left for a vacation. Below is the video which resulted from my sleepless weekend hackathon.
After the project was finished. I used it to complete what is called an “Honor’s Thesis” here at my University. It is an undergraduate research project, which once defended successfully allows the student to graduate “with honors” on their diploma. Looking back on the thesis now I would have done things differently – but isn’t that almost always the case. None-the-less it was a mile stone in my life and it is my work.
Let me know if you’d like a copy of it and I’ll be happy to send it to you!
I watched the video and, whoa! Very nice stuff! I would like to see the copy of your thesis, if possible
Thanks and good job!
Thanks for the feedback! I’m working on adapting this to some interesting immersive visualizations for my master’s work. I sent the paper, let me know what you think!
Looks cool. Did you implement your 3D render for the point cloud with openGL. Could you send me the code of that part?
Thanks for the feedback Brian. Yes I did, the project is hosted on my github – https://github.com/mathnathan/Kinect-Registration
Thanks for sharing, Nathan. Really nice job. Looking forward to your Thesis.
Hi,
I’m interested with your project about using this dual kinect
Can you share your thesis to me as well ?
Looking forward to your Thesis
Thank you
I have tried your code which is downloaded from https://github.com/mathnathan/Kinect-Registration and built in my Ubuntu 12.04 and using OpenCV2.4
But, I got an error like :
”
[22%] Built target freenect
[22%] Built target freenect_sync
Linking C shared library ../../../../lib/libfreenect_cv.so
/usr/bin/ld: cannot find -lcv
collect2: ld returned 1 exit status
make[2]: *** [lib/libfreenect_cv.so.0.0.1] Error 1
make[1]: *** [include/libfreenect/wrappers/opencv/CMakeFiles/freenect_cv.dir/all] Error 2
make: *** [all] Error 2
”
Have tried with other PC , but still got the same result as above
Is there any suggestion to solve this problem ?
Hey Edmond, It’s nice to see your interest in my project. I haven’t looked at it in a very long time. Perhaps tomorrow after work I will have sometime to sit down and try to recompile it in Ubuntu 12.04. I’ll keep you posted, and be expecting an email shortly with my thesis. Cheers!
Thank you for your prompt reply.
Btw, I’m using your “libfreenect” which is inside of the folder “include” of Kinect – Registration
When I tried building with “cmake ..” , it seems needs two other folders like “example” and “fakenect” instead of “include” folder.
Also, it needs the folder of “cpp” for the inside of folder “wrappers”.
So, I copy the folders from https://github.com/OpenKinect/libfreenect then put it in inside of the folder your “libfreenect”.
Then, I got the result like as above
But, I got an error like :
”
[22%] Built target freenect
[22%] Built target freenect_sync
Linking C shared library ../../../../lib/libfreenect_cv.so
/usr/bin/ld: cannot find -lcv
collect2: ld returned 1 exit status
make[2]: *** [lib/libfreenect_cv.so.0.0.1] Error 1
make[1]: *** [include/libfreenect/wrappers/opencv/CMakeFiles/freenect_cv.dir/all] Error 2
make: *** [all] Error 2
”
Hope it is clear.
Looking forward to see your thesis and the message for solving this problem.
Hi!Thats so awesome!I’m doing a final year (undergrad project too!) in point clouds.Its completely a new field for me and I’m overwhelmed by what platform to work with.I have a windows 64bit and am not sure whether to use cpp or Matlab?May I please get a copy of your thesis?Thanks!
Hi Nathan, I am doing something similar, may I have a look on your thesis.
Thanks very much!
Hi Nathan,
This looks very interesting. I’m working on geomorphological applications, either with multiple Kinects or one Kinect from multiple positions. I’d be fascinated to read your thesis, as it looks like you may save duplication of effort.
Ben
Pingback: Registration of Multiple Kinects | Visualization Lab at FSU
Nice work, thanks for sharing! Have you considered using 2D features like SIFT to automatize correspondence search?
I am very interested in a copy of your thesis!
Thanks!
Thanks Alex. Actually, YES! That was my first approach and in my thesis I outline the process. Unfortunately it was not as robust as the interactive matching and I needed results. I have discontinued working on this, however people in the lab here in my department have picked up my work and made some very interesting progress. We now use the PCL library and do registration using segmented planes from the image. We’ll be sure to blog about it soon on the Visualization Lab website. I’ll take your email and update you when we do. Thanks for reaching out Alex.
Hey Ben, thanks for reaching out. Sorry for the delayed reply on things, I’ve been slaving away on my graduate thesis. My work is old news now. We’ve made big progress since then on registration and 3D recording, I’ll take your email and update you when we post some of our work on the Lab’s blog. Cheers!
Hi, I am very interested in your theory, could u plz send it to me?
Very nice work!
I would like to see the copy of your thesis
However, I’m working on similar project with two Kinects using PCL (http://pointclouds.org/)
Do you have any suggestion or advise about that?
Thank you!
Hello Nathan,
U done really nice work.. i am very much curious to know how u done that..i want to go through ur graduation thesis and updated work in detail.. if possible n time permits to u den let me know what will be the potential applications for 3D reconstruction using multiple depth cameras..i am eagerly waiting for ur reply…my self also want to work with multiple depth cameras but unfortunately i am unfamiliar with linux ..so i have plan to work on windows platform using point cloud library..
Hi Nathan,
I like your work its very interesting. I’d like to read your thesis, can send me on my mail..
tanajikamble13@gmail.com
Hello,
I successfully build the program but i am unable to make executable file. Please let me know where i am getting wrong. I am very interested in you work and eagerly waiting for ur reply
sincos@sincos-300E4C-300E5C-300E7C:~/Desktop/Kinect-Registration-master/Build$ make
[100%] Building CXX object CMakeFiles/kinect_reg.dir/kinReg.cpp.o
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:48:1: error: ‘vector’ does not name a type
vector P_pts, Q_pts; // This is how they’re collected
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:80:31: error: ‘vector’ does not name a type
Mat convert_vector2Mat( const vector vec );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:80:31: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:80:37: error: expected ‘,’ or ‘…’ before ‘<’ token
Mat convert_vector2Mat( const vector vec );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:84:24: error: ‘vector’ does not name a type
void procrustes( const vector&, const vector&, Mat&, Mat& );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:84:24: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:84:30: error: expected ‘,’ or ‘…’ before ‘<’ token
void procrustes( const vector&, const vector&, Mat&, Mat& );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:99:1: error: ‘vector’ does not name a type
vector rgbCV;
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:100:1: error: ‘vector’ does not name a type
vector depthCV;
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘int main(int, char**)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:106:9: error: ‘rgbCV’ was not declared in this scope
rgbCV.push_back( freenect_sync_get_rgb_cv(cam) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:107:9: error: ‘depthCV’ was not declared in this scope
depthCV.push_back( freenect_sync_get_depth_cv(cam) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:121:60: error: ‘namedWindow’ was not declared in this scope
namedWindow( “Camera 0 | Camera 1″, CV_WINDOW_AUTOSIZE );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void cbRender()’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:145:5: error: ‘rgbCV’ was not declared in this scope
rgbCV.clear();
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:146:5: error: ‘depthCV’ was not declared in this scope
depthCV.clear();
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void cbKeyPressed(unsigned char, int, int)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:196:21: error: ‘P_pts’ was not declared in this scope
procrustes( P_pts, Q_pts, trans, rot );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:196:28: error: ‘Q_pts’ was not declared in this scope
procrustes( P_pts, Q_pts, trans, rot );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void loadBuffers(int, unsigned int (*)[640], short int (*)[640][3], unsigned char (*)[640][3])’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:273:5: error: ‘rgbCV’ was not declared in this scope
rgbCV.push_back( freenect_sync_get_rgb_cv(cameraIndx) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:274:5: error: ‘depthCV’ was not declared in this scope
depthCV.push_back( freenect_sync_get_depth_cv(cameraIndx) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:285:53: error: expected primary-expression before ‘>’ token
Vec3b color = rgbCV[cameraIndx].at(row,col);
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘float getDepth(int, int, int)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:332:23: error: ‘depthCV’ was not declared in this scope
float d = (float)(depthCV[cam].at(x,y));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:332:39: error: expected primary-expression before ‘short’
float d = (float)(depthCV[cam].at(x,y));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:332:39: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:335:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x,y+1));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:335:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:338:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x,y-1));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:338:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:341:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x+1,y));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:341:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:344:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x-1,y));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:344:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:347:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x-1,y-1));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:347:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:350:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x-1,y+1));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:350:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:353:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x+1,y-1));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:353:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:356:37: error: expected primary-expression before ‘short’
d = (float)(depthCV[cam].at(x+1,y+1));
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:356:37: error: expected ‘)’ before ‘short’
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void displayCVcams()’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:415:19: error: ‘rgbCV’ was not declared in this scope
cvtColor( rgbCV[cam], tmp, CV_RGB2BGR );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:415:47: error: ‘cvtColor’ was not declared in this scope
cvtColor( rgbCV[cam], tmp, CV_RGB2BGR );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:419:27: error: ‘rgbCV’ was not declared in this scope
Mat rgb = joinFrames( rgbCV[0], rgbCV[1] );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:420:40: error: ‘imshow’ was not declared in this scope
imshow( “Camera 0 | Camera 1″, rgb );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:424:28: error: ‘waitKey’ was not declared in this scope
char key = waitKey( 10 );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: At global scope:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:469:24: error: ‘vector’ does not name a type
void procrustes( const vector& P_pts,
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:469:24: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:469:30: error: expected ‘,’ or ‘…’ before ‘<’ token
void procrustes( const vector& P_pts,
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void procrustes(int)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:474:9: error: ‘P_pts’ was not declared in this scope
if( P_pts.size() == 0 || Q_pts.size() == 0 ) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:474:30: error: ‘Q_pts’ was not declared in this scope
if( P_pts.size() == 0 || Q_pts.size() == 0 ) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:479:29: error: ‘P_pts’ was not declared in this scope
P = convert_vector2Mat( P_pts );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:482:29: error: ‘Q_pts’ was not declared in this scope
Q = convert_vector2Mat( Q_pts );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void cbMouseEvent(int, int, int, int, void*)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:552:21: error: ‘P_pts’ was not declared in this scope
P_pts.push_back( Vec3f( col, row, getDepth(0,row,col) ) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:559:21: error: ‘P_pts’ was not declared in this scope
P_pts.pop_back();
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:564:17: error: ‘P_pts’ was not declared in this scope
P_pts.push_back( Vec3f( col, row, getDepth(0,row,col) ) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:566:28: error: ‘Q_pts’ was not declared in this scope
float Qz = Q_pts.back()[2];
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:596:21: error: ‘Q_pts’ was not declared in this scope
Q_pts.push_back( Vec3f( col-640, row, getDepth(1,row,col-640) ) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:602:21: error: ‘Q_pts’ was not declared in this scope
Q_pts.pop_back();
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:606:17: error: ‘Q_pts’ was not declared in this scope
Q_pts.push_back( Vec3f( col-640, row, getDepth(1,row,col-640) ) );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:607:28: error: ‘P_pts’ was not declared in this scope
float Pz = P_pts.back()[2];
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: At global scope:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:636:31: error: ‘vector’ does not name a type
Mat convert_vector2Mat( const vector vec ) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:636:31: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:636:37: error: expected ‘,’ or ‘…’ before ‘<’ token
Mat convert_vector2Mat( const vector vec ) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘cv::Mat convert_vector2Mat(int)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:640:30: error: ‘vec’ was not declared in this scope
for( int i = 0; i < (int)vec.size(); i++ )
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:643:15: error: ‘vec’ was not declared in this scope
Mat m( 3, vec.size(), CV_32F );
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘cv::Vec3f transformPoint(const Vec3f&)’:
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:700:53: error: ‘vector’ has not been declared
float flann_knn(Mat& m_destinations, Mat& m_object, vector& ptpairs, vector& dists = vector()) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:700:59: error: expected ‘,’ or ‘…’ before ‘<’ token
float flann_knn(Mat& m_destinations, Mat& m_object, vector& ptpairs, vector& dists = vector()) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:700:115: error: a function-definition is not allowed here before ‘{’ token
float flann_knn(Mat& m_destinations, Mat& m_object, vector& ptpairs, vector& dists = vector()) {
^
/home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:791:1: error: expected ‘}’ at end of input
}
^
make[2]: *** [CMakeFiles/kinect_reg.dir/kinReg.cpp.o] Error 1
make[1]: *** [CMakeFiles/kinect_reg.dir/all] Error 2
make: *** [all] Error 2
I unable to access data from two kinect which are connected to same machine. I build the complete code on ubuntu 14.04 platform on my laptop. I made some necessary changes in the code. I really struggled lot to run this code successfully but now m facing hardware issue and unable to access multiple kinect data streams. After running executable i will get result as shown below: Let me know can i connect multiple Kinect on same machine or i need some external usb controller. Waiting for ur reply..
Thanking you,
Varsha
/Desktop/libfreenect-master/wrappers/opencv/Build$ ./kinReg
Failed to submit isochronous transfer 0: -1
Failed to submit isochronous transfer 1: -1
Failed to submit isochronous transfer 2: -1
Failed to submit isochronous transfer 3: -1
Failed to submit isochronous transfer 4: -1
Failed to submit isochronous transfer 5: -1
Failed to submit isochronous transfer 6: -1
Failed to submit isochronous transfer 7: -1
Failed to submit isochronous transfer 8: -1
Failed to submit isochronous transfer 9: -1
Failed to submit isochronous transfer 10: -1
Failed to submit isochronous transfer 11: -1
Failed to submit isochronous transfer 12: -1
Failed to submit isochronous transfer 13: -1
Failed to submit isochronous transfer 14: -1
Failed to submit isochronous transfer 15: -1