Registering Two Point Clouds


I finished this project last year and it’s been ‘dying’ to be posted. My goal was to take point clouds obtained from two Kinects and register them into one coordinate system. After the registration process was completed, the new point cloud would be much more robust with many of the obstructed blank spots filled in.



The project follows a simple algorithm.

  1. Use libfreenect to obtain point clouds from both Kinects
  2. Use OpenGL to display the point clouds in an interactice virtual environment
  3. Use OpenCV to display the RGB streams for the user to select correspondences
  4. Calculate the transformations using Procrustes Analysis and the correspondence matrices
  5. Apply the translation and rotation to the point clouds visualized in OpenGL

The implementation of this algorithm can be found here.

After a few weeks of stagnant development with the project, I made a bet with my friends and advisor that I could finish the project in one weekend before I left for a vacation. Below is the video which resulted from my sleepless weekend hackathon.

After the project was finished. I used it to complete what is called an “Honor’s Thesis” here at my University. It is an undergraduate research project, which once defended successfully allows the student to graduate “with honors” on their diploma. Looking back on the thesis now I would have done things differently – but isn’t that almost always the case. None-the-less it was a mile stone in my life and it is my work.

Let me know if you’d like a copy of it and I’ll be happy to send it to you!

This entry was posted in 3D Scene Reconstruction. Bookmark the permalink.

22 Responses to Registering Two Point Clouds

  1. Giseli says:

    I watched the video and, whoa! Very nice stuff! I would like to see the copy of your thesis, if possible :) Thanks and good job!

  2. Nathan Crock says:

    Thanks for the feedback! I’m working on adapting this to some interesting immersive visualizations for my master’s work. I sent the paper, let me know what you think!

  3. brian lee says:

    Looks cool. Did you implement your 3D render for the point cloud with openGL. Could you send me the code of that part?

  4. Nathan Crock says:

    Thanks for the feedback Brian. Yes I did, the project is hosted on my github – https://github.com/mathnathan/Kinect-Registration

  5. Brian lee says:

    Thanks for sharing, Nathan. Really nice job. Looking forward to your Thesis.

  6. Edmond Ngantung says:

    Hi,
    I’m interested with your project about using this dual kinect
    Can you share your thesis to me as well ?
    Looking forward to your Thesis
    Thank you

  7. Edmond Ngantung says:

    I have tried your code which is downloaded from https://github.com/mathnathan/Kinect-Registration and built in my Ubuntu 12.04 and using OpenCV2.4
    But, I got an error like :

    [22%] Built target freenect
    [22%] Built target freenect_sync
    Linking C shared library ../../../../lib/libfreenect_cv.so
    /usr/bin/ld: cannot find -lcv
    collect2: ld returned 1 exit status
    make[2]: *** [lib/libfreenect_cv.so.0.0.1] Error 1
    make[1]: *** [include/libfreenect/wrappers/opencv/CMakeFiles/freenect_cv.dir/all] Error 2
    make: *** [all] Error 2

    Have tried with other PC , but still got the same result as above
    Is there any suggestion to solve this problem ?

  8. Nathan Crock says:

    Hey Edmond, It’s nice to see your interest in my project. I haven’t looked at it in a very long time. Perhaps tomorrow after work I will have sometime to sit down and try to recompile it in Ubuntu 12.04. I’ll keep you posted, and be expecting an email shortly with my thesis. Cheers!

  9. Edmond Ngantung says:

    Thank you for your prompt reply.
    Btw, I’m using your “libfreenect” which is inside of the folder “include” of Kinect – Registration
    When I tried building with “cmake ..” , it seems needs two other folders like “example” and “fakenect” instead of “include” folder.
    Also, it needs the folder of “cpp” for the inside of folder “wrappers”.
    So, I copy the folders from https://github.com/OpenKinect/libfreenect then put it in inside of the folder your “libfreenect”.
    Then, I got the result like as above
    But, I got an error like :

    [22%] Built target freenect
    [22%] Built target freenect_sync
    Linking C shared library ../../../../lib/libfreenect_cv.so
    /usr/bin/ld: cannot find -lcv
    collect2: ld returned 1 exit status
    make[2]: *** [lib/libfreenect_cv.so.0.0.1] Error 1
    make[1]: *** [include/libfreenect/wrappers/opencv/CMakeFiles/freenect_cv.dir/all] Error 2
    make: *** [all] Error 2

    Hope it is clear.
    Looking forward to see your thesis and the message for solving this problem.

  10. Bhavani says:

    Hi!Thats so awesome!I’m doing a final year (undergrad project too!) in point clouds.Its completely a new field for me and I’m overwhelmed by what platform to work with.I have a windows 64bit and am not sure whether to use cpp or Matlab?May I please get a copy of your thesis?Thanks!

  11. Lin says:

    Hi Nathan, I am doing something similar, may I have a look on your thesis.
    Thanks very much!

  12. Ben Evans says:

    Hi Nathan,

    This looks very interesting. I’m working on geomorphological applications, either with multiple Kinects or one Kinect from multiple positions. I’d be fascinated to read your thesis, as it looks like you may save duplication of effort.
    Ben

  13. Pingback: Registration of Multiple Kinects | Visualization Lab at FSU

  14. Alex says:

    Nice work, thanks for sharing! Have you considered using 2D features like SIFT to automatize correspondence search?
    I am very interested in a copy of your thesis!
    Thanks!

  15. Nathan Crock says:

    Thanks Alex. Actually, YES! That was my first approach and in my thesis I outline the process. Unfortunately it was not as robust as the interactive matching and I needed results. I have discontinued working on this, however people in the lab here in my department have picked up my work and made some very interesting progress. We now use the PCL library and do registration using segmented planes from the image. We’ll be sure to blog about it soon on the Visualization Lab website. I’ll take your email and update you when we do. Thanks for reaching out Alex.

  16. Nathan Crock says:

    Hey Ben, thanks for reaching out. Sorry for the delayed reply on things, I’ve been slaving away on my graduate thesis. My work is old news now. We’ve made big progress since then on registration and 3D recording, I’ll take your email and update you when we post some of our work on the Lab’s blog. Cheers!

  17. Keith says:

    Hi, I am very interested in your theory, could u plz send it to me?

  18. James Chen says:

    Very nice work!
    I would like to see the copy of your thesis
    However, I’m working on similar project with two Kinects using PCL (http://pointclouds.org/)
    Do you have any suggestion or advise about that?
    Thank you!

  19. Tanaji Kamble says:

    Hello Nathan,
    U done really nice work.. i am very much curious to know how u done that..i want to go through ur graduation thesis and updated work in detail.. if possible n time permits to u den let me know what will be the potential applications for 3D reconstruction using multiple depth cameras..i am eagerly waiting for ur reply…my self also want to work with multiple depth cameras but unfortunately i am unfamiliar with linux ..so i have plan to work on windows platform using point cloud library..

  20. Tanaji says:

    Hi Nathan,

    I like your work its very interesting. I’d like to read your thesis, can send me on my mail..
    tanajikamble13@gmail.com

  21. Varsha Kamble says:

    Hello,
    I successfully build the program but i am unable to make executable file. Please let me know where i am getting wrong. I am very interested in you work and eagerly waiting for ur reply

    sincos@sincos-300E4C-300E5C-300E7C:~/Desktop/Kinect-Registration-master/Build$ make
    [100%] Building CXX object CMakeFiles/kinect_reg.dir/kinReg.cpp.o
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:48:1: error: ‘vector’ does not name a type
    vector P_pts, Q_pts; // This is how they’re collected
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:80:31: error: ‘vector’ does not name a type
    Mat convert_vector2Mat( const vector vec );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:80:31: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:80:37: error: expected ‘,’ or ‘…’ before ‘<’ token
    Mat convert_vector2Mat( const vector vec );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:84:24: error: ‘vector’ does not name a type
    void procrustes( const vector&, const vector&, Mat&, Mat& );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:84:24: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:84:30: error: expected ‘,’ or ‘…’ before ‘<’ token
    void procrustes( const vector&, const vector&, Mat&, Mat& );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:99:1: error: ‘vector’ does not name a type
    vector rgbCV;
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:100:1: error: ‘vector’ does not name a type
    vector depthCV;
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘int main(int, char**)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:106:9: error: ‘rgbCV’ was not declared in this scope
    rgbCV.push_back( freenect_sync_get_rgb_cv(cam) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:107:9: error: ‘depthCV’ was not declared in this scope
    depthCV.push_back( freenect_sync_get_depth_cv(cam) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:121:60: error: ‘namedWindow’ was not declared in this scope
    namedWindow( “Camera 0 | Camera 1″, CV_WINDOW_AUTOSIZE );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void cbRender()’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:145:5: error: ‘rgbCV’ was not declared in this scope
    rgbCV.clear();
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:146:5: error: ‘depthCV’ was not declared in this scope
    depthCV.clear();
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void cbKeyPressed(unsigned char, int, int)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:196:21: error: ‘P_pts’ was not declared in this scope
    procrustes( P_pts, Q_pts, trans, rot );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:196:28: error: ‘Q_pts’ was not declared in this scope
    procrustes( P_pts, Q_pts, trans, rot );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void loadBuffers(int, unsigned int (*)[640], short int (*)[640][3], unsigned char (*)[640][3])’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:273:5: error: ‘rgbCV’ was not declared in this scope
    rgbCV.push_back( freenect_sync_get_rgb_cv(cameraIndx) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:274:5: error: ‘depthCV’ was not declared in this scope
    depthCV.push_back( freenect_sync_get_depth_cv(cameraIndx) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:285:53: error: expected primary-expression before ‘>’ token
    Vec3b color = rgbCV[cameraIndx].at(row,col);
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘float getDepth(int, int, int)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:332:23: error: ‘depthCV’ was not declared in this scope
    float d = (float)(depthCV[cam].at(x,y));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:332:39: error: expected primary-expression before ‘short’
    float d = (float)(depthCV[cam].at(x,y));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:332:39: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:335:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x,y+1));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:335:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:338:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x,y-1));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:338:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:341:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x+1,y));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:341:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:344:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x-1,y));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:344:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:347:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x-1,y-1));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:347:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:350:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x-1,y+1));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:350:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:353:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x+1,y-1));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:353:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:356:37: error: expected primary-expression before ‘short’
    d = (float)(depthCV[cam].at(x+1,y+1));
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:356:37: error: expected ‘)’ before ‘short’
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void displayCVcams()’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:415:19: error: ‘rgbCV’ was not declared in this scope
    cvtColor( rgbCV[cam], tmp, CV_RGB2BGR );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:415:47: error: ‘cvtColor’ was not declared in this scope
    cvtColor( rgbCV[cam], tmp, CV_RGB2BGR );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:419:27: error: ‘rgbCV’ was not declared in this scope
    Mat rgb = joinFrames( rgbCV[0], rgbCV[1] );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:420:40: error: ‘imshow’ was not declared in this scope
    imshow( “Camera 0 | Camera 1″, rgb );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:424:28: error: ‘waitKey’ was not declared in this scope
    char key = waitKey( 10 );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: At global scope:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:469:24: error: ‘vector’ does not name a type
    void procrustes( const vector& P_pts,
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:469:24: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:469:30: error: expected ‘,’ or ‘…’ before ‘<’ token
    void procrustes( const vector& P_pts,
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void procrustes(int)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:474:9: error: ‘P_pts’ was not declared in this scope
    if( P_pts.size() == 0 || Q_pts.size() == 0 ) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:474:30: error: ‘Q_pts’ was not declared in this scope
    if( P_pts.size() == 0 || Q_pts.size() == 0 ) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:479:29: error: ‘P_pts’ was not declared in this scope
    P = convert_vector2Mat( P_pts );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:482:29: error: ‘Q_pts’ was not declared in this scope
    Q = convert_vector2Mat( Q_pts );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘void cbMouseEvent(int, int, int, int, void*)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:552:21: error: ‘P_pts’ was not declared in this scope
    P_pts.push_back( Vec3f( col, row, getDepth(0,row,col) ) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:559:21: error: ‘P_pts’ was not declared in this scope
    P_pts.pop_back();
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:564:17: error: ‘P_pts’ was not declared in this scope
    P_pts.push_back( Vec3f( col, row, getDepth(0,row,col) ) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:566:28: error: ‘Q_pts’ was not declared in this scope
    float Qz = Q_pts.back()[2];
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:596:21: error: ‘Q_pts’ was not declared in this scope
    Q_pts.push_back( Vec3f( col-640, row, getDepth(1,row,col-640) ) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:602:21: error: ‘Q_pts’ was not declared in this scope
    Q_pts.pop_back();
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:606:17: error: ‘Q_pts’ was not declared in this scope
    Q_pts.push_back( Vec3f( col-640, row, getDepth(1,row,col-640) ) );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:607:28: error: ‘P_pts’ was not declared in this scope
    float Pz = P_pts.back()[2];
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: At global scope:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:636:31: error: ‘vector’ does not name a type
    Mat convert_vector2Mat( const vector vec ) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:636:31: error: ISO C++ forbids declaration of ‘parameter’ with no type [-fpermissive]
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:636:37: error: expected ‘,’ or ‘…’ before ‘<’ token
    Mat convert_vector2Mat( const vector vec ) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘cv::Mat convert_vector2Mat(int)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:640:30: error: ‘vec’ was not declared in this scope
    for( int i = 0; i < (int)vec.size(); i++ )
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:643:15: error: ‘vec’ was not declared in this scope
    Mat m( 3, vec.size(), CV_32F );
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp: In function ‘cv::Vec3f transformPoint(const Vec3f&)’:
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:700:53: error: ‘vector’ has not been declared
    float flann_knn(Mat& m_destinations, Mat& m_object, vector& ptpairs, vector& dists = vector()) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:700:59: error: expected ‘,’ or ‘…’ before ‘<’ token
    float flann_knn(Mat& m_destinations, Mat& m_object, vector& ptpairs, vector& dists = vector()) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:700:115: error: a function-definition is not allowed here before ‘{’ token
    float flann_knn(Mat& m_destinations, Mat& m_object, vector& ptpairs, vector& dists = vector()) {
    ^
    /home/sincos/Desktop/Kinect-Registration-master/kinReg.cpp:791:1: error: expected ‘}’ at end of input
    }
    ^
    make[2]: *** [CMakeFiles/kinect_reg.dir/kinReg.cpp.o] Error 1
    make[1]: *** [CMakeFiles/kinect_reg.dir/all] Error 2
    make: *** [all] Error 2

  22. Varsha Kamble says:

    I unable to access data from two kinect which are connected to same machine. I build the complete code on ubuntu 14.04 platform on my laptop. I made some necessary changes in the code. I really struggled lot to run this code successfully but now m facing hardware issue and unable to access multiple kinect data streams. After running executable i will get result as shown below: Let me know can i connect multiple Kinect on same machine or i need some external usb controller. Waiting for ur reply..
    Thanking you,
    Varsha

    /Desktop/libfreenect-master/wrappers/opencv/Build$ ./kinReg
    Failed to submit isochronous transfer 0: -1
    Failed to submit isochronous transfer 1: -1
    Failed to submit isochronous transfer 2: -1
    Failed to submit isochronous transfer 3: -1
    Failed to submit isochronous transfer 4: -1
    Failed to submit isochronous transfer 5: -1
    Failed to submit isochronous transfer 6: -1
    Failed to submit isochronous transfer 7: -1
    Failed to submit isochronous transfer 8: -1
    Failed to submit isochronous transfer 9: -1
    Failed to submit isochronous transfer 10: -1
    Failed to submit isochronous transfer 11: -1
    Failed to submit isochronous transfer 12: -1
    Failed to submit isochronous transfer 13: -1
    Failed to submit isochronous transfer 14: -1
    Failed to submit isochronous transfer 15: -1

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>