18 Apr

Blobify Yourself with Kinect and Bubble Boy

Posted by Mike Categories: Blog
Blobify Yourself with Kinect and Bubble Boy

UPDATE (4/3/12): I’ve fixed the “marchingCubes” error. Please click the link below to get an updated script. I’m leaving this page up for reference only.

UPDATED SCRIPT

Here’s a simple modification to a sketch that renders polygonal meshes from point clouds. All I did was strap a kinect to it. Most of the credit goes to Daniel Schiffman and Rui Madeira.

Dependencies…

  • You’ll need Schiffman Open Kinect library, as posted here.
  • The ToxicLibs Library
  • Marching Cubes example sketch here.

Getting started…

Download and install the libraries above. Once you have the libraries installed, download the sample sketch listed above. Run the sample sketch to make sure everything is working.

Modify the sketch…

I won’t go into too much detail since it would just be easier to copy what I have and play around with it in order to find out how it works. Basically, we’ll import the openkinect library to begin using the kinect. We’ll modify the setup function to prepare some new classes and set the screen correctly. After that we’ll change the “mousepressed” if statement to update metaBalls with points from the kinect instead of points from a mouse. We’ll do some not-so-scientific math in order to get the screen to center and call it good.

Below is the working example, if you have any suggestions let me know! Also, if anyone out there knows of a more efficient meshing library from point clouds I’m all ears, I’d like to be able to render poly in real time and at high resolution, but so far I haven’t been able to find an efficient way of doing this!

The sketch…

import toxi.geom.Vec3D;
import processing.opengl.*;
import rui.marchingCubes.*;

// kinect
import org.openkinect.*;
import org.openkinect.processing.*;

MarchingCubes mc;
Vec3D rotationAxis;

Boolean bUseFill;

// kinect
Kinect kinect;
float a = 0;
// Size of kinect image
int w = 640;
int h = 480;
int kWidth  = 640;
int kHeight = 480;
// depth mapping and tilt
boolean depth = true;
boolean rgb = false;
boolean ir = false;
float deg = 8; // Start at 15 degrees
PImage depthImg;
int minDepth =  40;
int maxDepth = 860;
// set initial record to false
boolean record = false;
int counter = 0;
// print custom file
boolean printFile = false;
ArrayList points;
PrintWriter output;
// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[2048];

void setup(){
  size(1024, 600, OPENGL);
  Vec3D aabbMin = new Vec3D(-width/2, -height/2, -250);
  Vec3D aabbMax = new Vec3D(width/2, height/2, 250);
  Vec3D numPoints = new Vec3D(50,50,50);
  float isoLevel = 1;
  mc = new MarchingCubes(this, aabbMin, aabbMax, numPoints, isoLevel);

  rotationAxis = new Vec3D();

  bUseFill = false;

  // kinect
  kinect = new Kinect(this);
  kinect.start();
  kinect.enableDepth(true);
  kinect.tilt(deg);
  // We don't need the grayscale image in this example
  // so this makes it more efficient
  kinect.processDepthImage(false);
  // get depthImg to constrain
  depthImg = new PImage(kWidth, kHeight);
  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }
  points = new ArrayList();
  output = createWriter("points.txt");

}

void draw(){
  background(255);
  lights();

  // kinect
  int[] depth = kinect.getRawDepth();
  int skip = 50;
  //translate(width/750,height/750,-50);
    mc.reset();

    // original for loop
    println("entering loop");
    int nBalls = 0;
    for(int x=0; x<w; x+=skip) {
      for(int y=0; y<h; y+=skip) {
        int offset = x+y*w;
        int rawDepth = depth[offset];

        if(rawDepth >= minDepth && rawDepth <= maxDepth) {
          PVector v = depthToWorld(x,y,rawDepth);
          Vec3D metaBallPos = new Vec3D(v.x * 500, v.y * 300, v.z*300);
          mc.addMetaBall(metaBallPos, 100, 1);
          nBalls++;
        }

      }
    }
    println("done with loop, " + nBalls + " balls");
    // end original for loop

  mc.createMesh();
  if(bUseFill){
    fill(0,255,0);
    noStroke();
  }
  else {
    noFill();
    stroke(127);
  }

  pushMatrix();
  translate(width/2, height/2, 0);
  rotateX(rotationAxis.x);
  rotateY(rotationAxis.y);
  mc.renderMesh();
  popMatrix();
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  depthLookUp[depthValue];//rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

float rawDepthToMeters(int depthValue) {
  if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

void keyPressed(){
  if(key == CODED){
    if(keyCode == LEFT) rotationAxis.y += 0.05;
    if(keyCode == RIGHT) rotationAxis.y -= 0.05;
    if(keyCode == UP) rotationAxis.x -= 0.05;
    if(keyCode == DOWN) rotationAxis.x += 0.05;
  }
  else {
    if(key == ' '){
      bUseFill = !bUseFill;
    }
    if(key == 'r' || key == 'R'){
      mc.reset();
      rotationAxis.set(0,0,0);
    }
  }
}

void stop() {
  kinect.quit();
  super.stop();
}

Watch the video below to get a better understanding of the installation process (see it in action around 02:13):

That’s it! You should be able to copy this code, paste it over the original example sketch and away you go or just download the example below!

Download Bubble Boy

UPDATE:

I’ve added a new git repo here, please let me know if you wish to contribute!

UPDATE UPDATE:

I’ve been getting a lot of responses saying they are getting a rui.marchingcubes error. I’m looking into this, but for now I believe it has to do with the differences between processing 1.2.1 and 1.5.1 – if you have a newer version of processing (1.5.1) it will break with an error:

No library found for rui.marchingCubes

I still haven’t found a solve for this, since the marching cubes library is contained within the sketch folder itself, labeled “MarchingCubes.java”

If anyone has any suggestions other than to install an older version of processing (1.2.1), I’m all ears…

26 Comments on “ Blobify Yourself with Kinect and Bubble Boy ”

  1. Alex Nikolayevsky April 19, 2011 at 5:07 PM

    Awesome dude. This is really sweet.

  2. Live 3D Rendering | Gaming Now April 20, 2011 at 2:25 AM

    [...] Complete story with source [...]

  3. Kyu Hur April 21, 2011 at 6:54 PM

    Newbie to Processing (v1.5) environment.
    I get “No library found for rui.marchingCubes” when running MarchingCubes_example_002.pdf.
    Can any one help?

    • Mike April 22, 2011 at 2:09 AM

      Did you make sure and copy the entire directory into your sketch and not just the .pde file? If it’s unable to find the library it means processing doesn’t know how to find the files it needs. Download a new copy of the marching cubes sketch, unzip and drag the entire folder into you sketches library. This will bring everything processing needs with it.

      Additionally, make sure you correctly install the toxiclibs library!

      • Kyu Hur April 22, 2011 at 2:36 PM

        There are three java files (MarchingCubes.java, MarchingCubesTables.java, MCTriangle.java) in same directory where the pde file resides.
        Do they need to be in library form (jar file)?
        Thanks again for your help.

        • M_G April 27, 2011 at 1:21 PM

          I have the same problem as you, did fou fixed it?
          Thank you

          • Bill May 5, 2011 at 1:54 AM

            Did anyone get this working? I take it a jar file needed to be built from the 3 .java files here. Is there a link to a downloadable jar?

            Thanks…

          • Jared October 30, 2011 at 5:30 PM

            bump

          • Mike April 4, 2012 at 12:50 AM

            Hi M_G,

            Please see the fix here:

            http://iwearshorts.com/blog/bubble-boy2/

            for this issue.

            Mike

    • Mike April 4, 2012 at 12:49 AM

      Hi Kyu,

      You’re getting that error because of the way newer version of processing finds things. I updated a script here: http://iwearshorts.com/blog/bubble-boy2/ that will fix the issue.

      Mike

  4. Richard L. April 28, 2011 at 3:53 PM

    great post! I love it :)

  5. [...] Newell] dropped us a line about his latest project, Bubble boy! Which uses the Kinect point cloud functionality to render polygonal meshes in real time.  In the [...]

  6. Derrick May 3, 2011 at 10:39 PM

    As much work as is going into compiling and running it, you may want to look to outsource it to an EC2 cluster. It’d cost a bit but it may be worth it.. Of course, you may also want to look at just optimizing the code first, I suggest starting by making it in C. I love processing, but it’s terribly inefficient… Wonder if you could feed the data from processing into a cloud, have the rendering done, and then feed it back, in realtime?

    • Mike April 4, 2012 at 12:52 AM

      Good call. Java/processing is alot slower. I’d like to port it to C, shouldn’t be too hard I just haven’t gotten around to it. Ever play with Cinder? I think I’d like to make something like this with Cinder as my first project.

      ~Mike

  7. [...] Newell] dropped us a line about his latest project, Bubble boy! Which uses the Kinect point cloud functionality to render polygonal meshes in real time.  In the [...]

  8. [...] Newell] dropped us a line about his latest project, Bubble boy! Which uses the Kinect point cloud functionality to render polygonal meshes in real time.  In the [...]

  9. [...] Newell] dropped us a line about his latest project, Bubble boy! Which uses the Kinect point cloud functionality to render polygonal meshes in real time.  In the [...]

  10. [...] Processing demo uses the Kinect Point Cloud to present bubbles, representing the person behind the camera. That [...]

  11. HHH June 25, 2011 at 12:20 PM

    No library found for rui.marchingCubes !!!

    • Mike April 4, 2012 at 12:48 AM

      Hi HHH,

      Again, sorry for being so late on this. I have an updated script here that will work with newer versions of processing. Please refer to this post, it should fix your problem.

      http://iwearshorts.com/blog/bubble-boy2/

  12. [...] 3. Render 3D objects  Download [...]

  13. [...] 3. Render 3D objects Download [...]

  14. [...] 3. Render 3D objects Download [...]

  15. [...] NOTE: If you’re looking for the older version please refer here. [...]

  16. Benjamin August 29, 2012 at 9:16 AM

    hello,

    i have a workaround to use the library with version 1.5.1 (tested). you just need to delete the package line in the java files. then you can have them in the same directory as the pde file.
    works for me.

    best

  17. pod zastaw February 28, 2013 at 4:32 AM

    Usually I do not learn article on blogs, however I wish to say that
    this write-up very compelled me to try and do so!
    Your writing taste has been amazed me. Thanks, quite great post.

Leave a Reply
  1. XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>