Blobify Yourself with Kinect and Bubble Boy

UPDATE (4/3/12): I’ve fixed the “marchingCubes” error. Please click the link below to get an updated script. I’m leaving this page up for reference only.

UPDATED SCRIPT

Here’s a simple modification to a sketch that renders polygonal meshes from point clouds. All I did was strap a kinect to it. Most of the credit goes to Daniel Schiffman and Rui Madeira.

Dependencies…

  • You’ll need Schiffman Open Kinect library, as posted here.
  • The ToxicLibs Library
  • Marching Cubes example sketch here.

Getting started…

Download and install the libraries above. Once you have the libraries installed, download the sample sketch listed above. Run the sample sketch to make sure everything is working.

Modify the sketch…

I won’t go into too much detail since it would just be easier to copy what I have and play around with it in order to find out how it works. Basically, we’ll import the openkinect library to begin using the kinect. We’ll modify the setup function to prepare some new classes and set the screen correctly. After that we’ll change the “mousepressed” if statement to update metaBalls with points from the kinect instead of points from a mouse. We’ll do some not-so-scientific math in order to get the screen to center and call it good.

Below is the working example, if you have any suggestions let me know! Also, if anyone out there knows of a more efficient meshing library from point clouds I’m all ears, I’d like to be able to render poly in real time and at high resolution, but so far I haven’t been able to find an efficient way of doing this!

The sketch…

import toxi.geom.Vec3D;
import processing.opengl.*;
import rui.marchingCubes.*;

// kinect
import org.openkinect.*;
import org.openkinect.processing.*;

MarchingCubes mc;
Vec3D rotationAxis;

Boolean bUseFill;

// kinect
Kinect kinect;
float a = 0;
// Size of kinect image
int w = 640;
int h = 480;
int kWidth  = 640;
int kHeight = 480;
// depth mapping and tilt
boolean depth = true;
boolean rgb = false;
boolean ir = false;
float deg = 8; // Start at 15 degrees
PImage depthImg;
int minDepth =  40;
int maxDepth = 860;
// set initial record to false
boolean record = false;
int counter = 0;
// print custom file
boolean printFile = false;
ArrayList points;
PrintWriter output;
// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[2048];

void setup(){
  size(1024, 600, OPENGL);
  Vec3D aabbMin = new Vec3D(-width/2, -height/2, -250);
  Vec3D aabbMax = new Vec3D(width/2, height/2, 250);
  Vec3D numPoints = new Vec3D(50,50,50);
  float isoLevel = 1;
  mc = new MarchingCubes(this, aabbMin, aabbMax, numPoints, isoLevel);

  rotationAxis = new Vec3D();

  bUseFill = false;

  // kinect
  kinect = new Kinect(this);
  kinect.start();
  kinect.enableDepth(true);
  kinect.tilt(deg);
  // We don't need the grayscale image in this example
  // so this makes it more efficient
  kinect.processDepthImage(false);
  // get depthImg to constrain
  depthImg = new PImage(kWidth, kHeight);
  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
  }
  points = new ArrayList();
  output = createWriter("points.txt");

}

void draw(){
  background(255);
  lights();

  // kinect
  int[] depth = kinect.getRawDepth();
  int skip = 50;
  //translate(width/750,height/750,-50);
    mc.reset();

    // original for loop
    println("entering loop");
    int nBalls = 0;
    for(int x=0; x<w; x+=skip) {
      for(int y=0; y<h; y+=skip) {
        int offset = x+y*w;
        int rawDepth = depth[offset];

        if(rawDepth >= minDepth && rawDepth <= maxDepth) {
          PVector v = depthToWorld(x,y,rawDepth);
          Vec3D metaBallPos = new Vec3D(v.x * 500, v.y * 300, v.z*300);
          mc.addMetaBall(metaBallPos, 100, 1);
          nBalls++;
        }

      }
    }
    println("done with loop, " + nBalls + " balls");
    // end original for loop

  mc.createMesh();
  if(bUseFill){
    fill(0,255,0);
    noStroke();
  }
  else {
    noFill();
    stroke(127);
  }

  pushMatrix();
  translate(width/2, height/2, 0);
  rotateX(rotationAxis.x);
  rotateY(rotationAxis.y);
  mc.renderMesh();
  popMatrix();
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  depthLookUp[depthValue];//rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}

float rawDepthToMeters(int depthValue) {
  if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

void keyPressed(){
  if(key == CODED){
    if(keyCode == LEFT) rotationAxis.y += 0.05;
    if(keyCode == RIGHT) rotationAxis.y -= 0.05;
    if(keyCode == UP) rotationAxis.x -= 0.05;
    if(keyCode == DOWN) rotationAxis.x += 0.05;
  }
  else {
    if(key == ' '){
      bUseFill = !bUseFill;
    }
    if(key == 'r' || key == 'R'){
      mc.reset();
      rotationAxis.set(0,0,0);
    }
  }
}

void stop() {
  kinect.quit();
  super.stop();
}

Watch the video below to get a better understanding of the installation process (see it in action around 02:13):

That’s it! You should be able to copy this code, paste it over the original example sketch and away you go or just download the example below!

Download Bubble Boy

UPDATE:

I’ve added a new git repo here, please let me know if you wish to contribute!

UPDATE UPDATE:

I’ve been getting a lot of responses saying they are getting a rui.marchingcubes error. I’m looking into this, but for now I believe it has to do with the differences between processing 1.2.1 and 1.5.1 – if you have a newer version of processing (1.5.1) it will break with an error:

No library found for rui.marchingCubes

I still haven’t found a solve for this, since the marching cubes library is contained within the sketch folder itself, labeled “MarchingCubes.java”

If anyone has any suggestions other than to install an older version of processing (1.2.1), I’m all ears…

27 thoughts on “Blobify Yourself with Kinect and Bubble Boy

  1. Newbie to Processing (v1.5) environment.
    I get “No library found for rui.marchingCubes” when running MarchingCubes_example_002.pdf.
    Can any one help?

  2. Did you make sure and copy the entire directory into your sketch and not just the .pde file? If it’s unable to find the library it means processing doesn’t know how to find the files it needs. Download a new copy of the marching cubes sketch, unzip and drag the entire folder into you sketches library. This will bring everything processing needs with it.

    Additionally, make sure you correctly install the toxiclibs library!

  3. There are three java files (MarchingCubes.java, MarchingCubesTables.java, MCTriangle.java) in same directory where the pde file resides.
    Do they need to be in library form (jar file)?
    Thanks again for your help.

  4. As much work as is going into compiling and running it, you may want to look to outsource it to an EC2 cluster. It’d cost a bit but it may be worth it.. Of course, you may also want to look at just optimizing the code first, I suggest starting by making it in C. I love processing, but it’s terribly inefficient… Wonder if you could feed the data from processing into a cloud, have the rendering done, and then feed it back, in realtime?

  5. Did anyone get this working? I take it a jar file needed to be built from the 3 .java files here. Is there a link to a downloadable jar?

    Thanks…

  6. Good call. Java/processing is alot slower. I’d like to port it to C, shouldn’t be too hard I just haven’t gotten around to it. Ever play with Cinder? I think I’d like to make something like this with Cinder as my first project.

    ~Mike

  7. hello,

    i have a workaround to use the library with version 1.5.1 (tested). you just need to delete the package line in the java files. then you can have them in the same directory as the pde file.
    works for me.

    best

  8. Usually I do not learn article on blogs, however I wish to say that
    this write-up very compelled me to try and do so!
    Your writing taste has been amazed me. Thanks, quite great post.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.