Helpful C++ Resources & Books

I often Google search things like how to manipulate strings, vectors, data structures like lists or maps and find resources on stackexchange or stackoverflow. But it is also helpful to have some concrete resources for C++. Here are some that I’ve been recommended before:

C++ Tutorial
Another C++ Tutorial
Bjarne Stroustrup’s The C++ Programming Language
Deitel & Deitel’s C++ How to Program

For developing on the iPhone, the Apple developer pages are essential reading: Apple iOS Developer Library

Also some openFrameworks specific resources:

Jeff Crouse’s Introduction to openFrameworks
openFrameworks Forum (ask questions, find answers to questions already asked)
openFrameworks Documentation
oepnFrameworks addons

Places to find interesting applications:

creativeapplications.net
createdigitalmotion.com
createdigitalmusic.com
thecreatorsproject.com

And my previous courses using openFrameworks:

Audiovisual Processing for iOS
More…

Assignments

Assignment 1: Mobile
EDIT: Note that group presentations will occur in RHB256 on Monday the 18th and Tuesday the 19th at 6 p.m.

Assignement 2: Computer Vision

Introduction

For the first 5 weeks, we will cover mobile development. Please make sure you already have a coding environment setup to use the iOS branch of openFrameworks v0073: openFrameworks. You will want to get the iOS version which only works on OSX for these first 5 weeks. The latter 5 weeks we will work on Computer Vision. For this portion, you will need to get another branch of openFrameworks suitable for your operating system (OSX/Windows/Linux). NOTE: when you download openFrameworks, there is no application included which you run (like Processing). You basically get a zip file with a bunch of code in different directories.

To start programming in openFrameworks, we usually start with a “project file”. In OSX, this is an extension of “.xcodeproj” and for Visual Studio, it is “.vcproj” or “.sln”. We open this project file using an IDE, an integrated development environment. For OSX, this is XCode, and for Windows, this is Code Blocks or Visual Studio. You can get all of these programs for free as a student (and legally too). The IDE comes with a lot of tools to help us program and has nothing to do with openFrameworks at all. openFrameworks is simply a huge set of libraries that make it easy to code complicated programs quickly. The IDE gives us a lot of features like code completion (automatically guessing what we want to code), compiler (turning our C++ code into assembly/machine code), linker (combining all of our compiled code into an executable program), and debugger (executing a program and being able to look at where the program is in your code at the same time).

Lecture

Introduction to iOS, openFrameworks, and XCode.

iOS
Week 1 Slides

openFrameworks:
– http://www.openframeworks.cc/
– open-source creative coding toolkit for writing software
– provides many high-level classes and abstractions which are cross-platform for a collection of libraries including: OpenCV, Vector/Matrix Math, Graphics using OpenGL, Mesh, OBJ, VBO, Shaders, Audio/Video input/output/saving, Kinect, File/Directory access/manipulation, TCP/IP/UDP/OSC, Threading, Physics engine, Synthesizers, iPhone/Android integration (GPS/Compass/UI/OpenGL/Maps/Audio/Video), …
– project generator (v0072+) gets you started with a new openFrameworks project quickly.

XCode
– how to set the current “scheme”, “target settings”, “build settings”, and “architecture”.
– including header/source files
– including frameworks
– creating new files

obj-c + c++:
– ability to retain cross-platform functionality in c++ code
– no need to touch obj-c unless doing specific interface code or extending functionality of existing openFramework classes

Lab

Try running the following examples included with openFrameworks: emptyExample, graphicsExample, fontsExample. Spend a 4-5 minutes looking at the testApp.h and testApp.mm files for each project to get an idea for how the program was created. Be sure to have a look at the touchDown, touchUp, and touchMoved methods to get an idea for how to access where the user is touching on a screen.

Now use the project generator to create a new openFrameworks project that we’ll open using XCode. The project generator is an application which creates an XCode project and sets up all the openFrameworks libraries and path settings within the project so we don’t have to. You can find it inside your openFrameworks download inside the directory, “projectGenerator”. Give your sketch any name, such as “lab1-ex1”, use the default location of “myApps”, and click the “GENERATE” button. We won’t need to use any other features of the project generator at this time.

Navigate to the directory where you told the projectGenerator to create an app and open the “.xcodeproj” file. For your first exercise, create a class which draws a button (e.g. using ofRect). Have the button change color (ofSetColor) when a user presses within the boundaries of the button. You’ll want to define points for the button using ofPoint and test whether the touch coordinates are within the button’s boundaries using ofWithin. Try having more than one on the screen. See if you can have the button perform a specific action, such as making a sound, or recording the camera image.

Please, while trying this exercise, DO NOT COPY AND PASTE FROM EXISTING CODE. You may use existing code as a reference, but manually type every single character and refer to what you are typing mentally in your mind. Reflect on what each statement is doing and if you have *ANY* doubt, please ask someone next to you or feel free to ask me. Don’t give up if you do not finish by the end of the lab. As a baseline, if you have about 12 hours of lecture/lab time per week between 5 courses, you should be spending 6 hours at home per week for this course outside of lectures/labs reflecting on this material and pursuing the lab/lecture material in greater detail.

Lecture

Introduction to ofxMaximilian and ofxBox2D.

ofxMaximilian
Maximilian is an open-source, lightweight, c++ audio library developed here at Goldsmiths, University of London by Mick Grierson and Chris Kiefer. You can use it for doing synthesis (oscillators, granular time/pitch stretching/shifting, atomic), filtering (LPF, band, HPF, envelope), compression, limiting, mixing, sound fx (delay, flanging, chorus, distortion), FFT/IFFT, and sample loading/writing (WAVE).

Have a look at the basic library here:
maximilian.h
maximilian.cpp

And some examples.

ofxBox2D
Box2D is an open-source library for 2D physical modeling. With it, you can easily simulate 2D physics and collisions using really highly optimized code.

Lab

Part 1
If you were having trouble in last week’s assignment, feel free to have a look at my code for the button class below. I’ve commented it so read through the functions and try to understand what they do.

myButton.h

/*
 *  Created by Parag K. Mital - http://pkmital.com
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2013 Parag K. Mital. All rights reserved.
 *
 *   Permission is hereby granted, free of charge, to any person
 *   obtaining a copy of this software and associated documentation
 *   files (the "Software"), to deal in the Software without
 *   restriction, including without limitation the rights to use,
 *   copy, modify, merge, publish, distribute, sublicense, and/or sell
 *   copies of the Software, and to permit persons to whom the
 *   Software is furnished to do so, subject to the following
 *   conditions:
 *
 *   The above copyright notice and this permission notice shall be
 *   included in all copies or substantial portions of the Software.
 *
 *   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 *   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *   OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *   NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *   HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *   WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *   FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *   OTHER DEALINGS IN THE SOFTWARE.
 */

// header files declare classes...

#pragma once

#include "ofMain.h" // this lets us use openFrameworks

// enum's are like integers but have names for their values instead of numbers.
// so instead of an int being equal to 0, you have a buttonState equal to button_default...
// and instead of an int being equal to 1, you have a buttonState equal to button_pressed...
enum buttonState {
    button_default = 0,
    button_pressed = 1
};

class myButton
{
public:

    //--------------------------------------------------------------
    // this is a constructor.  every c++ class has one.  it is the same name as the class
    // it is automagically called when you create a new instance of this class.  you
    // can put any setup of your object inside of this function.
    myButton();

    //--------------------------------------------------------------
    // we'll draw our button inside this function
    void draw();

    //--------------------------------------------------------------
    // we create a few "setters" which will change our internal variables (like color, position, size).
    void setColorForState(buttonState stateToChange, ofColor colorToChangeItTo);
    void setPosition(int x, int y);
    void setPosition(ofPoint pt);
    void setSize(int w, int h);

    //--------------------------------------------------------------
    // we provide a method to easily get access to our button's position
    ofPoint getPosition();

    //--------------------------------------------------------------
    // check if the button's state is currently "pressed"
    bool isPressed();

    //--------------------------------------------------------------
    // a method to call whenever the user is touching the screen...
    // we can test if the user touched our button and change the state of our button accordingly...
    // we return true if the button was pressed.
    bool touchDown(ofTouchEventArgs touch);

    //--------------------------------------------------------------
    // Our button class holds a few variables which help us keep track of our state, location and color...
    buttonState             myState;
    ofRectangle             myLocation;
    ofColor                 myColorForDefault, myColorForPressed;
};

myButton.cpp

/*
 *  Created by Parag K. Mital - http://pkmital.com
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2013 Parag K. Mital. All rights reserved.
 *
 *   Permission is hereby granted, free of charge, to any person
 *   obtaining a copy of this software and associated documentation
 *   files (the "Software"), to deal in the Software without
 *   restriction, including without limitation the rights to use,
 *   copy, modify, merge, publish, distribute, sublicense, and/or sell
 *   copies of the Software, and to permit persons to whom the
 *   Software is furnished to do so, subject to the following
 *   conditions:
 *
 *   The above copyright notice and this permission notice shall be
 *   included in all copies or substantial portions of the Software.
 *
 *   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 *   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *   OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *   NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *   HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *   WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *   FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *   OTHER DEALINGS IN THE SOFTWARE.
 */

#include "myButton.h"

//--------------------------------------------------------------
myButton::myButton() {
    // our constructor is called automagically.
    // we put our initialization inside of here.

    // let's set every state's color to be white
    myColorForDefault = ofColor::white;
    myColorForPressed = ofColor::white;

    // and set our state to be the default one
    myState = button_default;

    // and set a simple default position and size for our button
    myLocation.x = 0;
    myLocation.y = 0;
    myLocation.width = 0;
    myLocation.height = 0;
}

//--------------------------------------------------------------
void myButton::draw() {

    // set the color based on our current state
    if (myState == button_default) {
        ofSetColor(myColorForDefault);
    }
    else if(myState == button_pressed) {
        ofSetColor(myColorForPressed);
    }

    // then draw a simple rectangle using our position.
    ofRect(myLocation.x,
           myLocation.y,
           myLocation.width,
           myLocation.height);
}

//--------------------------------------------------------------
void myButton::setColorForState(buttonState stateToChange,
                                ofColor colorToChangeItTo)
{
    // we check which state is being changed
    if (stateToChange == button_default) {
        // and change the color accordingly
        myColorForDefault = colorToChangeItTo;
    }
    else if(stateToChange == button_pressed) {
        myColorForPressed = colorToChangeItTo;
    }
}

//--------------------------------------------------------------
void myButton::setPosition(int x, int y)
{
    // constrain the location to be within the bounds of the application window
    myLocation.x = MIN(ofGetWidth() - myLocation.width, MAX(0,x));
    myLocation.y = MIN(ofGetHeight() - myLocation.height, MAX(0,y));
}

//--------------------------------------------------------------
void myButton::setPosition(ofPoint pt)
{
    // constrain the location to be within the bounds of the application window
    myLocation.x = MIN(ofGetWidth() - myLocation.width, MAX(0,pt.x));
    myLocation.y = MIN(ofGetHeight() - myLocation.height, MAX(0,pt.y));
}

//--------------------------------------------------------------
void myButton::setSize(int w, int h)
{
    // change our internal variables to match what size the user told us to be
    myLocation.width = w;
    myLocation.height = h;
}

//--------------------------------------------------------------
ofPoint myButton::getPosition()
{
    // return our internal position
    return myLocation.position;
}

//--------------------------------------------------------------
bool myButton::touchDown(ofTouchEventArgs touch)
{
    bool bTouched = false;
    // check if the touch's x,y position is inside the button
    if (myLocation.inside(touch.x, touch.y)) {
        bTouched = true;
        // if so, the user touched us!  let's change states by simply adding 1
        // to our current value.  this "cycles" through all the possible values (only 2).
        // if we had more possible values for buttonState, we would use a modulus of
        // however many different states we had, rather than 2...
        myState = (buttonState)((myState + 1) % 2);
    }
    return bTouched;
}

//--------------------------------------------------------------
bool myButton::isPressed()
{
    // check if we are being pressed
    return myState == button_pressed;
}

Part 2
If you are comfortable using the above class, now try integrating some maximilian objects to do some synthesis or sample loading/playback. Try just a simple oscillator using maxiOsc to begin with. See if you can create a button which outputs a sine tone. Can you also modulate it based on your touch coordinates? How about delay?

Part 3 [optional]

Now try to record using the audioIn function. Remember to change ofSoundStreamSetup to use a single channel for input. Can you record sound from the microphone and use a button to play it back? Try also changing the drawing of the buttons to use more descriptive images instead.

Part 4 [super optional]

Create an application which creates buttons where the user pressed down, recording samples from the microphone as the user holds down. When they release, create the final button in the position where the user let go. Can you use ofxAccelerometer or ofxBox2D to model the physics of this button? Perhaps the button can play whenever it bounces off a wall (have a look at the “Contact Listener” example in ofxBox2d). You could also use the physically modeled speed of the button to change the speed of playback.

Lecture

Provisioning
Certificates and provisioning profiles. Apple Developer program and Development Provisioning Assistant.
ofxAddons
Great place to find additional code you can import into your application.
http://ofxaddons.com
Map Kit
GPS, Maps API.

Lab

Part 1
If you were having trouble in last week’s assignment, feel free to have a look at my code for the “super optional” part of the lab. This depends on having the ofxBox2d addon included in your project.

testApp.h

/*
 *  Created by Parag K. Mital - http://pkmital.com
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2013 Parag K. Mital. All rights reserved.
 *
 *   Permission is hereby granted, free of charge, to any person
 *   obtaining a copy of this software and associated documentation
 *   files (the "Software"), to deal in the Software without
 *   restriction, including without limitation the rights to use,
 *   copy, modify, merge, publish, distribute, sublicense, and/or sell
 *   copies of the Software, and to permit persons to whom the
 *   Software is furnished to do so, subject to the following
 *   conditions:
 *
 *   The above copyright notice and this permission notice shall be
 *   included in all copies or substantial portions of the Software.
 *
 *   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 *   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *   OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *   NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *   HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *   WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *   FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *   OTHER DEALINGS IN THE SOFTWARE.
 */
#pragma once

#include "ofMain.h"
#include "ofxiPhone.h"
#include "ofxiPhoneExtras.h"

#include "maximilian.h"

#include "ofxBox2d.h"

// we create an additional class here that will be used to help us keep track of every
// recorded sound.  
class customData {
public:
    customData()
    {
        myID = currentSoundLUT;
        currentSoundLUT++;
    }
    // this id stores a number that corresponds to a number in a list 
    int myID;
    static int currentSoundLUT;
};


class testApp : public ofxiPhoneApp{
	
    public:
        void setup();
        void update();
        void draw();
        void exit();
	
        void touchDown(ofTouchEventArgs & touch);
        void touchMoved(ofTouchEventArgs & touch);
        void touchUp(ofTouchEventArgs & touch);
        void touchDoubleTap(ofTouchEventArgs & touch);
        void touchCancelled(ofTouchEventArgs & touch);

        void lostFocus();
        void gotFocus();
        void gotMemoryWarning();
        void deviceOrientationChanged(int newOrientation);
    
        void audioOut(float *buf, int size, int ch);
        void audioIn(float *buf, int size, int ch);
    
        
        // this is the function for contacts
        void contactStart(ofxBox2dContactArgs &e);
        void contactEnd(ofxBox2dContactArgs &e);
    
    //	the box2d world
	ofxBox2d						box2d;
    //	default box2d circles
	vector		<ofxBox2dCircle>	circles;		  
    //  storage for all the recordings we make
    vector<vector<float> >          recordings;
    
    // keep a buffer of audio samples for the current recording in progress
    vector<float>                   currentRecording;
    
    // and the current offset for this recording as it is being recorded
    int                             currentRecordingOffset;
    
    // also keep a table of indexes which store all sounds being played back
    vector<int>                     currentPlayingSounds;
    
    // and the offsets or the current sample being played from these sounds
    vector<int>                     currentPlayingSoundsOffset;
    
    // maximilian's maxiDyn which gives us a compressor to compress our sounds
    maxiDyn                         dyn;
    
    // and a boolean to check if we are currently recording
    bool                            bRecording;
};

testApp.mm

/*
 *  Created by Parag K. Mital - http://pkmital.com
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2013 Parag K. Mital. All rights reserved.
 *
 *   Permission is hereby granted, free of charge, to any person
 *   obtaining a copy of this software and associated documentation
 *   files (the "Software"), to deal in the Software without
 *   restriction, including without limitation the rights to use,
 *   copy, modify, merge, publish, distribute, sublicense, and/or sell
 *   copies of the Software, and to permit persons to whom the
 *   Software is furnished to do so, subject to the following
 *   conditions:
 *
 *   The above copyright notice and this permission notice shall be
 *   included in all copies or substantial portions of the Software.
 *
 *   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 *   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *   OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *   NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *   HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *   WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *   FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *   OTHER DEALINGS IN THE SOFTWARE.
 */
#include "testApp.h"

int customData::currentSoundLUT = 0;

//--------------------------------------------------------------
void testApp::setup() {
    bRecording = false;
    
	// initialize the accelerometer
	ofxAccelerometer.setup();
    
    // we initalize the box2d world.  there are parameters for the gravity and the bounding box of this world.
    // we also tell box2d how fast we need it to be.  it will try to run at this speed at best.
    // we also "register grabbing" which means we are able to interact with any objects we add to the box2d world.
	box2d.init();
	box2d.setGravity(0, 10);
	box2d.createBounds();
	box2d.setFPS(30.0);
	box2d.registerGrabbing();

    // add listeners. these are functions that are called only when a certain action is performed
    // we add 2 listeners that are executed whenever box2d's world discovered that some objects
    // have contacted each other.  for instance, a ball hit a wall, or a ball hit another ball.
    // we get 2 listeners, one for the start of the contact, and one for the end of it..
	ofAddListener(box2d.contactStartEvents, this, &testApp::contactStart);
	ofAddListener(box2d.contactEndEvents, this, &testApp::contactEnd);

    // setup maximilian audio settings
    maxiSettings::setup(44100, 1, 512);
    
    // setup openframeworks audio setings
    // (gives us the audioIn/audioOut callbacks)
    ofSoundStreamSetup(1, 1, 44100, 512, 3);
    
    // this forces audio output to the speaker rather than the headphone jack
    // (better to have these commands execute in response to a button press)
    //UInt32 category = kAudioSessionOverrideAudioRoute_Speaker;
    //AudioSessionSetProperty(kAudioSessionProperty_OverrideAudioRoute, sizeof(category), &category);
}

//--------------------------------------------------------------
void testApp::update()
{
    // get the current accelerometer reading as an ofPoint
    ofPoint currentForce = ofxAccelerometer.getForce();
    
    // reverse the y-direction as our screen coordinates are upside down in y
    currentForce.y = -currentForce.y;
    
    // and multiply by a factor to make the force "heavier"
    currentForce *= 9.8;
    
    // now we update our box2d world with the readings from our accelerometer
    box2d.setGravity(currentForce);
    
    // and tell box2d to update everything else
	box2d.update();
}

//--------------------------------------------------------------
void testApp::draw(){
    // set the background to be black every draw iteration
    ofBackground(0);

    // let's draw all circles - they know their internal position
	for(int i=0; i<circles.size(); i++) {
		ofFill();
		ofSetHexColor(0xf6c738);
		circles[i].draw();
	}
	
	// draw the box2d ground
	box2d.drawGround();

}

//--------------------------------------------------------------
void testApp::exit(){

}

//--------------------------------------------------------------
void testApp::audioOut(float *buf, int size, int ch)
{
    // first clear the sound output to be 0 by default
    memset(buf, 0, sizeof(float)*size*ch);
    
    // we create an iterator which goes through the list of sounds
    vector<int>::iterator it = currentPlayingSounds.begin();
    
    // and keep an iterator which keeps the current offset of that sound
    vector<int>::iterator it2 = currentPlayingSoundsOffset.begin();
    
    // now loop through all sounds that are playing
    while(it != currentPlayingSounds.end())
    {
        // we loop through the current audio frame's size and add all recordings that are currently playing
        for (int i = 0; i < size; i++) {
            // we use the currentPlayingSound's iterator to pick out the recording
            // and the offset to index the recording
            buf[i] += (recordings[(*it)][i + *it2]);
        }
        
        // and update the offset's to reflect that we just read "size" number of samples
        // this way, the next audio frame will play the recorded samples "size" samples farther than
        // this audio frame has played
        (*it2) += size;
        
        // we now check if the recorded sample has finished playing
        if((*it2) >= recordings[(*it)].size())
        {
            // if so, we delete it from the sounds that are playing
            it = currentPlayingSounds.erase(it);
            it2 = currentPlayingSoundsOffset.erase(it2);
        }
        else
        {
            // otherwise, keep it in the list, and move on to the next sound
            it++;
            it2++;
        }
    }
    
    // we use a compressor to keep the sound within [-1,1] bounds. otherwise, the sound will clip
    // if we add more than 1 sound to the output... 
    for (int i = 0; i < size; i++) {
        buf[i] = dyn.compressor(buf[i], 0.7);
    }
    
}

//--------------------------------------------------------------
void testApp::audioIn(float *buf, int size, int ch){
    
    // check if the user is recording a sound (pushed down on the screen)
    if (bRecording)
    {
        // get the current recording's size
        int idx = currentRecording.size();
        
        // now change it to be "size" samples bigger
        currentRecording.resize(idx + size);
        
        // loop through the length of size
        for (int i = 0; i < size; i++) {
            
            // and add all the samples of the current audio frame to the current recording
            currentRecording[idx + i] = buf[i];
        }
    }
}



//--------------------------------------------------------------
void testApp::contactStart(ofxBox2dContactArgs &e) {
    if(e.a != NULL && e.b != NULL) {

    }
}



//--------------------------------------------------------------
void testApp::contactEnd(ofxBox2dContactArgs &e) {
    
    // .a and .b store the contacting objects
    // we make sure they are not null, not sure why, but we do... 
    if(e.a != NULL && e.b != NULL) {
        
        // we get the data for each of these objects and CAST it to our custom data type
        customData * aData = (customData*)e.a->GetBody()->GetUserData();
		customData * bData = (customData*)e.b->GetBody()->GetUserData();
		
        // we first make sure that it is a circle, as the walls do not have custom data
		if(e.a->GetType() == b2Shape::e_circle && aData) {
            // then add to our table of playing sounds the circle's ID
			currentPlayingSounds.push_back(aData->myID);
            // also initialize the offset of this sound to be the 0th sample so it starts from the beginning
			currentPlayingSoundsOffset.push_back(0);
		}
		
        // we first make sure that it is a circle, as the walls do not have custom data
		if(e.b->GetType() == b2Shape::e_circle && bData) {
            // then add to our table of playing sounds the circle's ID
			currentPlayingSounds.push_back(bData->myID);
            // also initialize the offset of this sound to be the 0th sample so it starts from the beginning
			currentPlayingSoundsOffset.push_back(0);
		}
    }
}

//--------------------------------------------------------------
void testApp::touchDown(ofTouchEventArgs & touch){

    bRecording = true;
}

//--------------------------------------------------------------
void testApp::touchMoved(ofTouchEventArgs & touch){

}

//--------------------------------------------------------------
void testApp::touchUp(ofTouchEventArgs & touch){
    
    // done recording
    bRecording = false;
    
    // let's add the recorded sound to all of our recordings
    recordings.push_back(currentRecording);
    
    // and erase the buffer which helps us record sounds
    currentRecording.resize(0);
    
    // here we keep a circle that will be modeled by box2d
    float r = 20;
    ofxBox2dCircle circle;
    circle.setPhysics(3.0, 0.53, 0.1);
    circle.setup(box2d.getWorld(), touch.x, touch.y, r);
    circle.setData(new customData());
    circles.push_back(circle);
}

//--------------------------------------------------------------
void testApp::touchDoubleTap(ofTouchEventArgs & touch){

}

//--------------------------------------------------------------
void testApp::touchCancelled(ofTouchEventArgs & touch){
    
}

//--------------------------------------------------------------
void testApp::lostFocus(){

}

//--------------------------------------------------------------
void testApp::gotFocus(){

}

//--------------------------------------------------------------
void testApp::gotMemoryWarning(){

}

//--------------------------------------------------------------
void testApp::deviceOrientationChanged(int newOrientation){

}

Part 2
Before you can develop on a particular device however, you must obtain a certificate for development and a provisioning profile for each device you want to develop on. E-mail me your e-mail address and first and last name. I will send you an invitation to join our developer account so you don’t have to pay for one. After you’ve registered for an iOS account (you should have received an e-mail from me in the lab), log on developer.apple.com and find the iOS Provisioning Portal. If you can’t find the iOS Provisioning Portal, make sure you have been invited to join the team iOS account, and you have signed in with the account you created with this invitation. Try re-logging into developer.apple.com as well. Once you find the iOS Provisioning Portal, click on “certificates”, then the button for “request a certificate”, and follow the steps for requesting a certificate. (You will be asked to upload a “Certificate Signing Request”. For the CA e-mail address, leave it blank. Make sure you save the CSR to disk.) Next, in the iOS Provisioning Portal, find the devices section. Now click on how-to. This will tell you how to locate your device id for your iPad/iPod/iPhone. Send me this 40-bit hex key in an e-mail.

After I’ve received your certificate signing request and device id, I will update your information in the system. You can then go to the iOS Provisioning Portal and download your certificate. Do not do this before I have updated the system. Simply open this certificate and it will be installed in your KeyChain. If your certificate does not appear here, then either you have not submitted a signing request to me, or I have not yet accepted it. Next, in the iOS Provisioning Portal, navigate to “Provisioning”. Find the profile called, “wcc”, and download this file. Open it and it should appear in your XCode’s Organizer.

Part 3
Try installing an application you have made on a device.

Lecture

Provisioning
Please have a look at last week’s lab notes on how to get applications installed on your phone. If you are not an “iOS developer” on our team account, or have not submitted a “certificate signing request”, or have not submitted your device’s identifier to me, you will not be able to develop on a device. Make sure you do these things before downloading a certificate and provisioning profile from the iOS Provisioning Portal.
Core Motion
The ofxUIAccelerometer which ships with openFrameworks v0073 uses a deprecated library for accessing the accelerometer. Please download ofxCoreMotion instead. This gives you access to the Accelerometer, Gyrometer, Magnetometer, Digital Compass.
Core Location
Using core location, you can access the device’s latitude, longitude, and altitude. This is particularly useful when using MapKit, as you can add annotations or reposition the map to particular latitudes and longitudes. You may also think of logging the GPS information, creating a path, for instance, which you could use for any variety of applications (e.g. drawing, 3D printing maybe, I don’t know really).
OSC
Have a look at the oscReceiver and oscSender examples for how to receive and send information from your app across a network. You could talk to a program on your laptop, another iOS device, or through a web server, for instance.
Image Picker
openFrameworks provides a simple interface to the iOS Image Picker, ofxiPhoneImagePicker. This is a handy little class which lets you access the photo library and take images/videos.
Twitter
Starting in iOS 5, it is very easy to send twitter message, get a list of tweets for a user, and attach photos or links to a tweet, using the Twitter Framework. In iOS 6, there is also native Facebook integration.
Twitter + Image Picker example
You’ll have to add the “Twitter” framework to your project (Build Phases/Link Binary With Libraries). Also, I edited the ofxiPhoneImagePicker (addons/ofxiPhone/src/utils) class to include the function on lines 118:

    UIImage * getUIImage()
    {
        return [imagePicker getUIImage];
    }

testApp.h

#pragma once

#include "ofMain.h"
#include "ofxiPhone.h"
#include "ofxiPhoneExtras.h"

class testApp : public ofxiPhoneApp{
	
    public:
        void setup();
        void update();
        void draw();
        void exit();
	
        void touchDown(ofTouchEventArgs & touch);
        void touchMoved(ofTouchEventArgs & touch);
        void touchUp(ofTouchEventArgs & touch);
        void touchDoubleTap(ofTouchEventArgs & touch);
        void touchCancelled(ofTouchEventArgs & touch);

        void lostFocus();
        void gotFocus();
        void gotMemoryWarning();
        void deviceOrientationChanged(int newOrientation);
    
    ofImage img;
    ofxiPhoneImagePicker imgPicker;

};

testApp.mm

#include "testApp.h"
#import <Twitter/TWTweetComposeViewController.h>

//--------------------------------------------------------------
void testApp::setup(){	

}

//--------------------------------------------------------------
void testApp::update(){
    if (imgPicker.imageUpdated) {
        imgPicker.imageUpdated = false;
        
        TWTweetComposeViewController *tweetView = [[TWTweetComposeViewController alloc] init];
        
        TWTweetComposeViewControllerCompletionHandler
        completionHandler =
        ^(TWTweetComposeViewControllerResult result) {
            switch (result)
            {
                case TWTweetComposeViewControllerResultCancelled:
                    NSLog(@"Twitter Result: canceled");
                    break;
                case TWTweetComposeViewControllerResultDone:
                    NSLog(@"Twitter Result: sent");
                    break;
                default:
                    NSLog(@"Twitter Result: default");
                    break;
            }
            [ofxiPhoneGetViewController() dismissModalViewControllerAnimated:YES];
        };
        [tweetView setCompletionHandler:completionHandler];
        [tweetView addImage:imgPicker.getUIImage()];    // i added a simple "getter" to the image picker class to return the UIImage
        [tweetView setInitialText:@"Look how freaking easy it is to send a tweet using openFrameworks and iOS 5+"];

        [ofxiPhoneGetViewController() presentModalViewController:tweetView animated:YES];
                                   
        imgPicker.close();

    }
}

//--------------------------------------------------------------
void testApp::draw(){
	
}

//--------------------------------------------------------------
void testApp::exit(){

}

//--------------------------------------------------------------
void testApp::touchDown(ofTouchEventArgs & touch){
    imgPicker.openCamera();
}

//--------------------------------------------------------------
void testApp::touchMoved(ofTouchEventArgs & touch){

}

//--------------------------------------------------------------
void testApp::touchUp(ofTouchEventArgs & touch){

}

//--------------------------------------------------------------
void testApp::touchDoubleTap(ofTouchEventArgs & touch){

}

//--------------------------------------------------------------
void testApp::touchCancelled(ofTouchEventArgs & touch){
    
}

//--------------------------------------------------------------
void testApp::lostFocus(){

}

//--------------------------------------------------------------
void testApp::gotFocus(){

}

//--------------------------------------------------------------
void testApp::gotMemoryWarning(){

}

//--------------------------------------------------------------
void testApp::deviceOrientationChanged(int newOrientation){

}

Lab

Please make sure you have organized a group and have discussed with me what you plan to do. You have to submit a 300 word document as a group to me by the end of the night. Make sure you say what you plan on doing, how you plan to do it, and why it is interesting. A good structure for writing this document should start with a context (who cares about it?), a motivation (why is it important?), your approach (how will you do it?), how your approach is integral to what you set out to do (why your approach is a good idea?), and what you plan to achieve (what will your approach achieve?). Remember I don’t expect miracles, but I do expect well thought out ideas. Be sure to also include who the group members are, how many devices you have, and if you need a device.

Lecture

Obtaining the current network SSID
You can use the short code fragment below to figure out the SSID of the current WiFi connection. You will have to add the framework, SystemConfiguration, for this to work.

...

#import <SystemConfiguration/CaptiveNetwork.h>

...

{

...

    // getting an array of supported networking interfaces
    NSArray *ifs = (id)CNCopySupportedInterfaces();
    
    // now output them 
    NSLog(@"%s: Supported interfaces: %@", __func__, ifs);
    
    // can also get info about each interface
    id info = nil;
    
    // let's loop through each interface
    for (NSString *ifnam in ifs) {
        
        // get the info for it
        info = (id)CNCopyCurrentNetworkInfo((CFStringRef)ifnam);
        
        // and output its info, such as what network SSID it is currently connected to
        NSLog(@"%s: %@ => %@", __func__, ifnam, info);
        if (info && [info count]) {
            break;
        }
        
        // release the memory
        [info release];
    }
    [ifs release];

...

}

Adding Annotations to MapKit
You’ll have to modify ofxiPhoneMapKit.h and ofxiPhoneMapKit.mm to include the following (note, do not add the ellipses, …, obviously):

ofxiPhoneMapKit.h

...

class ofxiPhoneMapKit : public ofxiPhoneMapKitListener {
public:

...

    void selectedAnnotation(string title, string subtitle);
    void unselectedAnnotation(string title, string subtitle);
    void addAnnotation(double latitude, double longitude, string title, string subtitle);
    void removeAnnotation(double latitude, double longitude, string title, string subtitle);
    void removeAnnotation(string title, string subtitle);

...

private:

...

    vector< MKPointAnnotation * > annotations;

...

};

ofxiPhoneMapKit.mm

...

void ofxiPhoneMapKit::selectedAnnotation(string title, string subtitle) {
	for(std::list<ofxiPhoneMapKitListener*>::iterator it=listeners.begin(); it!=listeners.end(); ++it) {
		ofxiPhoneMapKitListener* o = *it;
		o->selectedAnnotation(title, subtitle);
	}
}


void ofxiPhoneMapKit::unselectedAnnotation(string title, string subtitle) {
	for(std::list<ofxiPhoneMapKitListener*>::iterator it=listeners.begin(); it!=listeners.end(); ++it) {
		ofxiPhoneMapKitListener* o = *it;
		o->unselectedAnnotation(title, subtitle);
	}
}


void ofxiPhoneMapKit::addAnnotation(double latitude,
                                    double longitude,
                                    string title,
                                    string subtitle)
{
    CLLocationCoordinate2D annotationCoord;
    
    annotationCoord.latitude = latitude;
    annotationCoord.longitude = longitude;
    
    MKPointAnnotation *annotationPoint = [[MKPointAnnotation alloc] init];
    annotationPoint.coordinate = annotationCoord;
    annotationPoint.title = ofxStringToNSString(title);
    annotationPoint.subtitle = ofxStringToNSString(subtitle);
    [mapView addAnnotation:annotationPoint];
    annotations.push_back(annotationPoint);
}

void ofxiPhoneMapKit::removeAnnotation(string title,
                                       string subtitle)
{
    
    for (vector<MKPointAnnotation *>::iterator it = annotations.begin(); it != annotations.end(); it++) {
        if ([(*it).title isEqualToString:ofxStringToNSString(title)] &&
            [(*it).subtitle isEqualToString:ofxStringToNSString(subtitle)]) {
            ofLog(OF_LOG_NOTICE, "Removing annotation\n");
            [mapView removeAnnotation:(*it)];
            [(*it) release];
            annotations.erase(it);
            break;
        }
    }
    
}

void ofxiPhoneMapKit::removeAnnotation(double latitude,
                                       double longitude,
                                       string title,
                                       string subtitle)
{
    
    for (vector<MKPointAnnotation *>::iterator it = annotations.begin(); it != annotations.end(); it++) {
        if ((*it).coordinate.latitude == latitude &&
            (*it).coordinate.longitude == longitude &&
            [(*it).title isEqualToString:ofxStringToNSString(title)] &&
            [(*it).subtitle isEqualToString:ofxStringToNSString(subtitle)]) {
            [mapView removeAnnotation:(*it)];
            [(*it) release];
            annotations.erase(it);
            break;
        }
    }
    
}

...

Now we register the 2 new callbacks to be part of the ofxiPhoneMapKitListener class by editing the ofxiPhoneMapKitListener.h file to look like:

ofxiPhoneMapKitListener.h

...

class ofxiPhoneMapKitListener { 
public:
    virtual ~ofxiPhoneMapKitListener() {};
	virtual void regionWillChange(bool animated) {}
	virtual void regionDidChange(bool animated) {}
	virtual void willStartLoadingMap() {}
	virtual void didFinishLoadingMap() {}
	virtual void errorLoadingMap(string errorDescription) {}
	virtual void selectedAnnotation(string title, string subtitle) {}
	virtual void unselectedAnnotation(string title, string subtitle) {}
};

These functions are virtual, meaning they are allowed to be overridden by any classes that inherit from this one. By default, if you do not override them, they are defined to be {}, meaning they do nothing. If you want them to do something more, you can edit your testApp.h and add the declaration of these two functions. They will get called whenever the user selects/unselects any annotations that have been added to the mapkit. Add them to testApp.h and testApp.mm like so:

testApp.h

...

class testApp : public ofxiPhoneApp, ofxiPhoneMapKitListener{
public:

...

        // additional optional callbacks for annotations
        void selectedAnnotation(string title, string subtitle);
        void unselectedAnnotation(string title, string subtitle);

...

};

testApp.mm

...

//--------------------------------------------------------------
void testApp::selectedAnnotation(string title, string subtitle){
	printf("testApp::selectedAnnotation | title: %s, subtitle: %s\n", title.c_str(), subtitle.c_str());
}

//--------------------------------------------------------------
void testApp::unselectedAnnotation(string title, string subtitle){
	printf("testApp::unselectedAnnotation | title: %s, subtitle: %s\n", title.c_str(), subtitle.c_str());
}

...

Assuming you have created an object mapKit of type ofxiPhoneMapKit and coreLocation of type ofxiPhoneCoreLocation and did all the proper setup (have a look at the examples if you are unsure), you can easily add annotations to the map by writing:

mapKit.addAnnotation(coreLocation.getLatitude(), coreLocation.getLongitude(), "This is the Title", "I'm the subtitle...");

Gestures
If you want to use some of the SDK’s inbuilt gesture recognizers, you can easily get started with this addon, ofxGestureRecognizer. This lets you detect pan, pinch, swipe, and tap gestures.

Alternatives
Being the last lecture focusing on iOS development with openFrameworks, I wanted to give you a few other popular approaches you could take for programming 2d/3d audiovisual interaction/games.

Native
Meaning only using the iOS SDK. Of course you can create graphics in CoreAnimation etc… and interact with the device using the native libraries. If you require physics however you may not want to recreate these libraries, as they don’t exist in the iOS SDK. Requires the most amount of time/effort.
cocos2d
Focuses on extending 2D interaction. Community-driven github repository. OpenGLES 2.0ES support. Very popular so lots of community resources. Also has HTML5 implementation.
oolong
Focus on extending 3D interaction. Incorporates Bullet Physics Engine. Very open google code repository. OpenGLES 2.0ES support. Good for those that already program OpenGLES. Not much documentation.
libcinder
Very similar to openFrameworks. Very nicely written code. Templated. Generally not as targeted for education/learning as openFrameworks is, so for people more comfortable with coding. OpenGLES 2.0+ support.
Unity3D
OpenGL 2.0ES+ support. Focus on 3D interaction. Very powerful and simple to use. Licensing options required for all features and depending on how you sell your app/game.

Lab

Work on your project!

Introduction

The next 5 weeks will focus on developing an understanding of what is possible with computer vision. In particular, we will focus on using Intel’s OpenCV within the openFrameworks library. Your final project will be similar to the mobile module where you will work in groups to create an application/game/installation/experience based in computer vision. For the next 5 weeks, makes sure you have a working version of openFrameworks v074+ for your operating system. Note that we will not be restricted to using XCode/OSX for these weeks (HOORAY!).

OpenCV Links

Keep in mind links for the OpenCV documentation and Wiki pages. Actually, this page is a good compendium of different resources, and also includes a link to a PDF Manual of OpenCV 2.3.1. More information is also in each of the lab sheets below.

Other Computer Vision courses

Bob Fisher’s IVR Introduction to Vision and Robotics & AV Advanced Vision classes at University of Edinburgh
James Hays’s CS 143 Introduction to Computer Vision class at Brown
Trevor Darrell’s CS 280 Computer Vision class at University of California, Berkeley
Antonio Torralba’s 6.869 Advances in Computer Vision class at MIT
Kristen Grauman’s CS 378 Computer Vision class at University of Texas, Austin

Recommended Books

Forsyth & Ponce – Computer Vision: a Modern Approach
Norvig – Artificial Intelligence: a Modern Approach
Bishop – Pattern Recognition and Machine Learning
Ballard & Brown – Computer Vision [free eBook]

Lecture

lecture-1-introduction-to-computer-vision-optimized

Lab

cv1

Interactive RGB Colorspace

testApp.h

 
#pragma once

#include "ofMain.h"
#include "ofxCvMain.h"

class testApp : public ofBaseApp{

public:
		
    // redeclaration of functions (declared in base class)
    void setup();
    void update();
    void draw();

    void keyPressed(int key);

    ofVideoGrabber camera;
    
    ofxCvColorImage im_color;
    ofxCvGrayscaleImage im_red, im_green, im_blue;
    ofxCvGrayscaleImage im_gray;
    ofxCvGrayscaleImage im_value;
    

    int imgWidth, imgHeight;
    
    bool bShowBlended;
};

testApp.cpp

#include "testApp.h"

// here we "define" the methods we "declared" in the "testApp.h" file

// i get called once
void testApp::setup(){
	
	// do some initialization
	imgWidth = 320;
    imgHeight = 240;
    
    bShowBlended = false;
    
	// set the size of the window
	ofSetWindowShape(imgWidth * 6, imgHeight);
	
	// the rate at which the program runs (FPS)
	ofSetFrameRate(30);
	
	// setup the camera
	camera.initGrabber(imgWidth, imgHeight);
    im_color.allocate(imgWidth, imgHeight);
    im_red.allocate(imgWidth, imgHeight);
    im_green.allocate(imgWidth, imgHeight);
    im_blue.allocate(imgWidth, imgHeight);
    im_gray.allocate(imgWidth, imgHeight);
    im_value.allocate(imgWidth, imgHeight);
}

// i get called in a loop that runs until the program ends
void testApp::update(){
	camera.update();
    
    if(camera.isFrameNew())
    {
        // copy the pixels from the camera object into an ofxCvColorImage object
        im_color.setFromPixels(camera.getPixels(), imgWidth, imgHeight);
        
        im_gray = im_color;
        
        // get each color channel
        im_color.convertToGrayscalePlanarImages(im_red, im_green, im_blue);
        
        im_color.convertRgbToHsv();
        im_color.convertToGrayscalePlanarImage(im_value, 2);
    }
}

// i also get called in a loop that runs until the program ends
void testApp::draw(){
    // background values go to 0
	ofBackground(0);
	
	// draw the camera
    ofSetColor(255, 255, 255);
	camera.draw(imgWidth * 0,0);
    
    if(bShowBlended)
    {
        // blending mode for adding pictures together
        ofEnableAlphaBlending();
        ofEnableBlendMode(OF_BLENDMODE_ADD);
        
        // full red energy
        ofSetColor(255, 0, 0);
        im_red.draw(imgWidth * 1,0);
        
        // full green energy
        ofSetColor(0, 255, 0);
        
        // draw using an offset from the center of the screen determined by the mouse position
        im_green.draw(imgWidth * 1 + (mouseX - ofGetScreenWidth()/2) / 10.0,0);
        
        // full blue energy
        ofSetColor(0, 0, 255);
        
        // offset just like above, but 2x as much
        im_blue.draw(imgWidth * 1 + (mouseX - ofGetScreenWidth()/2) / 5.0,0);
        
        ofDisableAlphaBlending();
    }
    else
    {
        ofSetColor(255, 0, 0);
        im_red.draw(imgWidth * 1,0);
        
        ofSetColor(0, 255, 0);
        im_green.draw(imgWidth * 2,0);
        
        ofSetColor(0, 0, 255);
        im_blue.draw(imgWidth * 3,0);
    }
    
    ofSetColor(255, 255, 255);
    im_gray.draw(imgWidth * 4, 0);
    im_value.draw(imgWidth * 5, 0);
}

void testApp::keyPressed(int key)
{
	switch (key) {
		case 's':
			camera.videoSettings();
			break;
            
        // press space to switch between modes    
        case ' ':
            bShowBlended = !bShowBlended;
            break;
		default:
			break;
	}
}

Motion Tracking + Interactive Video Player

testApp.h

#pragma once

#include "ofMain.h"
#include "ofxOpenCv.h"

class testApp : public ofBaseApp{

public:
    void setup();
    void update();
    void draw();	
     
    float                   alpha;
    float					sum;
    ofVideoGrabber			camera;

    ofxCvColorImage			color_img;

    ofxCvGrayscaleImage 	gray_img;
    ofxCvGrayscaleImage 	gray_previous_img;
    ofxCvGrayscaleImage 	gray_diff;

    ofVideoPlayer			video;

    vector<ofxCvGrayscaleImage> previous_imgs;

    int                     img_width, img_height;
};

testApp.cpp

#include "testApp.h"

using namespace cv;

//--------------------------------------------------------------
void testApp::setup(){

    // keep variables for our image size
    img_width = 320;
    img_height = 240;
    
    // value for our first order linear filter
    // this controls how much we mix in previous results
    // the larger the value, the larger the weight of the 
    // previous result.  this is a very essential and basic
    // technique in digital signal processing also known as a 
    // low pass filter.
    alpha = 0.5;
    
	// change the window to hold enough space for 2 movies (1 row x 2 columns of movies)
	ofSetWindowShape(img_width * 2, img_height);
	
	ofSetFrameRate(30);
	
	// initialize our camera with a resolution of 320z240
	camera.initGrabber(img_width, img_height);
	
	// load a movie in and set it to loop, and then start it (play())
	video.loadMovie("sunra_pink.mov");
	video.setLoopState(OF_LOOP_NORMAL);
	video.play();
	
	sum = 0;
	
	// these are (wrappers for) opencv image containers 
	// we'll use for image processing
	// we are going to find the difference between successive frames
    color_img.allocate(img_width, img_height);
	gray_img.allocate(img_width, img_height);
	gray_previous_img.allocate(img_width, img_height);
	gray_diff.allocate(img_width, img_height);
	previous_imgs.push_back(gray_previous_img);
	previous_imgs.push_back(gray_previous_img);
	previous_imgs.push_back(gray_previous_img);
	
}

//--------------------------------------------------------------
void testApp::update(){
	// background to black
	ofBackground(0);
	
	// update the camera
	camera.update();
	
	if (camera.isFrameNew()) {
		// set the color image (opencv container) to the camera image
		color_img.setFromPixels(camera.getPixels(), img_width, img_height);
		// convert to grayscale
		gray_img = color_img;
		// calculate the difference image
		gray_diff = gray_img;
		// compute the absolute difference with the previous frame's grayscale image
		gray_diff.absDiff(previous_imgs[0]);
		
        Mat diff_mat(gray_diff.getCvImage());
        Scalar s1 = mean(diff_mat);
        
        
		// store the current grayscale image for the next iteration of update()
		previous_imgs.push_back(gray_img);
		if (previous_imgs.size() > 10) {
			previous_imgs.erase(previous_imgs.begin());
		}
		
		// let's threshold the difference image,
		// all values less than 10 are 0, all values above 10 become 255
		//gray_diff.threshold(10);
		
		// here we will find the sum and then average of all the pixels in the difference image
		// this will be used for a simple measure of "motion" 
        // we use a low-pass filter, a first order filter which combines the current 
        // value with the previous one, using a linear weighting.
		sum = (alpha) * sum + 
        (1 - alpha) * s1[0] / 10.0f;//cvSum(gray_diff.getCvImage()).val[0] / (float)img_width / (float)img_height / 10.0;
		
        
		// let's change the speed of our movie based on the motion value we calculated
		video.setSpeed(sum);
		video.update();
	}
	
	
}

//--------------------------------------------------------------
void testApp::draw(){
	ofEnableAlphaBlending();
	ofEnableBlendMode(OF_BLENDMODE_ADD);
	
	color_img.draw(0, 0, img_width, img_height);
	gray_diff.draw(0, 0, img_width, img_height);

	ofDisableAlphaBlending();
    
    video.draw(img_width, 0);
    
	// draw the sum of the motion pixels divided by the number of motion pixels 
	// (average of difference values)
	char buf[256];
	sprintf(buf, "%f", sum);
	ofDrawBitmapString(buf, 20, 20);
}

Lecture

lecture-2-image-features-optimized

Lab

You’ll need this zip file for the second part of the lab: [pkmImageFeatureDetector.zip]
cv2

Detecting and Drawing Image Features

testApp.h

 
#pragma once

#include "ofMain.h"
#include "ofxCvMain.h"

using namespace cv;

class testApp : public ofBaseApp{

public:
		
    // redeclaration of functions (declared in base class)
    void setup();
    void setupFeatures();
    
    void update();
    void draw();

    void keyPressed(int key);
    
    int width, height;
    float scalar;
    
    ofVideoGrabber camera;

    ofxCvColorImage cv_color_img;
    ofxCvGrayscaleImage cv_luminance_img;
    
    Mat mat_image;
    
    unsigned int current_detector; 
    vector<string> feature_detectors;
    
    cv::Ptr<FeatureDetector> feature_detector;
    vector<KeyPoint> keypoints;
};

testApp.cpp

#include "testApp.h"

// here we "define" the methods we "declared" in the "testApp.h" file

// i get called once
void testApp::setup(){
    
    scalar = 1.0f;
    width = 640;
    height = 480;
    
    camera.initGrabber(width, height);
    
    cv_color_img.allocate(width, height);
    cv_luminance_img.allocate(width, height);
    
    feature_detectors.push_back("SURF");
    feature_detectors.push_back("DynamicSURF");
    feature_detectors.push_back("PyramidSURF");
    feature_detectors.push_back("GridSURF");
    feature_detectors.push_back("SIFT");
    feature_detectors.push_back("STAR");
    feature_detectors.push_back("DynamicSTAR");
    feature_detectors.push_back("PyramidSTAR");
    feature_detectors.push_back("GridSTAR");
    feature_detectors.push_back("FAST");
    feature_detectors.push_back("DynamicFAST");
    feature_detectors.push_back("PyramidFAST");
    feature_detectors.push_back("GridFAST");
    feature_detectors.push_back("GFTT");
    feature_detectors.push_back("PyramidGFTT");
    feature_detectors.push_back("MSER");
    feature_detectors.push_back("PyramidMSER");
    feature_detectors.push_back("HARRIS");
    feature_detectors.push_back("PyramidHARRIS");
    
    current_detector = 0;
    feature_detector = FeatureDetector::create(feature_detectors[current_detector]);
    
    ofSetFrameRate(60.0f);
    ofSetWindowShape(width * scalar, height * scalar);
}

// i get called in a loop that runs until the program ends
void testApp::update(){
    camera.update();
    if (camera.isFrameNew()) {
        cv_color_img.setFromPixels(camera.getPixelsRef());
        cv_color_img.convertRgbToHsv();
        cv_color_img.convertToGrayscalePlanarImage(cv_luminance_img, 2);
        
        mat_image = Mat(cv_luminance_img.getCvImage());
        
        keypoints.clear();
        feature_detector->detect(mat_image, keypoints);
    }
}

// i also get called in a loop that runs until the program ends
void testApp::draw(){
    ofBackground(0);
    
    ofPushMatrix();
    ofScale(scalar, scalar);
    
    ofSetColor(255, 255, 255);
    camera.draw(0, 0);
    
    ofNoFill();
    ofSetColor(200, 100, 100);
    vector<KeyPoint>::iterator it = keypoints.begin();
    while(it != keypoints.end())
    {
        ofPushMatrix();
        float radius = it->size/2;
        ofTranslate(it->pt.x - radius, it->pt.y - radius, 0);
        ofRotate(it->angle, 0, 0, 1);
        ofRect(0, 0, radius, radius);
        ofPopMatrix();
        it++;
    }
    ofPopMatrix();
    
    ofSetColor(255, 255, 255);
    string draw_string = ofToString(current_detector+1) + string("/") + 
                        ofToString(feature_detectors.size()) + string(": ") +
                        feature_detectors[current_detector];
    ofDrawBitmapString(draw_string, 20, 20);
    draw_string = string("# of features: ") + ofToString(keypoints.size());
    ofDrawBitmapString(draw_string, 20, 35);
    draw_string = string("fps: ") + ofToString(ofGetFrameRate());
    ofDrawBitmapString(draw_string, 20, 50);
}

void testApp::keyPressed(int key){
    if(key == 'n')
    {
        current_detector = (current_detector + 1) % feature_detectors.size();
        feature_detector = FeatureDetector::create(feature_detectors[current_detector]);
    }
}

Detecting Planar Objects

testApp.h

/*
*  Created by Parag K. Mital - http://pkmital.com 
*  Contact: parag@pkmital.com
*
*  Copyright 2011 Parag K. Mital. All rights reserved.
* 
*	Permission is hereby granted, free of charge, to any person
*	obtaining a copy of this software and associated documentation
*	files (the "Software"), to deal in the Software without
*	restriction, including without limitation the rights to use,
*	copy, modify, merge, publish, distribute, sublicense, and/or sell
*	copies of the Software, and to permit persons to whom the
*	Software is furnished to do so, subject to the following
*	conditions:
*	
*	The above copyright notice and this permission notice shall be
*	included in all copies or substantial portions of the Software.
*
*	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,	
*	EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
*	OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
*	NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
*	HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
*	WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
*	FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
*	OTHER DEALINGS IN THE SOFTWARE.
*/
#ifndef _TEST_APP
#define _TEST_APP

#include "ofMain.h"

#include "pkmImageFeatureDetector.h"

#include "ofxOpenCv.h"

const int CAM_WIDTH = 320;
const int CAM_HEIGHT = 240;
const int SCREEN_WIDTH = CAM_WIDTH*2;
const int SCREEN_HEIGHT = CAM_HEIGHT + 75;

class testApp : public ofBaseApp {

	public:

	~testApp();
	void setup();
	 
	void update();
	void draw();
    void drawKeypoints(vector<KeyPoint> keypts);

	void keyPressed  (int key);
	void mouseMoved(int x, int y );
	void mouseDragged(int x, int y, int button);
	void mousePressed(int x, int y, int button);
	void mouseReleased(int x, int y, int button);
	void windowResized(int w, int h);
	
	ofVideoGrabber			camera;
	
	ofxCvColorImage			color_img, color_roi_img;
	ofxCvGrayscaleImage		gray_search_img, 
                            gray_template_img;
    
	float					x_start, 
                            x_end, 
                            y_start, 
                            y_end;
    
    cv::Point2f             low_pass_bounding_box[4],
                            prev_pass_bounding_box[4];
    
    float                   alpha;
	
    bool					choosing_img, 
                            chosen_img;
	
    pkmImageFeatureDetector	detector;
    
    vector<cv::KeyPoint>    img_template_keypoints,
                            img_search_keypoints;
	
};
#endif

testApp.cpp

/*
 *  Created by Parag K. Mital - http://pkmital.com 
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2011 Parag K. Mital. All rights reserved.
 * 
 *	Permission is hereby granted, free of charge, to any person
 *	obtaining a copy of this software and associated documentation
 *	files (the "Software"), to deal in the Software without
 *	restriction, including without limitation the rights to use,
 *	copy, modify, merge, publish, distribute, sublicense, and/or sell
 *	copies of the Software, and to permit persons to whom the
 *	Software is furnished to do so, subject to the following
 *	conditions:
 *	
 *	The above copyright notice and this permission notice shall be
 *	included in all copies or substantial portions of the Software.
 *
 *	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,	
 *	EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *	OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *	NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *	HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *	WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *	FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *	OTHER DEALINGS IN THE SOFTWARE.
 */

#include "testApp.h"
//--------------------------------------------------------------
testApp::~testApp(){
}
void testApp::setup(){
	
	// init video input
	camera.initGrabber(CAM_WIDTH,CAM_HEIGHT);
	camera.setUseTexture(true);
	
	// window setup
	ofSetWindowShape(SCREEN_WIDTH, SCREEN_HEIGHT);
	ofSetVerticalSync(true);
	ofSetFrameRate(60);
	ofBackground(0,0,0);
	
	// allocate stuff
	color_img.allocate(CAM_WIDTH, CAM_HEIGHT);
	gray_search_img.allocate(CAM_WIDTH, CAM_HEIGHT);
    
    alpha = 0.6;
    
	choosing_img = false;
	chosen_img = false;
	
}

//--------------------------------------------------------------
void testApp::update(){
	
	camera.update();
	if(camera.isFrameNew())
	{
		// get camera img into iplimage
		color_img.setFromPixels(camera.getPixels(), CAM_WIDTH, CAM_HEIGHT);
		color_img.convertRgbToHsv();
		if (chosen_img) {
			color_img.convertToGrayscalePlanarImage(gray_search_img, 2);
			detector.setImageSearch(gray_search_img.getCvImage());
			detector.update();
            
            img_search_keypoints = detector.getImageSearchKeypoints();
            
            ofCircle(detector.dst_corners[0].x, detector.dst_corners[0].y, 10);
            
            low_pass_bounding_box[0] = detector.dst_corners[0] * (1-alpha) + prev_pass_bounding_box[0] * alpha;
            low_pass_bounding_box[1] = detector.dst_corners[1] * (1-alpha) + prev_pass_bounding_box[1] * alpha;
            low_pass_bounding_box[2] = detector.dst_corners[2] * (1-alpha) + prev_pass_bounding_box[2] * alpha;
            low_pass_bounding_box[3] = detector.dst_corners[3] * (1-alpha) + prev_pass_bounding_box[3] * alpha;
            
            prev_pass_bounding_box[0] = low_pass_bounding_box[0];
            prev_pass_bounding_box[1] = low_pass_bounding_box[1];
            prev_pass_bounding_box[2] = low_pass_bounding_box[2];
            prev_pass_bounding_box[3] = low_pass_bounding_box[3];
		}
	} 
}

//--------------------------------------------------------------
void testApp::draw(){
	ofBackground(0,0,0);
	
    ofSetColor(255, 255, 255);
	// camera image
	camera.draw(0, 0);
    
	if (chosen_img) {
        ofSetColor(200, 100, 100);
        drawKeypoints(img_search_keypoints);
        
        
		ofPushMatrix();
		ofTranslate(CAM_WIDTH, 0, 0);
        ofSetColor(255, 255, 255);
		gray_template_img.draw(0, 0);
        ofSetColor(200, 100, 100);
        drawKeypoints(img_template_keypoints);
		ofPopMatrix();
		
		ofSetColor(200, 200, 200);
		
		ofLine(low_pass_bounding_box[0].x, low_pass_bounding_box[0].y,
			   low_pass_bounding_box[1].x, low_pass_bounding_box[1].y);
		
		ofLine(low_pass_bounding_box[2].x, low_pass_bounding_box[2].y,
			   low_pass_bounding_box[1].x, low_pass_bounding_box[1].y);
		
		ofLine(low_pass_bounding_box[2].x, low_pass_bounding_box[2].y,
			   low_pass_bounding_box[3].x, low_pass_bounding_box[3].y);
		
		ofLine(low_pass_bounding_box[0].x, low_pass_bounding_box[0].y,
			   low_pass_bounding_box[3].x, low_pass_bounding_box[3].y);
        
	}
	
    
	// draw a rectanlge around the current selection
	if (choosing_img) {
		int x = mouseX;
		int y = mouseY;
		
		ofNoFill();
		ofRect(x_start < x ? x_start : x, 
			   y_start < y ? y_start : y, 
			   abs(x_start - x), 
			   abs(y_start - y));
		
	}
	
	
}

void testApp::drawKeypoints(vector<KeyPoint> keypts)
{
    vector<KeyPoint>::iterator it = keypts.begin();
    while(it != keypts.end())
    {
        ofPushMatrix();
        float radius = it->size/2;
        ofTranslate(it->pt.x - radius, it->pt.y - radius, 0);
        ofRotate(it->angle, 0, 0, 1);
        ofRect(0, 0, radius, radius);
        ofPopMatrix();
        it++;
    }

}


//--------------------------------------------------------------
void testApp::keyPressed  (int key){
	
	switch (key){			
		case 's':
			camera.videoSettings();
			break;
        case 'n':
        {
            detector.changeDetector();
            if(chosen_img)
               img_template_keypoints = detector.getImageTemplateKeypoints();
            break;
        }
		case '1':
			break;
		case '2':
			break;
			
		case 'b':
			break;
			
	}
}

//--------------------------------------------------------------
void testApp::mouseMoved(int x, int y ){
}

//--------------------------------------------------------------
void testApp::mouseDragged(int x, int y, int button){
}

//--------------------------------------------------------------
void testApp::mousePressed(int x, int y, int button){
	
	// start a rectangle selection
	if(!choosing_img)
	{
		choosing_img = true;
		x_start = x;
		y_start = y;
	}
}

//--------------------------------------------------------------
void testApp::mouseReleased(int x, int y, int button){
	
	// end the rectangle selection
	if (choosing_img) {
		choosing_img = false;
		x_end = x;
		y_end = y;
		
		if(x_start > x_end)
			std::swap(x_start, x_end);
		if(y_start > y_end)
			std::swap(y_start, y_end);
		
		int w = x_end - x_start;
		int h = y_end - y_start;
        
		
		cvSetImageROI(color_img.getCvImage(), 
					  cvRect(x_start, 
							 y_start, 
							 w, h));
		
		if (color_roi_img.bAllocated) {
			gray_template_img.clear();
            color_roi_img.clear();
		}
		gray_template_img.allocate(w, h);
        color_roi_img.allocate(w, h);
        color_roi_img = color_img;
        color_roi_img.convertToGrayscalePlanarImage(gray_template_img, 2);
		cvResetImageROI(color_img.getCvImage());

		detector.setImageTemplate(gray_template_img.getCvImage());
        
        img_template_keypoints = detector.getImageTemplateKeypoints();
        
		chosen_img = true;
	}
	
}

//--------------------------------------------------------------
void testApp::windowResized(int w, int h){
	
}

Lecture

This week we will develop a system for blob detection and tracking.

lecture-3-blob-tracking-optimized

Lab

For the second part of the lab, you’ll need this zip file: [pkmBlobTracker.zip]. Extract the contents of this zip file into the “src” directory of a new openframeworks project that references the ofxOpenCv addon. Be sure that these files are added to your project by simply dragging them to your project navigator/solution explorer (where you see the list of files).
cv3

Lecture

There are no slides for this or next week. Instead, we will look at code together. This week, we focus on faces.

There are few specific tasks you may want to achieve when looking at faces:

* Detection: Think of this as a binary exercise. Is this is a face or not?
* Recognition: Think of this as a classification of a possible set of faces. Is this John or Jill?
* Tracking: This involves either of the previous stages, as well as understanding the same face has persisted over time. From frame t to frame t+1, is this the same face I just saw?
* 2D/3D: If you just want the bounding box of the face in an image, this is 2D detection. However, if you’d also like to know the face’s orientation, this is knowing the object’s pose, and is a 3D detection problem.
* Expression: This is classifying the information in the face to a set of possible emotions. These emotions can be discrete states meaning only one of a possible set of emotions are possible. Or, more likely, these are “valences”, and are likelihoods of being any possible emotion.

If you have already had a look at the OpenCV examples that come with openFrameworks, you’ll have seen the “opencvHaarFinderExample”. This is an example of using Haar-like features for detecting a face. It’s not very good, but it is one of the most commonly used face detectors since it is easily accessible. Many better face detectors will use this one as a “first-pass”, and then prune out the false-positives with a better detection algorithm (e.g. ASM/AAM models).

If you’d like to know more about the face though, such as its pose or expression, you’ll have to start looking at other models. This paper is in my opinion the current state-of-the-art for face detection, pose estimation, and facial shape modeling. There is a C++ library here.

There is also the work of Jason Saragih. He has built a very nice model called FaceTracker for discovering landmarks of faces which you could use to train a facial expression library. Check out Jason’s vimeo page to see some examples of how this technology can be extended. Kyle McDonald has also written a very extensive openFrameworks library for using this library called ofxFaceTracker. He has also demonstrated its use with Arturo Castro on Face Substitution. Have a look at Kyle’s course on Appropriating New Technologies to see other information on facial detection and also many cultural references.

To get started with FaceTracker, you have a lot of options. You can go straight to Jason’s course and start detecting faces without openFrameworks. Or you can get into ofxFaceTracker and see the examples Kyle has created. Just a note, ofxFaceTracker requires ofxCv. Some examples in ofxFaceTracker also require ofxControlPanel. Be sure to have both of these addons copied to your addons folder of openFrameworks. You can also use FaceOsc, a program written by Kyle to stream information via OSC from FaceTracker. This is a binary application (no source required, though it is in ofxFaceTracker’s branch), so you can get started with face tracking right away if you’d like.

Inside of ofxFaceTracker, Kyle has also started to write an “Expression” engine. There is an example that tries to classify different expressions that you train. Like any classification algorithm, this requires examples. In classification, there are two phases: training and testing. Training requires you to have a set of examples for every class (i.e. facial expression). Testing is then when you try and of the classes (i.e. facial expressions) and see if it correctly guesses the correct class (i.e. facial expression). There is a huge amount of literature on ways of doing classification. Pattern Recognition and Machine Learning by Christopher Bishop is a great book for learning these.

Kyle’s approach to classification of facial expressions is based on the nearest neighbor of a set of training examples. Given a set of classified examples (the training data), any new face can be classified by finding the nearest example (computed using the euclidean distance or L2-norm). Have a look at the “example-expression” app to see how this works in practice.

In practice, you will find that face detection algorithms are noisy, pick out a lot of false positives, and work better for lighter skin types than darker ones. As well, the more beardy/hairy you are, the more likely the algorithm will fail. Babies as well have very different facial profiles and will perhaps fail on any classifiers built for adults. Lighting, as with pretty much every camera-based computer vision algorithm is very important. Infrared lighting can help you achieve much more control of your environment. Keep this in mind if you are having poor detection results.

If your detection algorithm has found the eye, does this mean I can start to control the mouse pointer? No. There is a huge giant leap to go from detecting the eye in a camera image, to understanding where the eye is looking in reference to a monitor. For an open-source project of eye-tracking, have a look at these two projects: EyeWriter (cross-platform C/C++) and ITU GazeGroup’s GazeTracker (Windows only, written in C-Sharp). The latter is much more advanced though is written in C# (Windows only).

FaceShift is another technology that you should check out. It’s still in beta but you can get a 30-day license for free. They describe their work:

Our software analyzes the face motions of an actor, and describes them as a mixture of basic expressions, plus head orientation and gaze. This description is then used to animate virtual characters for use in movie or game prodcution. We have astonishing real time tracking and a high quality offline postprocessing in a single, convenient application.

Kyle McDonald has written a nice openFrameworks wrapper, ofxFaceShift, for interfacing with this library. He also wrote a binary similar to FaceOsc, FaceShiftOsc, which just outputs some essential information. Check out his video tutorial with associated links and guide here.

This technology requires the Kinect or similar depth sensing device. Note, if you are interested in facial capture, the Carmine 1.09 is a better device to use. Next week, we’ll look in more depth at depth sensors and the capabilities of using OpenNI with the Kinect.

Lab

Please make sure you have a group by now and an idea ready to discuss. Next week I will need 300 words from every group. This week, I will be coming to every lab to get a better sense of what you are planning to do and to give you feedback on your ideas. Please work towards your group projects.

Lecture

OpenNI
This is an open-source SDK for developing middleware libraries for Natural Interaction. Middleware acts as a bridge between hardware specific routines, and more general routines. This means that you may have a number of different hardware, such as the Asus Xtion or the Kinect XBox sensor, and you can use the same application so long as there is a middleware that talks to either device. The middleware has to be written with the general framework in mind, however. Also, the middleware itself may not necessarily be open-source.

Primesense, the makers of the sensor on board the Kinect XBox, have developed a middleware called NITE. This allows you to do skeleton tracking and hand tracking. This is perhaps the most popular middleware. There are others listed here which I have not tried, though which look promising.

Because the Kinect uses a Primesense sensor, you can use the NITE middleware to do skeletal tracking and hand tracking. However, this middleware expects you to have a driver which provides an interface to the hardware. Luckily the community has worked on doing all the USB sniffing etc… required for figuring out how to do this. Here are the sensor drivers for the Kinect. Before you download this, note if you are going to use ofxOpenNI, you won’t need to download this, or NITE, or OpenNI.

ofxOpenNI
Be sure to read the included README file for instructions on how to get it running. There are included compiled libraries for OSX. ofxOpenNI includes the SensorKinect, NITE, and OpenNI libraries required for working with the Kinect sensor. It also provides a simple to use openFrameworks library which communicates with the OpenNI library.

RGBDToolkit
Using a DSLR and a Kinect, you can create very vivid point clouds.

RGBDemo – Object/Scene Reconstruction
From the surface reconstruction or object model acquisition example, you can import the “ply” or “model” file into MeshLab. Here is a video by Kyle McDonald describing how you can transform these points into a set of faces describing a surface. Here is a video by Nicolas Burrus demo-ing the object acquisition process.

Going from point clouds (what the Kinect gives you, a bunch of 3D points, or a Depth Image) to a 3D model with “faces” (vertices describing triangles with associated normals) involves a processes called surface reconstruction. You could, for instance, model an arbitrary object in 3D, such as a car, face, or an entire room, and use this technique to build a digital 3D model of it. You could then import that model into a game engine for instance. Or maybe use our 3D printer and see what it looks like as a scale model.

Generally, there are a number of stages involved, all of which you can do in MeshLab.

Here are the parameters he uses in the video:

Render->Show Vertex Normals

Filters->Point Set -> Compute Normals
Neighbors 16
Flip Normals checked
-1000 Z

Filters->SamplingFilters-> Poisson-disk
Samples:5000
Base Mesh Sampling Checked

Filters->Point Set->Surface Reconstruction: Poisson
Octree: 12
Solver: 7

Filters->Remeshing->LS3, 3 Iterations (default)

Filters->Sampling->Vertex Attrib Transfer
Source: *.ply
Target: Poisson Mesh
Transfer Geometry
Transfer Normal

Filters->Cleaning->Remove Duplicate Vertex

These parameters are highly dependent on your data set. You will likely have to read more about these to understand how they effect the output in order to obtain better results. For surface reconstruction, also have a look at the ‘Marching Cubes’ and ‘Ball Pivoting’ algorithms instead of ‘Poisson’ to see how these compare.

Generally, you want to scan your object in very controlled light situations, with low IR as the Kinect is highly dependent on IR for its depth image construction. As well, you will want to move the Kinect at a very controlled speed and with similar rotation/translation around an object or scene. Ideally, nothing in the scene is changing, unless this is part of your process.

RGBDemo – 3D Projector/Camera Calibration

Also included in RGBDemo are routines for doing intrinsic and extrinsic calibration of a projector and camera. ‘Intrinsic’ parameters of a lens describe the transformation from camera/projector space to world space. In other words, it describes how the camera/projector’s lens warps what it sees. Think of a fish-eye lens image. This image is incredibly warped because of the shape of the lens. However, knowing the intrinsic parameters of this lens, you can un-warp the image back to what we’d see in the real-world (though we also have a warped perspective closer towards our periphery). What this means is any parallel lines in the world space (i.e. what we see), will also be parallel in the image space (i.e. what the camera sees).

Once you calculate the intrinsics of the camera, you can un-warp the image received by the camera device. This image will have all parallel lines mapped to parallel lines. You can also do this for a projector.

You can also do one more really cool thing: extrinsic calibration of a camera and projector. This means you can capture something with the camera, then project it back into the real world with the camera image aligned to the projector’s point of view. You would ideally do some processing on the image, or tracking, or augmented reality of this image before projecting it back into the world.

PTAM/PTAMM
PTAM and associated blob.

ofxPTAMM
ofxPTAMM. Follow the steps of the included README as there a few things to do before getting the example project running. It comes with compiled sources for OSX 10.6 and 10.7, and iOS 5.0. There are instructions for getting it up and running for Linux and Windows. For OSX 10.7, I also had to add the frameworks “QTKit” and “CoreVideo” to my project.