I’ve updated C.A.R.P.E., a graphical tool for visualizing eye-movements and processing audio/video, to include a graphical timeline (thanks to ofxTimeline by James George/YCAM), support for audio playback/scrubbing (using pkmAudioWaveform), audio saving, and various bug fixes. This release has changed some parameters of the XML file and added others. Please refer to this example XML file for how to setup your own data:

See my previous post for information on the initial release.

Please fill out the following form if you’d like to use C.A.R.P.E..:

… Continue reading...

The post C.A.R.P.E. version 0.1.1 release first appeared on http://archive.pkmital.com.

Original video still without eye-movements and heatmap overlay copyright Dropping Knowledge Video Republic.

From 2008 – 2010, I worked on the Dynamic Images and Eye-Movements (D.I.E.M.) project, led by John Henderson, with Tim Smith and Robin Hill. We worked together to collect nearly 200 participants eye-movements on nearly 100 short films from 30 seconds to 5 minutes in length. The database is freely available and covers a wide range of film styles form advertisements, to movie and music trailers, to news clips. During my time on the project, I developed an open source toolkit, C.A.R.P.E. to complement D.I.E.M., or Computational Algorithmic Representation and Processing of Eye-movements (Tim’s idea!), for visualizing and processing the data we collected, and used it for writing up a journal paper describing a strong correlation between tightly clustered eye-movements and the motion in a scene. We also output visualizations of our entire corpus on our Vimeo channel. The project came to a halt and so did the visualization software. I’ve since picked up the ball and re-written it entirely from the ground up.

The image below shows how you can represent the movie, the motion in the scene of the movie (represented in … Continue reading...

The post Toolkit for Visualizing Eye-Movements and Processing Audio/Video first appeared on http://archive.pkmital.com.

Introduction

My work in audiovisual resynthesis aims to create models of how humans represent and attend to audiovisual scenes. Using pattern recognition of both audio and visual material, these models use large corpora of learned audiovisual material which can be matched to ongoing streams of incoming audio or visual material. The way audio and visual material is stored and segmented within the model is based heavily on neurobiology and behavioral evidence (the details are saved for another post). I have called the underlying model Audiovisual Content-based Information Description/Distortion (or ACID for short).

As an example, a live stream of audio may be matched to a database of learned sounds from recordings of nature, creating a re-synthesis of the audio environment at present using only pre-recorded material from nature itself. These learned sounds may be fragments of a bird chirping, or the sound of footsteps. Incoming sounds of someone talking may then be synthesized using the closest sounding material to that person talking, perhaps a bird chirp or a footstep. Instead of a live stream, one can also re-synthesize a pre-recorded stream. Consider using a database … Continue reading...

The post Intention in Copyright first appeared on http://archive.pkmital.com.

Working closely with my adviser Mick Grierson, I have developed a way to resynthesize existing videos using material from another set of videos. This process starts by learning a database of objects that appear in the set of videos to synthesize from. The target video to resynthesize is then broken into objects in a similar manner, but also matched to objects in the database. What you get is a resynthesis of the video that appears as beautiful disorder. Here are two examples, the first using Family Guy to resynthesize The Simpsons. And the second using Jan Svankmajer’s Food to resynthesize Jan Svankmajer’s Dimensions of Dialogue.

… Continue reading...

The post Concatenative Video Synthesis (or Video Mosaicing) first appeared on http://archive.pkmital.com.

Not to mention, our tools and visualizations are finally reaching an audience with interests in film, photography, and cognition. If you haven’t yet seen some of our videos, please head on over to our vimeo page, where you can see a range of videos embedded with eye-tracking of participants and many different visualizations of models of eye-movements using machine learning, or start by reading Tim’s post on … Continue reading...

The post Tim J Smith guest blogs for David Bordwell first appeared on http://archive.pkmital.com.

6DOF Head Tracking from pkmital on Vimeo.

interfacing seeingmachines faceapi with openFrameworks to control a 3D mario avatar

This is just with the non-commercial license. The full commercial license (~$3000?) gives you access to lip/mouth tracking and eye-brows, as well as much more flexibility in how to use their api with different/multiple cameras and accessing image data.

Of course, there are other initiatives at producing similar results. Mutual information based template trackers, for instance, seem to be state-of-art. Take a look at recent work by Panin and Knoll using OpenTL:

I imagine a lot of people would like this technology.… Continue reading...

The post 6DOF Head Tracking first appeared on http://archive.pkmital.com.

This uses the Facebook Graph API to get information about Facebook users in a very accessible manner.  Of course, it is only your “public” information that is accessible without authorization.  But once you “allow” an application to access your information, you’re allowing access to EVERYTHING.

Generally, these items are publicly known:

{
   "id": "0123456789",
   "name": "Parag K Mital",
   "first_name": "Parag",
   "middle_name": "K",
   "last_name": "Mital",
   "locale": "en_US"
}

and also your Profile picture.

Check out a montage of the first 3600 Facebook user’s profile pictures, obtained just by using the public url: http://graph.facebook.com/USER_ID/picture

And an image of the average of all 3600 profile images:

… Continue reading...

The post Facebook Graph API first appeared on http://archive.pkmital.com.

We are in the process of making all of our data freely available for research use. As well, we have also worked on tools for analyzing eye-movements during such dynamic scenes.

CARPE, or more bombastically known as Computational Algorithmic Representation and Processing of Eye-movements, allows one to begin visualizing eye-movement data together with the video data it was tracked with in a number of ways. It currently supports low-level feature visualizations, clustering of eye-movements, model selection, heat-map visualizations, blending, contour visualizations, peek-through visualizations, movie output, binocular data input, and more. The videos shown above on our Vimeo page were all created using this tool. Head over to Google code to check out the source code or download the binary. We are still in the process of stream-lining this process by creating manuals for new users and uploading more of the eye-tracking and video data so … Continue reading...

The post Dynamic Scene Perception Eye-Movement Data Videos and Analysis first appeared on http://archive.pkmital.com.

The post DIEM Website first appeared on http://archive.pkmital.com.

It was an interesting group of people that strangely somehow all had much in common. I had the fortune of presenting my research as it relates to DIEM in the place of Tim J. Smith.

The schedule:

Close-Up 2: Schedule for Wednesday 17^th June 2009

10am coffee and tech checks (G.11, William Robertson Building)

10.30 Welcome, Annette Davison (Music, ACE and Director, Cfpma), Martine Beugnet (LLC, Convener of Film Studies)

Who is who, where is what? People and resources for which the Cfpma will provide a point of convergence.

11am Individual Presentations (MAX 10 mins each):

Andrew Lawrence (African Studies, SSPS) — Difficult satire under austerity: the films of Sissako and Amoussou

Martine Beugnet (Film, LLC) — “The Wounded Screen”

Richard Williams (Architecture, ACE) — “The Modernist City on Film”

Stephen Cairns (Architecture, ACE) — “Cultures of Legibility: Emergent Urban Landscapes in Southeast Asia”

Simon Frith and Annette Davison (Music, ACE) — “The Role of Cinemas in the History of Live Music”

Mary Fogarty (Music, ACE) — “The … Continue reading...

The post CLOSE-UP 2 first appeared on http://archive.pkmital.com.