eye movements - http://archive.pkmital.com

Toolkit for Visualizing Eye-Movements and Processing Audio/Video

Parag K Mital — Fri, 06 Feb 2015 23:53:18 +0000

Original video still without eye-movements and heatmap overlay copyright Dropping Knowledge Video Republic.

From 2008 – 2010, I worked on the Dynamic Images and Eye-Movements (D.I.E.M.) project, led by John Henderson, with Tim Smith and Robin Hill. We worked together to collect nearly 200 participants eye-movements on nearly 100 short films from 30 seconds to 5 minutes in length. The database is freely available and covers a wide range of film styles form advertisements, to movie and music trailers, to news clips. During my time on the project, I developed an open source toolkit, C.A.R.P.E. to complement D.I.E.M., or Computational Algorithmic Representation and Processing of Eye-movements (Tim’s idea!), for visualizing and processing the data we collected, and used it for writing up a journal paper describing a strong correlation between tightly clustered eye-movements and the motion in a scene. We also output visualizations of our entire corpus on our Vimeo channel. The project came to a halt and so did the visualization software. I’ve since picked up the ball and re-written it entirely from the ground up.

The image below shows how you can represent the movie, the motion in the scene of the movie (represented in … Continue reading...

The post Toolkit for Visualizing Eye-Movements and Processing Audio/Video first appeared on http://archive.pkmital.com.

Facial Appearance Modeling/Tracking

Parag K Mital — Thu, 26 May 2011 22:19:10 +0000

I’ve been working on developing a method for automatic head-pose tracking, and along the way have come to model facial appearances. I start by initializing a facial bounding box using the Viola-Jones detector, a well known and robust detector used for training objects. This allows me to centralize the face. Once I know where the 2D plane of the face is in an image, I can register an Active Shape Model like so:

After multiple views of the possible appearance variations of my face, including slight rotations, I construct an appearance model.

The idea I am working with is using the first components of variations of this appearance model for determining pose. Here I show the first two basis vectors and the images they reconstruct:

As you may notice, these two basis vectors very neatly encode rotation. By looking at the eigenvalues of the model, you can also interpret pose.… Continue reading...

The post Facial Appearance Modeling/Tracking first appeared on http://archive.pkmital.com.

6DOF Head Tracking

Parag K Mital — Thu, 18 Nov 2010 01:41:26 +0000

The following demo works with SeeingMachines FaceAPI in openFrameworks controlling a Mario avatar. It also has some really poor gesture recognition (and learning but it’s not shown here), though a threshold on the rotation DOF would have produced better results for the simple task of looking up/down left/right gestures.

6DOF Head Tracking from pkmital on Vimeo.

interfacing seeingmachines faceapi with openFrameworks to control a 3D mario avatar

This is just with the non-commercial license. The full commercial license (~$3000?) gives you access to lip/mouth tracking and eye-brows, as well as much more flexibility in how to use their api with different/multiple cameras and accessing image data.

Of course, there are other initiatives at producing similar results. Mutual information based template trackers, for instance, seem to be state-of-art. Take a look at recent work by Panin and Knoll using OpenTL:

I imagine a lot of people would like this technology.… Continue reading...

The post 6DOF Head Tracking first appeared on http://archive.pkmital.com.

DIEM Website

Parag K Mital — Wed, 16 Sep 2009 16:26:00 +0000

The DIEM Project (Dynamic Images and Eye Movements) has a sleek new website which you can check out here: http://www.psy.ed.ac.uk/diem… Continue reading...

The post DIEM Website first appeared on http://archive.pkmital.com.

CLOSE-UP 2

Parag K Mital — Thu, 18 Jun 2009 01:49:00 +0000

An event organized by the new Center for Film, Performance, and Media Arts (CFPMA) was held today in Edinburgh University discussing recent topics in… film, performance, and media arts.

It was an interesting group of people that strangely somehow all had much in common. I had the fortune of presenting my research as it relates to DIEM in the place of Tim J. Smith.

The schedule:

Close-Up 2: Schedule for Wednesday 17^th June 2009

10am coffee and tech checks (G.11, William Robertson Building)

10.30 Welcome, Annette Davison (Music, ACE and Director, Cfpma), Martine Beugnet (LLC, Convener of Film Studies)

Who is who, where is what? People and resources for which the Cfpma will provide a point of convergence.

11am Individual Presentations (MAX 10 mins each):

Andrew Lawrence (African Studies, SSPS) — Difficult satire under austerity: the films of Sissako and Amoussou

Martine Beugnet (Film, LLC) — “The Wounded Screen”

Richard Williams (Architecture, ACE) — “The Modernist City on Film”

Stephen Cairns (Architecture, ACE) — “Cultures of Legibility: Emergent Urban Landscapes in Southeast Asia”

Simon Frith and Annette Davison (Music, ACE) — “The Role of Cinemas in the History of Live Music”

Mary Fogarty (Music, ACE) — “The … Continue reading...

The post CLOSE-UP 2 first appeared on http://archive.pkmital.com.

Eye movements during video

Parag K Mital — Sun, 01 Mar 2009 16:34:00 +0000

Part of my research entails investigating ways of visualizing eye-movement data during dynamic images.

Some vids:

And one of President Obama’s inaugural speech:

from VisCogEdinburgh… Continue reading...

The post Eye movements during video first appeared on http://archive.pkmital.com.