Massive dataset explored

Example face with holes remaining in the data

Another example

Same as above, different angle

Summary: Notes, tips, code samples, and pointers relating to FRGC (an ongoing series of posts)

HIS post provides some information of interest to those who may find themselves working with 70 GB of data and some programs [1, 2]. The latter is a FRGC Web site. The package comes with associated applications and scripts written in Java, C++, Perl, etc. The previous post about the dataset (FRGC ver2.0) offers a bit of background that is research-specific (relating to Dr. Ajmal Mian and his Ph.D. student Faisal R. Al-Osaimi), whereas the notes below are a bit more generic. This series of posts is not about statistical models of faces but only about the dataset. This recent message from Face Recognition Research Community contains MATLAB/Octave loader code for a data instance from the dataset, where each 3-D face weighs about 13 MB (compressed):

function [x, y, z, fl] = absload(fname) 
%ABSLOAD Read a UND database range image from file.
%   [X,Y,Z,FL] = ABSLOAD(FILENAME) reads the range image in FILENAME
%   into the variables X,Y,Z,FL.
%   FILENAME is a string that specifies the name of the file
%            to be openned
%   X,Y,Z are matrices representing the 3D co-ords of each point
%   FL    is the flags vector specifying if a point is valid

% open the file
fid = fopen(fname);

% read number of rows
r = fgetl(fid);
r = sscanf(r, '%d');

% read number of columns
c = fgetl(fid);
c = sscanf(c, '%d');

% read junk line
t = fgetl(fid); clear t;

% get flags
fl = fscanf(fid,'%d',[c r])';

% get x
x = fscanf(fid,'%f',[c r])';
% get y
y = fscanf(fid,'%f',[c r])';
% get z
z = fscanf(fid,'%f',[c r])';

% close the file
fclose(fid); 

This just handles one single image. There are many in the current collection:

find | grep .abs | wc
   4950    4950  257215

All of which are compressed:

find | grep .abs.gz | wc
   4950    4950  257215

To get a list of the 3-D faces:

find | grep .abs.gz | awk '/{print $1}' 1>~/files_list.txt

It yields something like the following:

(more…)

From Al-Osaimi et al., IJCV 2008

Summary: My attempt to reproduce some of the results of F. Al-Osaimi et al. and furthermore improve them using other methods and different datasets (with a 3-D scanner at our disposal)

HIS post provides some background about my next (current) research project, which deals with non-medical applications. The previous project dealt with cardiac imaging and I’ve packaged that code and published it along with other data that may be useful.

The group of A. Mian has done some fantastic work recently on 3-D face recognition and I shall attempt to reproduce some results with a NIST database. In their paper “An Expression Deformation Approach to Non-rigid 3D Face Recognition,” F. Al-Osaimi, M. Bennamoun, and A. Mian explain some good results from experiements that apply PCA to face images (paper published online in September 2008 by a leading computer vision journal, but access is restricted, so there is no link, either… unless one uses this copy).

Since I have extensive experience with NRR, PCA, and statistical models in general, this project suits me better than some previous ones. I have done limited work on analysis applied to sets of face images that are only rigidly or affinely registered.

The paper from the group in question is 22 pages long in the raw form and about 15 in IJCV. The abstract describes an idea and quantifies some results using known benchmarks and the “FRGC v2.0 dataset”. Then, the method is alluded to vaguely and not formalised until later. The phrasing could be improved somewhat to avoid repetition, e.g. in the following paragraph containing parts like: “2D face recognition has been extensively researched in the last two decades. However, unlike 3D face recognition its accuracy is adversely affected by many factors such as illumination and scale variations. In addition, 2D images undergoes affine transformations during acquisition. Moreover, handling pose variations in 3D scans is more feasible than 2D images. It is believed that 3D face recognition has the potential for more accuracy than 2D face recognition (Bowyer et al. 2006). On the other hand, the acquisition of 2D images is less intrusive than 3D acquisition. However, 3D acquisition technologies are continuously becoming cheaper and less intrusive (The International Conference on 3D Digital Imaging and Modeling, 1997–2007).”

“Most of the approaches in the literature are rigid,” says the text in page 2, just before the overview which states: “The main contribution of this paper is a non-rigid 3D face recognition approach. This approach robustly models the expression patterns of the human face and applies the model to morph out facial expressions from a 3D scan of a probe face before matching. Robust expression modeling and subsequent morphing gives our approach a better ability in differentiating between expression deformations and interpersonal disparities. Consequently, more interpersonal disparities are preserved for the matching stage leading to better recognition performance.”

The background section is followed by some classification of existing work, concluding with: “Our approach also falls into this category i.e. non-rigid 3D face recognition.”

1.1 presents a very good summary of related work and 1.2 a clear overview of the method and the ideas behind it, accompanied by a helpful diagram at the bottom of page 3 (Figure 1). The strategy is to use pairs of image of the same individuals, normalising them a bit, and then applying PCA to reduce the dimensionality that characterises expression variation.

Section 2 in page 4 starts by describing pre-processing steps that are essential yet specific to the limitation of the FRGC v.20 dataset. Page 5 starts presenting some visual examples of the approach, with some equations relating to PCA (along with more visual examples) in pages 6 and 7.

Section 3 begins to deal with some other experiments that are not just dealing with models in synthesis mode. The same dataset is being used (with about 5,000 3-D faces), but more data gets added to it. To quote, “The dataset is composed of two partitions: the training partition (943 scans) and the evaluation partition (4007 scans). [..] The FRGC dataset was augmented by 3006 scans that were acquired using a Minolta vivid scanner in our laboratory.”

Parameters and set sizes (those which are included) get tested in very large-scale experiments that yield ROC curves. These curves help show how to set the different parameters and enable one to measure advantages of one algorithm over another. Page 13 has some comparisons to other methods from the literature, with numbers summarised in a chart.

This is truly inspiring work and I shall spend the next few weeks learning from it as well as implementing something similar.

ODAY I AM releasing the final code from my current project. The following is the accompanying README file:

Homepage: http://schestowitz.com/Research/Cardiac/Cardiomat
Index of related documents: http://schestowitz.com/Research/Progress/2010-2011.htm

INTRODUCTION

Cardiomat is the name given to an Octave/MATLAB program whose intended use is heart tracking, with various different bits that help probe cardiac structure and perform basic measurements.

FILES

Contained in the package are the following M-files and sub-directories:

./Experiments
./Experiments/fiesta_simulation-old-experiments.m
./create_synthetic_set.m
./GUI
./GUI/cardiomat.png
./GUI/splash.m
./GUI/splash.fig
./GUI/get_parameters_from_user.m
./GUI/getoptionsgui.m
./GUI/cardiomat-title.jpg
./GUI/cardiomat.m
./GUI/cardiomat.jpg
./getdicomimages.m
./loadtaggedsequence.m
./README (this file)
./Imported
./Imported/findit
./Imported/findit/findit.m
./Imported/arrow
./Imported/arrow/arrow.m
./Imported/gradient
./Imported/gradient/gaussgradient.zip
./Imported/gradient/gaussgradient
./Imported/gradient/gaussgradient/gaussgradient.m
./Imported/gradient/gaussgradient/README.txt.txt
./Imported/gradient/gaussgradient/testgaussgradient.m
./Hybrid
./Hybrid/cardsim_frame_shortaxis.m
./Hybrid/shuffle_transform.m
./fiesta_simulation.m

In addition, the program depends on at least one function from biosig4octmat-2.50, whose homepage is located at http://biosig.sourceforge.net. That function helps find the centre and the radius of a best fit circle/sphere for a group of points.

The directory named “Hybrid” contains functions that were written jointly and “Imported” contains functions which were brought from the outside. The core functions in the package are GPLv3-licensed and the rest are most likely to be BSD-licensed (check the sources to verify). Some of the imported functions may work only in MATLAB and not in GNU Octave, which is a free/libre substitute for MATLAB that lacks some graphical features for the most part.

HOW TO RUN THE PROGRAM

The program was written to be compatible with both Octave and MATLAB. There are some features in it that may require a UNIX/Linux system (e.g. animated videos) and some that may require MATLAB. Additionally, in order to reduce dependency on MATLAB’s GUI utilities, the program can be run in GUI and CLI modes separately, with only the use of figures being the exception to this (figures are output rather than means of interaction).

Key functions reside in the main directory of the program (one function, getdicomimages.m, is deprecated) and a directory named “GUI” contains some of the newer parts that add a GUI.

An important interface function is called fiesta_simulation.m. The first line of code in this function is a boolean flag that says whether the program should be run in GUI mode or not. If use_gui is 0, then the program will simply run the list of tasks specified below. It is possible to give several jobs for the program to complete this way, e.g. in order to run many experiments overnight. If use_gui is non-zero (probably 1 but not necessarily), then the function is assumed to have been called from the file GUI/cardiomat.m, which is a wrapper that invokes a splash screen and some dialogues that serve as a graphical interface. In turn, parameters fed to these dialogues will be passed to fiesta_simulation.m. It might be necessary to modify some paths inside the files in case the functions and the files are located in different places than the ones specified in parameters. Such paths are named in fiesta_simulation.m, loadtaggedsequence.m (specifies output directories onto which to save figures), and splash.m (looks for program images). For a new user, the only things require modifications are likely to be directory paths and maybe the use_gui boolean, depending on this user’s preference.

To run the program in GUI mode, MATLAB is needed. There is also a parameter to specify in fiesta_simulation.m in order to inform loadtaggedsequence.m whether MATLAB or Octave gets used. It is the boolean which is also the last argument in the function call, being 0 for MATLAB and 1 for Octave. The GUI front end will always pass 0 under the assumption that it is run under MATLAB.

DATASETS

The specified directories include a path from which to open images, optionally with a wildcard to filter bits contained within a given directory, e.g. selecting a subset of images that are a range of numbers. Due to privacy/data anonymity constraints, no raw data accompanies this program, even though it can definitely be obtained from many places (see for example https://schestowitz.com/Weblog/archives/2010/11/14/segmented-and-tagged-datasets/).

COMPATIBILITY

There are separate levels of compatibility dilemmas. One is the program level and another is the operating system level. Although the program has been tested in both MATLAB and Octave, on both GNU/Linux and Windows, there may be trivial code changes to make in case errors show up. Most will be path related, so it is abundantly clear that emphasis should be put on making all functions reachable from within paths definition (e.g. “addpath(genpath(YOUR_PROGRAM_ROOT_DIR_HERE))”) and booleans should be set to do the right thing (GUI/CLI and MATLAB/Octave). The goal is to ensure the program can be run under as many platforms as possible, with either free/libre or proprietary software.

IN CASE OF DIFFICULTIES

If issues running the program are experienced, please contact the author (address above). Bugs are expected to exist (it is a research purposes program, so polish is not a priority) and input is not validated or sanitised to provide virtual ‘rails’ on which the user operates the program (sane values need to be given). Sample function calls are given in fiesta_simulation.m and /Experiments/fiesta_simulation-old-experiments.m. Some of them may use the old interface and thus lack some input arguments, but most of them show the type of experiments which were run using the program.

By sacrificing the comfort of the command-line interface (CLI)

ver the past few days I’ve been repackaging one of my programs such that it becomes easier to install and more user-friendly once installed. The program works in both MATLAB (proprietary software) and Octave (Free/libre software) and it now has a GUI and a name, which I arbitrarily and hastily named “Cardiomat”. Directory selection dialogues and some other user interface features are missing from Octave, so the code has become filled with some conditional statements so that it serves each user, no matter the choice of interpreters (proprietary or Free).

There are mixed feelings whenever time is dedicated to making a GUI. When a program is scriptable in the command-line sense, then it is a lot more flexible to the user and piping I/O is a lot simpler. To a new user, however, this can prove baffling. The GUI and splash screen may be necessary for medical doctors (and radiologists) to use, but it removes a lot of the power of the program, which can really end up incorporating both. When I created a 3-D othello game I made a command-line option available too as this enabled me to run simulations (e.g. game engines playing against one another) overnight, saving the results to files. With a GUI-only approach this is not possible. All my other scientific programs also had CLI interfaces which overrode or replaced the GUI. The conclusion of this whole matter is that no matter how much people like to vilify CLI ‘hackers’, those who embrace the CLI are usually in a position of advantage, especially where research is concerned. The Windows mindset requires people to repeatedly run tasks from GUIs, whereas the use of the CLI can automate all of this and provide more flexibility (I use vi as the editor for programming, still). The GUI is an area where MATLAB and Octave diverge somewhat, so here too there is a problem of universaility breaking. When one strictly sticks to the command line, there are universal interfaces and more a unified language, which tend to break if abstraction is taken that involves graphical interfaces (may also be platform dependent).

Anyway, the program can now handle anything including .bmp files and DICOM, with ginput handling image coordinate extraction (albeit bugs in Octave prevent it from working properly) and with a CLI option/fallback the program still performs the tasks it needs to. I will upload all the final source code shortly (I am the only developer, so there has been no repository yet). In future posts I am going to write about my next project and my next program, which will deal with 3-D face recognition (a departure from my current work on medical, intra-body imaging).

I’ve not been writing about my programming work for quite a few years (my detractors love to dig and smear me for my coding or my job), but I may try to carry on blogging about it. Here are some new screenshots:

HE previous post about validation with rotation put focus on large regions outside the heart, where there was greater separation between landmark points. In this case, initially I tested a general case where 120 points were put around a circular area in and outside the heart, showing more or less what got detected as the expected translation at each frame. I look at only one point at a time here, on purpose — meaning that for each point only its neighbours in the next frame are being compared and there is room for improvement if the whole spacial patch (e.g. 3×3 window) gets used in the current frame. This simplification was intentional here.

(more…)

S a first validation step I am running these basic sanity checks which track anatomy where the correct correspondences are known. These animations show how a collection of points spread around the boundary of a circle (but not put precisely on the circle, on purpose) move along with the image merely by tracking the signal, without any knowledge of what other points are doing or what rigid/affine transformation the image itself is subjected to. Moreover, the interpolation process causes little issue at all and even though points can only move up/down/left/right (and diagonally), they do manage to move consistently.

With a new Gaussian kernel-based filter I’ve put in place fusion that makes use of image gradient (x+y derivatives) possible alongside with image signal. The binary images resulting from edge detection might be of some use, but scaling is tricky.

ORKING with bare metal for the purpose of maximal hardware utilisation and performance is not the same as working with typical computer programs. Developing software for research is also different from software development for end users, where the software is treated as a product. Different scenarios require different methodologies and different levels of polish, and thus have different specifications and priorities. In research, brute force becomes essential to the success of one group or another.

I often find myself working on servers or clusters (only GNU/Linux), in which case it is useful to manage up to 50 computers, for example, remotely, at the same time. With full GUI sessions it can be a disorienting task, but that too can be achieved and I wrote about it roughly 6 years ago in this blog (back when I was a Ph.D. student).

Working from the command line at a desktop is perfectly acceptable if brains and not candy count, so platforms like Mac OS X are irrelevant. To do coding on the server I currently use vi as the editor in one terminal and the MATLAB session in another. On the desktop I would use QtOctave, as covered before (also in some screencasts). Unfortunate issues that had me revisit MATLAB (which in any case would be inevitable as means of ensuring compatibility with colleagues) led me back to the old days of developing over SSH (using KIO slaves and the likes of that). Quick access to the file can be gained using abbreviated wrappers such as:

./Main/Programs/Scripts/SSH/42
/opt/matlab/bin/matlab -nodesktop -nosplash -r "addpath(genpath('/home/S00/schestr0/'))"

where 42 is a script numbered after the machine number and the latter command initialises the command line. /home/S00/schestr0/ is using NFS and it can be reached from any computer at the department at the same time (multiple machines can write to the filesystem at the same time, too). If the files are edited locally, they can be passed across to the server using a simple command that transfers them in encrypted form.

scp /home/roy/Main/IT/Programs/get_dicom_heart/* -r
schestr0@eng041.cs.man.ac.uk:/home/S00/schestr0/get_dicom_heart/

Occasional local backups (usually nightly because it is resources-intensive over USB2) are achieved as follows:

mkdir /media/disk/Home/`date +%Y-%m-%d`

tar -cf - /home/roy|split -b 1000m - /media/disk/Home/`date +%Y-%m-%d`/Home-`date +%Y-%m-%d`.tar.

To decipher this, try:

man tar
man split

For example in:

tar -cf - /media/SEA_DISK/Home/|split -b 1000m - Baine-`date +%Y-%m-%d`

the backup is split into chunks of 1 gigabyte and the files are named by the date

To reassemble the above (resorting from backups):

cat *|tar -xf - 

No command line interface should be intimidating. Many powerful computer tasks are managed from it because the command line makes streamlining simpler and scripting jobs is a lot easier. My programs are written in a way that enables assigning many parameters like paths and wildcards, which lets the programs run for many hours and automatically produce output like images, videos, and text for inspection later. Research in computer vision or computer graphics requires heavy computation and for compelling experiments, the larger the sample sets, the more convincing the results. Use of computer resources therefore becomes the difference between failure and success and those who fail to master it can easily perish against the competition. Another valuable skill is knowing how to reuse code (legally of course) and this is where free/libre software orientation helps a lot. Publication in Open Access (OA) and availability of one’s own work — including code — can help gain more citations, which is the currency by which many publications are being judged.