Progress Report

September 22^nd, 2005

Overview

• Overlap-based evaluation results
• Model-based evaluation results
• Model-based and overlap-based comparison revisited
• Sensitivity results
• ISBI paper submission planning
• Thesis structure

Mean Pixel Displacement and Sample Points

• Values of the mean pixel displacement are estimated
• The values were not measures based on the actual set investigated
• Sample points in the curves are based on the average over 370 (10x37) random perturbations
• We used 100 random perturbations to estimate that average
• This may approximate the true values rather well
• The estimation can be based on more perturbations, if needed

Evaluation - Progress Overview

• Processing of 10 instantiations of perturbed brain sets completed
• Finally have some corresponding plots
• The plots indicate that we get a monotonic response
• As mis-registration increases, so do Specificity and Generalisation; Overlap likewise
• Recalculation of the errors
• Capability of producing numerous sensitivity plots

Visual Results - Plots of Interest

• The next few slides include fundamentally the same type of plot
• Different criteria are used for evaluation of image sets
• Overlap-based measures: 8 of these calculations
  • Tanimoto: 4 weightings
  • Dice: 4 weightings
• Model-based measures: 4 distinct shuffle distance evaluations
  • Shuffle neighbourhood of 1 pixel(Euclidean)
  • Shuffle neighbourhoods of 1.5, 2.1 and 3.7 pixels (consistent with previous experiments)

Model-based Evaluation: Specificity Results

Specificity as function of mis-registration, based on 10 instantiations

Model-based Evaluation: Generalisation Results

Generalisation as function of mis-registration, based on 10 instantiations

Tanimoto Plot

The Tanimoto overlap measure versus the mean of pixel displacement, i.e. degree of mis-registration

Dice Plot

The Dice overlap measure versus the mean of pixel displacement, i.e. degree of mis-registration

Previous Comparison - Model-Based Evaluation

Tanimoto complexity (w=3) versus shuffle distance with radius 3.7 pixels

Taking a More Comprehensive Overview

• 8 separate curves respresenting overlap versus mean pixel displacement (see next slide)
• The next stage should involve producing sensitivity plots for these 8 curves
• Another 4 curves from model-based evaluation
• We seek a method that is best at distinguishing reliably, telling apart registrations with fine-level differences

Label-based Evaluation Overview

A variety of related overlap calculations. These are compared by applying them to brain sets with increasing mis-registration

Label-based Evaluation - Observations

• Results from different overlap formulations are similar
• Curves are rather linear
• Tanimoto measure appears to be more sensitive
• Error bars might be overly optimistic

Label-based Evaluation - Next Steps

• Account for inter-instantiation uncertainty (error)
• Still need to reason about measures based on sensitivity
• Produce sensitivity plots, which can be derived from the data plotted earlier
• Compare best overlap-based measure with best of model-based measure

Combining Errors in Evaluation

• It appeared as though errors are satisfactorily small
• Needed to work out how to calculate the error bars for a sensitivity plot
• Error 'sources':
  • Instantiation (finite-sized)
  • Evaluation with finite-sized set
• One can account for errors pessimistically, simply summing them up
• Summation (a most-straight-forward aggregation) assumed from here onwards

Generalisation Sensitivity

Shown to the right: sensitivity of the Generalisability with or without accounting for the variability inherent in the instantiation process

Specificity Sensitivity

Shown to the right: sensitivity of the Specificity with or without accounting for the variability inherent in the instantiation process

Thesis Structure and Planning

• A 'slimmer' overview put together
• Put aside from the much more complex document
• Expansion of points, especially in the earlier sections
• Beginning to estimate length in terms of pages
• Thesis Structure Plan (PDF)

ISBI Submission Plan

• Basic template is in place
• More work on data presentation is yet required, but necessary files are at hand
• Rough outline included, which estimates section length
• Aim is to plan the 'shape' of the paper carefully
• Two-column style, 4 pages maximum
• ISBI Structure Plan (PDF)

Next Steps

• Incorporate error bars that account for inter-instantiation variability
• Plot sensitivity for overlap-based measures
• Circulate and agree on the structure of an ISBI submission
• Revise and expand prospective structure of thesis
• Absence from the 28^Mth until the 8^th (10 days)

Conferences - Links, Locations, and Deadlines

• ISBI 2006, Virginia, USA, 1 November 2005
• CVPR 2006, New York, USA, 7 November 2005
• ICPR 2006, Hong Kong, 15 December 2005
• MICCAI 2006, Denmark, 10 March 2006