Progress Report
September 13th, 2005
Overview
- Second Year Report
- Model-based evaluation progress
- error bars for instantiation
- averaging over instantiations
- refinement of sample points
- Prospective thesis structure
progress Report - Second Year
- 2nd Year Progress Report (PDF, HTML)
- overview on some relatively recent progress
- to a lesser extent, decisions on future directions
- outlining and hierarchically composing overall progress
- To be discussed later in this session (report)
Monotonic Response in Evaluation
- Hours spent centralising output files to obtain evaluation averages
- Results were quite pleasing
- Averaged over 5 instantiations (out of 10 to be completed)
- Currently processing instantiation 9 (at the time of writing of this report)
- 6th and 7th sample point (from the left) will eventually be farther apart
- Estimation of mean pixel displacement was rough and it will be refined later (see later figures in this report)
- Can potentially make use of up to 16 sample points
Evaluation Generalisability Averages
Generalisation ability of mis-registered data, averaged over 5 instantiations
Evaluation Specificity Averages
Specificity of mis-registered data, averaged over 5 instantiation
Refining Sample Point Estimation
- 100 instantiations used to infer mean pixel displacement
- Improving sampling of the curve with just 8 points
- New sample points:
[0,0.0643,0.249,0.685,1.36,2.21,2.76,4.15]
- Possibility of alternative plots (e.g. bending energy)
- Also, possibility of extra points insertion
Perturbation-Related Collected Information
- Also recorded in the process of perturbation:
- standard deviation of average displacement
- average of maximum displacement
- average of minimum displacement
- average of mean bending energy
- standard deviation of mean bending energy
Uncertainty in Sample Points
- The questionable idea of horizontal error bars
- Account for uncertainty over the mean pixel displacement
- Might be implicitly included in the existing (vertical) error bars
- Horizontal error bars have been left aside and out of the plots
- Exclusion of such errors avoids possible confusion and/or clutter
- Visual example follows - both dimensions (horizonal and vertical) should wind up comfortably isolated
Sources of Error in Perturbation and Evaluation
The three possible sources of error to consider
Errors and Curve Sampling
- We need to calculate the mean pixel displacements at each sample point more accurately
- As for errors due to instantiation, we wish to find out and aggregate the uncertainty resulting from perturbation itself (the uncertainty involved in the instantiation process)
- In figures that are yet to be shown:
- accuracy of the plots has been improved
- some information has been added
Errors and Curve Sampling - Ctd.
- Repeated perturbations run to re-calculate mean pixel displacement (as described earlier)
- Subsequently, graphs have been re-plotted show sample points which lie more accurately upon the X axis
- The sample points are better isolated than before
- The errors associated with mean pixel displacement have shrunk
Errors and Curve Sampling - Ctd.
- In a consistent maroon shade: the newly-added errors
- These errors have to be accounted for due to the finite number of instantiations, namely 5 in this case
- Note the strange behaviour of the error bars at shuffle distance 3x3 (radius 1.5 pixels)
Generalisation - Refined Plots with Errors
Generalisation - Error bars due to 2 separate factors are now shown
Specificity - Refined Plots with Errors
Specificity - Error bars due to 2 separate factors are now shown
Progress on Overlap-based Evaluation
- All perturbed data sent to Bill
- 9 additional instantiations (unperturbed analysed already)
- Around 3 GB of Analyse-formatted perturbed brains
- Results to be obtained soon, maybe reaching completion later this week
Summary
- Model-based evaluation is nearing its end
- Error bars need to be modified as agreed
- (Label) Overlap-based evaluation is nearing its end
- Further thought needed on summarisation of work