PDF version of this entire document
In my work which this thesis presents, values returned for various measures are dependent upon the size of the sets, the dimensionality and a few other free parameters. This makes it difficult to argue about and distinguish between results from different experiments, unless all conditions (free parameters) remained identical. For example, an experiment performed with large images and small sets cannot trivially be compared against other experiments involving small images and very large sets. In order for all results to be numerically comparable, there needs to be a normalisation stage, which accounts for the many free parameters simultaneously. My preliminary results came from experiments that suffered from complexity in the number of parameters and decisions on how to scale various measures to get a reasonable single number.
The problem can be corrected by calculating the self-normalising pseudo-entropy of graphs []. This graph represents the distances between images (edges) where vertices are individual images. Entropic graphs is an area that I explored in great depth, yet it turned out to be rather complex due to the need to estimate many parameters using a Monte-Carlo simulation. Despite the fact that my efforts to adopt this method have been curtailed, there is place to propose another paradigm for dealing with this issue. The ad hoc nature of Specificity and Generalisation leaves a lot of room for new measures to evolve. Whether an alternative method would be equally cheap to compute remains an unknown. As in many such large-scale problems, simplicity has its merits, too.
Twining and Taylor [] extended my work and extended previous work on shape. They addressed the problem above shortly after my experimental work for this thesis had been completed. In their paper on Kullback-Leibler (KL) divergence, a principled theoretic method is presented. They show the theory to be true, but do not show just how good an estimate it is for a finite-sized datasets and sample sets. Rather than generate large datasets and use brute force, they make use of an analytical approach where they directly evaluate the level of agreement between training images and the model pdf corresponding to these images. The measures of Generalisation and Specificity, as defined before, are replaced by similar measures that estimate cross-entropy. A newly-defined measure, generalised specificity, was tested and shown to be a satisfactory estimate even when dealing with small sets of training images (or any arbitrary data for that matter). Since it does not require synthesis and is derived directly from the set, no heuristics need to be used. Twining and Taylor show that in the limit, one gets out KL divergence, whereas my own experimental investigation looked at finite-sized datasets and sample sets.