PDF version of this entire document

Measuring Model Quality

A good model of a set of training data should possess several properties. Firstly, the model should be able to extrapolate and interpolate effectively from the training data, to produce a range of images from the same general class as those seen in the training set. We refer to this property as generalisation ability. Conversely, the model should not produce images which cannot be considered as valid examples of the class of image modelled. That is, a model built from brain images should only generate images which could be considered as valid images of the brain. We call this the specificity of the model. In previous work, quantitative measures of specificity and generalisation were used to evaluate shape models []. The extension of these ideas to images (as opposed to shapes) is presented here. Figure  provides an overview of the approach. It shows how clouds of points can be used to derive Specificity and Generalisation (as also shown in Figure  and explained later in this chapter) in a Monte-Carlo fashion. Twining and Taylor recently showed that, by using Kullback-Leibler (KL) divergence, one can extract the same type of measures based on a sound theoretical basis []. This will be further explained in Chapter 9. The KL divergence is essentially the relative entropy which measures the difference between two probability distributions, which in this case are the training images and the model pdf corresponding to these images.

Figure: set (points) and model pdf (shading) in image space. Left: A model which is specific, but not general. Right: A model which is general, but not specific.

Figure: model evaluation framework: A model is constructed from the training set and used to generate synthetic images. The training set and the set generated by the model can be viewed as clouds of points in image space ( $\mathbf{I}_{i}$ represented by stars, and $\mathbf{I}_{\mu}$ represented by dots).