PDF version of this document

MODEL-BASED ASSESSMENT

Model-based NRR assessment is closely correlated (if not analogous) to the evaluation of appearance models. In order to carry out such evaluation, many synthetic images are drawn from the model. A finite number of such synthetic images get generated and then compared against the training set of the same model. In that respect, data which is representative of the model is 'locked onto' that model's seminal data, i.e. the training dataset. Since the number of synthetic images is finite, there remains some susceptibility to an error. The estimate of the model-data match suffers if the number of instantiations, upon which the evaluation is based, remains overly low.

Distances are calculated between pairs of images, namely images in the training set and synthetic images derived from the model. These distances are computed in accordance with the shuffle distance principles, whereby each pixel is compared against a corresponding neighbourhood of pixels in another image. Since there are never enough such distances to attain a stable measure, errors should be associated with the size of the set under consideration. Having got a collection of distances for a large number of possible image pairings, the standard deviation must be computed. Subsequently, the standard error can be computed as well.

Below lies a figure where the registration quality (horizontal axis) affects the measured quality as perceived by the model-based assessment method.

Inter-set error is somewhat of a simpler case. Since we repeat our experiments 10 times and obtain different values each time, there is a certain error associated with the measure. That error is calculated simply by taking the standard error over corresponding values across the instantiations. The errors are bound to be rather large if the measured quantity varies greatly among datasets or if the number of instantiations is fairly low.