It is useful to compare the performance of different measures of
model quality. For a given measure, the level of model
degradation that can be detected in the validation experiments
described above depends on both the change in the value of the
measure as a function of model degradation and the uncertainty in
the value. To quantify this, we define the sensitivity
of a measure as follows.
where is the value of the measure for some degree
of degradation
,
is the mean error in the
estimate of
over the range.
is reciprocal of the change in
required for
to change by one noise standard error, which
indicates the lower limit of change in quality
which can be
detected by the measure. Sensitivities for the specificity and
generalisation for different values of shuffle radius are shown in
Figure 8. These results demonstrate
that specificity is a more sensitive measure of model quality than
generalisation, and that the use of shuffle distance improves the
sensitivities of both measures over those obtained using Euclidean
distance.