PDF version of this entire document

Notes on New Method for Calculating
NRR Assessor Sensitivity

We are dealing with a curve where a measure $m$ is calculated for increasing magnitudes of perturbation/misregistration, $d$. At each level of misregistration $d_{i}$ we have one particular value of $m$, with corresponding standard error, $\sigma_{i}.$

As well as this standard error which is related to the number of samples we use to measure $m$ (the uncertainty in the measure), we have another uncertainty, due to repetition, or separate ``trials''.

i $\leftarrow$ $m_{i}$, i.e. the index $i$ is related to the measures whereas, on the other hand, $j$ indexes the trials. $m_{ij}$ is the measure at one particular point for a particular trial and $S$ (or $D$ in prior papers), the sensitivity of the measure, is what we seek to identify.

To measure the mean of $m$, we use the summation thus

$\bar{m}_{i}=\begin{array}{c} N\\ \sum\\ j\end{array}\frac{m_{ij}}{N}$.

And the errors are summarised in a messy fashion in the sets of equations below.

$\sigma_{\bar{m}_{i}}=\frac{\sigma_{m_{ij}}}{\sqrt{N}}$

$\sigma_{\sigma_{\bar{m}_{i}}}$= $\frac{\sigma_{m_{ij}}}{\sqrt{2N}}$

$\sigma_{\bar{m_{ij}}}\pm\sigma_{\sigma_{\bar{m}_{ij}}}$

$\sqrt{\begin{array}{c} n\\ \sum\\ j\end{array}\sigma_{\sigma_{\bar{m}_{i}}}^{2}}$

$S+\sigma_{s}$

< $\bar{\sigma}_{\bar{m}_{i}}$> < $\bar{\sigma}_{\sigma_{\bar{m}_{i}}}$>

$N$ is the number of repeated experiments, among the instantiations that we have.

We fit a function to the measures curve (e.g. Specificity or overlap) and its error bars and then compute the ratio of the curve over the mean of all inter-instantiation error bars. Error should be added and aggregated too, in lines with some of the rules above.