The final stage which is arguably the most fascinating one involves
the use of the model above, as well as the correlations learned and
recorded for that model. It is possible to carry out a search which
is driven by the difference calculated between the model and a given
target image. In practical terms this means that fitting of the existing
model will slowly be improved until the model approximately covers
the target^{16}. The model state then holds (in the form of parameter values) some
information about the target image and this information can be further
analysed.

The search is reliant on error (or conversely similarity) measures which are repeatedly calculated after each attempted parameterisation of the model. Each such change in parameter values is primarily guided by the matrices that express the correlation between variation modes (the Similarity transformations as well as modes of appearance change) and the intensity values.

The model, as shown in Figure 4, is initially placed somewhere inside the image frame, with reasonable proximity to its target. If the model is placed too far from its target, there is a danger that it will not be able to converge to the target correctly. It will most likely get stuck in a local minimum and the outcome can be severe in a more crucial application such as medical imaging (or perhaps more drastically, computer-guided or -aided surgery). The reason why good initialisation is essential is that significantly large displacements are rarely learned off-line and the difference between the target and the model is quite meaningless unless there is at least some partial overlap or commonality.

The algorithm which is used to perform the search sensibly has a general form that resembles the following:

- Place the appearance model somewhere in the image, preferably at the
centre where the target of interest is likely to lie
^{17}. - For the appearance model in its current state and the static target
do:
- Calculate the difference between the model and the target.
- Using the correlations learned off-line
^{18}, set new values for the parameters . - Compute the new difference measure between the model and the target.
- Save the new state of the appearance model if the difference has been lowered, i.e. similarity is being approached.
- If not, try re-adjusting the parameter change, potentially with inclusion of a scaling coefficient and so on. This often achieves good results, although it is a heuristics-driven technique.

- Iterate while no convergence has been reached and improvements are still observed at times.

Figure 4: Model and target fitting

The technique of matching an appearance model to a target image is well-depicted by a staged simulation, a video clip or a large sequence of images as in Figure 4 above. Surprisingly, only a few dozens of iterations are required in order to get good results. This of course depends on the algorithm, the magnitude of the problem and its internal intricacies, e.g. fitting a perfectly round ball versus a human hand.