Assessment Algorithm
To compare automatic and visual annotations in micro-events

Stéphanie DEVUYST
Université de Mons
Faculté Polytechnique de Mons - TCTS Lab
31, Boulevard Dolez
B-7000 Mons (Belgium)
ph : +32 (65) 37.47.20
fax: +32 (65) 37.47.29

Introduction Assessment method Algorithms Download More


During the DREAMS project, we developped and tested several automatic procedures to detect micro-events such as sleep spindles, K-complexes, REMS, etc.

Unfortunately, we observed that perfomances completed in the same field by other authors were hardly comparable because their methodology, their databases and their assessment methods were radically different.

To solve this problem and allow evaluation and comparison between other future works, we made our database, our visual scorings and our automatic scorings freely available on the web (Here). In addition, we proposed a unique assessment method using a well defined terminology and from which it is possible to establish all the desired confusion matrices. This assessment method is presented below and the corresponding algorithm, implemented under Matlab, can be downloaded Here.

Assessment method

Our assessment algorithm can take into account the visual scoring of one or two experts.

Knowing the start time and duration of the annotated micro-events (on one hand, visually and on the other hand, by the automatic algorithm), it identifies the quantity of each possible covering as illustrated on Fig. 1. These various possible configurations are gathered in 4 categories: type T1 (A, B, or C) corresponds to a correct automatic detection since at least one of the two experts has scored the event like such; type T2 corresponds to a false detection; type T3 (A, B, C, or D) corresponds to a missing detection with respect to one or both experts; and type T5 (A, B, or C) corresponds to multiple coverings implying automatic detection.

Fig. 1. Illustration of the various possible coverings between the diverse micro-events scored by expert 1 (Vis1), scored by expert 2 (Vis2) and automatically detected (Aut).

Once the number of these various types is known, it is easy to deduce the number of true positive (#TP), the number of false positive (#FP) and the number of false negative (#FN) of the different confusion matrixes (see table I). Moreover, the number of true negative (#TN) can be aproximate by (the total duration of the excerpt/the average duration of the micro-event) - (number of true positives + number of false positives + number of false negatives).

Table. 1. Confusion matrices (where "EventDuration" is the average duration of the micro-event and "TotalDuration" is the total duration of the analyzed excerpt).

Finally, it is easy to deduce from these confusion matrices, the parameters commonly used in literature namely:

Return to menu

Algorithms inputs and outputs

Algorithms for the assessment method has been developped under Matlab.

Return to menu

Algorithms download

Do "right click" and then "save target as" to download.

Return to menu


Return to menu