Computational Attention

Attention and Images Memorability

Matei Mancas — 2013-05-23T11:54:00+00:00

More coming soon.

What is memorability ?

The image memorability consists in the faculty of an image to be recalled after a period of time. Recently, the memorability of an image database was measured and some factors responsible for this memorability were highlighted.

Link between visual attention and memorability

We investigate the role of visual attention in image memorability around two axis. The first one is experimental and uses results of eye-tracking performed on a set of images of different memorability scores. The second investigation axis is predictive and we show that attention-related features can advantageously replace low-level features in image memorability prediction. From our work it appears that the role of visual attention is important and should be more taken into account along with other low-level features.

Rarity-based saliency models

Matei Mancas — 2012-09-13T15:12:00+00:00

This project describes the latest developments of the idea we initiated in 2007: what attracts bottom-up attention in an image is not a feature by itself but features' global rarity in the image and their local contrast. This method was initially called LG2 (Local Global 2) in Matei's PhD thesis [Computational Attention: Towards Attentive Computers] and it is called now RARE2012. We provide here a description of this approach.

Architecture

It uses a sequential bottom-up features extraction where first low-level features as luminance and chrominance are computed and from those results medium-level features as image orientations are extracted (level 1). In the second time, a rarity mechanism is then applied (level 2). Finally, we fuse maps into a single saliency map (level 3).

Some visual results

We present bellow some qualitative results of the model on different databases.

First row: Human eye-tracking density maps of the images.
Second row: RARE saliency map results for the corresponding images.

Toronto database (Bruce and Tsotsos)

Kootstra's database

Jian Li's database

Codes

You can download some ressources here
You may find the RARE2007 codes which are those developed for the initial approach in 2007 and RARE2012 which is the architecture described above.

Related Publications

N. RICHE, M. MANCAS, M. DUVINAGE, M. MIBULUMUKINI, B. GOSSELIN, T. DUTOIT, 2013, "RARE2012: A multi-scale rarity-based saliency detection with its comparative statistical analysis", Signal Processing: Image Communication, issn:0923-5965,, doi:10.1016/j.image.2013.03.009.
N. RICHE, M. MANCAS, B. GOSSELIN, T. DUTOIT, "RARE: a New Bottom-Up Saliency Model", Proceedings of IEEE ICIP, 2012.
M. MANCAS, "Relative Influence of Bottom-Up and Top-Down Attention", Attention in Cognitive Systems, Lecture Notes in Computer Science, , 2009.
M. MANCAS, "Computational Attention: Towards Attentive Computers", Similar edition, CIACO University Distributors, 2007.

Dynamic saliency models

Matei Mancas — 2012-09-13T14:12:00+00:00

2D (RGB Videos)

This project deals with the selection of relevant motion from multi-object movement. The proposed method is based on a multi-scale local contrast and global rarity quantification to compute bottom-up saliency maps. The algorithm only uses motion features (direction and speed) but can be easily generalized to other dynamic or static features. Video surveillance, social signal processing and, in general, higher level scene understanding can benefit from this method.

Secondly, we investigate the effect of the embodiment of attentive visual selection in a pan-tilt camera system. The constrained physical system is unable to follow the important fluctuations characterizing the maxima of a saliency map.


Abnormal motion detection: red areas are interesting due to the speed feature while cyan areas are interesting because of the direction features. White areas attract attention when using both features.	Pan-tilt simulating human gaze. The idea is to maximize the information acquired while minimizing the needed effort (minimizing pan-tilt motion and motion variation).

3D extension (RGB-D Kinect sensor)

We introduce here the use of the Kinect sensor and its depth map for saliency detection based on 3D features extraction and their rarity quantification to compute bottom-up saliency maps. We show that the use of 3D motion features namely the motion direction and velocity is able to achieve much better results than the same algorithm using only 2D information. This is especially true in close scenes with small groups of people or moving objects and frontal view.

The video bellow, shows people selection based on direction saliency:

The white square shows the selected people (the one worthy of attention).

Related publications

N. RICHE, M. MANCAS, D. ĆULIBRK, V. ĆRNOJEVIC, B. GOSSELIN, T. DUTOIT, 2012, "Dynamic saliency models and human attention: a comparative study on videos", Proceedings of the 11th ACCV, 2012.
N. RICHE, M. MANCAS and al., "3D Saliency for Abnormal Motion Selection: the Role of the Depth Map", Proceedings of the 8th ICVS, 2011.
M. MANCAS and al., "From Saliency to eye gaze: embodied visual selection for a pan-tilt-based robotic head", Proceedings of the 7th ISVC, 2011.
M. MANCAS, N. RICHE and al., "Abnormal motion selection in crowds using bottom-up saliency", Proceedings of the IEEE ICIP, 2011.

Saliency benchmarks

Matei Mancas — 2012-09-12T18:26:00+00:00

[Videos Benchmark] [Still Images Benchmark]

Videos Benchmark

ASCMN video benchmark

Here you can download the ASCMN database.The database contains 24 videos split into 5 classes :

The following models were already tested on the ASCMN database:

Model	Link	Features
Mancas	Paper	Uses motion features only.
Culibrk	Paper	Uses motion and static features.
SEO	Web	Uses spatio-temporal ressemblance.
SUN	Web	Uses spatio-temporal statistics.

By using the following metrics:

Metric	Link	Features
AUROC	Web	Area under the ROC curve: focus on saliency location at gaze positions.
NSS	Web	Normalized scanpath saliency: focus on saliency values at gaze positions.
KL Divergence	Web	Kullback–Leibler divergence: focus on the discrepancy of saliency and gaze distributions. Thanks to Andrei Zaharescu for fruitful discussions about this metric.

And the results are here:

[click to enlarge]

If you want to test your own saliency model, please:

Download the ASCMN database
Replace in the script "Evaluation.m" the line under the comment "Your saliency map" by your own model
Run the script and you will get a .mat file with the metrics results for each video frame
Send us your .mat file so we can add your model to this webpage
If you use the codes and database, cite the paper that you can find here

Still Images Benchmark

Saliency Evaluation An attempt of objective model evaluation using several eye-tracking databases and several similarity metrics. The AUROC measure here does not take into account the centered bias and provides fare evaluation of bottom-up models. (University of Southern California Ali Borji 2010)
Saliency Benchmark Benchmark data set containing 300 natural images with eye-tracking data to compare model performances. Several metrics are used and our model is in the top five (M.I.T. Tilke Judd 2012)

Saliency and Social Gaming

Matei Mancas — 2012-09-10T17:01:00+00:00

This project deals with the design of an intelligent game system capable of selecting the players who exhibits the most outstanding behavior from groups of people playing on a network. Players can be all in the same place (maximum 3) or in several locations if connected to the web. In that way, the total amount of players can be very large. Tests were made with 3 players in a single location and with 2 different locations of 2 players each. The system uses both static and dynamic features extracted from the upper part of the players’ bodies such as symmetry, contraction index, motion index or height. Those features are extracted using the RGB-D Kinect sensor and their relative contrast and time evolution enable an adaptive selection of the most salient or different behavior with- out any complex rules. First users’ feedback and eye tracking tests are shown and applications to social interactions are presented.

KinAct: social interaction for data processing - Virtuastarch from NUMEDIART on Vimeo.

Events

Our social game has already been demonstrated to a large public during festivals. You can check all the related events here.

Related publications

F. ZAJEGA, M. MANCAS, R. BEN MADHKOUR, J. LEROY, N. RICHE, F. ROCCA, Y. P. RYBARCZYK, T. DUTOIT, 2012, "KinAct: the attentive social game demonstration", Demonstration at ACCV 2012.
J. LEROY, N. RICHE, F. ZAJEGA, M. MANCAS, J. TILMANNE, B. GOSSELIN, T. DUTOIT, "The Attentive Machine: be Different!", Intetain 2011.
M. MANCAS, N. RICHE, J. LEROY, B. GOSSELIN, T. DUTOIT, "Toward a Social Attentive Machine", AAAI Fall Symposium 2011.
M. MANCAS and al., "Human-motion saliency in multi-motion scenes and in close interaction", 9th International Gesture Workshop, 2011.
M. MANCAS, D. GLOWINSKI and al., "Gesture Saliency: a Context-aware Analysis", GECHCI, LNAI 5934, 2009.
M. MANCAS, D. GLOWINSKI and al., "Real-Time Motion Attention and Expressive Gesture Interfaces", JMUI, 2009.

Saliency for Crowds Analysis

Matei Mancas — 2012-09-10T16:16:00+00:00

Video processing for dense crowds is a field of computer vision which has specific properties as it is impossible to obtain individual object tracking or it is difficult to acquire databases of specific events. The main applications of crowd monitoring are in video surveillance but applications around social signal emerge.

Our approach is a real-time contribution to abnormal event detection and it uses a model of the human attention. It is possible to precisely locate the area into the crowd where abnormal or surprising events occur.

The implementation is based on a two-step analysis of the perceived motion speed of different areas within the crowd. In a first step bottom-up instantaneous attention is applied on the speed feature. In a second step, repetitive behaviors (in terms of speed) located in some scene areas are used to build models which are able to inhibit part of the motion in those areas if it is not novel. As individual tracking is not an option in crowd analysis, a motion grouping pre-processing is needed: the crowd movements are segmented according to their motion speed and spatial density.

The video bellow demonstrates the pre-processing step during which the crowd motion is segmented into low quantity of motion to high quantity of motion (from left to right) and from low to high density (from top to down). The initial video is on the right side:

Crowd segmentation of the right initial video. The first 6 images display people motion quantity and the degrees of compactness of the crowd from

Related Publications

M. MANCAS, 2010, "Attention-based Dense Crowds Analysis", Proc. of the 11st International WIAMIS 2010, Desenzano del Garda, Italy.
M. MANCAS, B. GOSSELIN, 2010, "Dense crowd analysis through bottom-up and top-down attention", Proc. of the BICS 2010, Madrid, Spain.

Local

Matei Mancas — 2012-09-10T09:31:00+00:00

A very simple local contrast algorithm inspired from LG1 and LG2 algorithms presented in Matei Mancas PhD thesis. This is NOT the LG1 or the LG2 method (LG2 method also called RARE2007 is available above) and no global information or spatial orientation is taken into account here. May be interesting for images where the local contrast is the most important.
This code is part of the model presented in this paper :

M. MANCAS, L. COUVREUR, B. GOSSELIN, B. MACQ, 2007, "Computational Attention for Event Detection", Proceedings of ICVS Workshop on Computational Attention & Applications (WCAA-2007) , Bielefeld, Germany, Mar 2007.

Global

Matei Mancas — 2012-09-10T09:26:00+00:00

A very simple global rarity algorithm inspired from the algorithms presented in Matei Mancas PhD thesis. This is NOT the LG1 or the LG2 method (LG2 method also called RARE2007 is available above), but only a fast implementation of the global approach. No local information or spatial orientation is used here. May be interesting for images with rare defects which have low contrast.
If you use this code, please cite this paper :

M. MANCAS, C. MANCAS-THILLOU, B. GOSSELIN, B. MACQ, 2006, "A rarity-based visual attention map -application to texture description -", Proc. of IEEE International conference on Image Processing (ICIP 2006), Atlanta, USA.

Computational Attention

Attention and Images Memorability

More coming soon.

What is memorability ?

Link between visual attention and memorability

Related Links

Rarity-based saliency models

Architecture

Some visual results

Codes

Dynamic saliency models

2D (RGB Videos)

3D extension (RGB-D Kinect sensor)

Related publications

Saliency benchmarks

Videos Benchmark

Still Images Benchmark

Saliency and Social Gaming

Events

Related publications

Saliency for Crowds Analysis

Related Publications

Related Publications

Local

Global