CBIR

Content-based image retrieval

Videos!

ImageSorter.
Picasa. For manage your digital photo collections.
oSkope visual search – A fun way to search. Amusing system for visual search and representation of search results.
Viewzi.com the visual search engine.
Searchme Demo.
PicQuest – A content based image retrieval system.
AmiCBIR – Video Demo.
LIRE Demo.
PIcporta – Color Search.

Content-Based Image Retrieval.
Sorting and Ranking Images in Photoshop Lightroom.
Visual Search – Revolutionary Visual Based Search Results.
Get Connected – Web Trends – Visual Search Engines.
Browsing images by color and geometrical composition.

FaceTracer: A Search Engine for Large Collections of Images with Faces.
Facesaerch.com – face search engine.

Mobile Visual Search: Products.
Mobile Visual Search: Wine.

Written by Sergey Egorov

No tags for this post.

What we can, what we can’t and what purpose do we attain to?

We develop the software for automatic adult images detection. And we press towards adaptation the best up-to-date methods (based on analysis of colours, texture, shapes; with using machine learning techniques) for improvement our porno filter. We dynamically move to this aim attainment and at present time our filter is not bad at all (see screenshot).

We can detect human faces (see screenshot) and we are actively working on the creation system for identification people by faces (by means of using the face recognition methods).

We can search by image sample (by colour and texture, see example). We try different methods for matching images by colour and texture, compare them with each other, choose the best or combine different methods for perfomance and quality improvement. We plan to extend our mechanism by spatial layout and shapes matching.
As a result we represent the image as feature vector of definite length (not very long, approximately from ten to thirty real numbers; see example). And then compare these vectors by means of assigned metrics.
So it is not difficult for us to search similar images in findings (as, for example, Google and Bing do), to sort images by similarity with sample and so on.

By means of combining our similarity matching mechanism with clustering algorithms, we can cluster image space (see example – there are two clusters) and detect images duplicates (see example). It is very actual task and problem, from which the most modern search engines suffer (for example, Bing). We try to accelerate our technique by using LSH algorithms and k-NN algorithm.

We can visualize the image space (see example) and classify images by dominant colour (by one of the basic colours; see example: query “flowers” in Yahoo, dominant colour – pink).

Written by Sergey Egorov

No tags for this post.

Another interesting projects, related to images and CBIR!

Interesting system for online image processing and the article about it (in Russian).
Wavelet-based Virtual Microscope from Wang Research Group.
Image Comparer (Russian Software) and survey (in Russian).
ImageStamper from Behold “is a free tool for keeping dated, independently verified copies of license conditions associated with creative commons images”.
The survey of Image Dupeless (in Russian). Image duplicates detection.

Written by Sergey Egorov

No tags for this post.

Conferences!

Two very important events in the world of Information Retrieval, Image Retrieval and Computer Vision:

Written by Sergey Egorov

No tags for this post.

Visualization aids us to perceive information!

Different interesting variants of visual results presentations:

Spatial Visualization for Content-Based Image Retrieval
Google Demo
Viper – Multimedia Information Retrieval (Geneve University)
Interesting browser plugin (survey in Russian) and the official site
Tag cloud for images (right top corner)
CloudTuner – visual constructor (Israel search engine)
CoolFlick and the article about it
One more example
Sbrows (the interesting navigation)

Written by Sergey Egorov

No tags for this post.

Specific web search engines, services, startups and research groups – search of next generation!

What is a new search concept? Watch this video and read this article (in Russian) to realize it fully.

Video retrieval (by means of CBIR techniques):

Viewdle – video search engine. Faces search in video-archives.
Video Google Demo. Search of similar scenes by image fragments.

TV news processing. Carnegie Mellon University.

Columbia University:

An object oriented video search engine. Search of similar scenes by image and sketch sample.
VisualSEEk – A joint spaital-feature image search engine. “The system finds the images that contain the most similar arrangements of similar regions.”

Image retrieval:

Tineye. Search by Image Query. It is a very convenient service with good quality performance. Recently Google has took a great interest in Tineye.
Picitup and it’s video on YouTube. They’ve bravely assumed the challenge “Can anyone do for images what Google does for text?”. And, we must admit, it is now one of the best visual search services. They offer CelebrityMatchUp (by faces), Visual Shopping (search similar products by colours, shapes, textures, brand and much more), Visual Search (first you search by text, and then precise by colour, simple shapes, simple categories).
GazoPa and the article about it (in Russian). It is a private search engine (only by invitation), you can search by example, sketch and outlines (by colour, texture, shape, spatial arrangement and worse by faces). Recently “GazoPa for iPhone” has been released.
QBIC (IBM’s Query By Image Content) and it’s State Hermitage Museum Project. You can search by sample and outline (by colour and set of colours with their percentage, by spatial organization of colours, by simple geometric figures and their spatial arrangement).

Photodate. Search people by photo and help in revealing the infringement of copyright.
Photo Finder on facebook. Scan photos and detect all faces appearing on them, recognize & auto tag (automatically match names to all faces), face galleries (search for friends and browse their face galleries – discover the photos you never saw before).
PolarRose. “Polar Rose detects and matches the faces in your photos – so you can easily name people and share photos with your Facebook friends”.

Like.com – Visual Shopping. You can search by brand, by category and by visual features: “Color Match”, “Shape Match”, “Pattern Match” (by means of texture), “Detail Search” (sub-image retrieval).
Etsy (”Shop by Color”). It is online shop of handmade goods. Unfortunately, you may choose only one colour, but the process of choosing colour is finely animated.

Visuvi Visual Search. Offers “Medical Image Search”, “Product Search” (Visual Shopping), search people (by faces) and products in social networks.
IRMA (Image Retrieval in Medical Applications). The student project.
NeuroInformatics (Mitre). NeuroImagery Retrieval & Visualization, including medical applications.

MFIRS (Multi-Features Image Retrieval System). Here user can choose the metrics and the method of images matching.
CIRES – Content Based Image REtrieval System and it’s new site. There are several individual classes with proper settings for every one (metrics, weights for different distances, colour space).
Behold. Offers very interesting opportunity – find images tagged with definite keywords that look like a picture of something (animal, beach, bird, boat, building, car, city, cloud, face and so on).
Collage. “The City of London Libraries, Archives and Guildhall Art Gallery is host to COLLAGE, an image database containing over 20,000 works of art from its collections.”
DEVA Group.
Viper – Multimedia Information Retrieval (Geneve University).
ImBrowse. A Browser for Large Image Databases.
Cbir.ru. Russian Massmedia Laboratory Project – search in image collections and videostreams.
Japan CBIR.

Retrievr. Interactive search by sketch, but results do not impress.
Search by sketch system. And video about it.

Viim and the article about it (in Russian). Do not impress. Service does not work correctly. According to the article, “big red ball” finds some red balls, “blue triangle” – some triangles.
Tiltomo and the article about it (in Russian). You can find similar photos by “theme” or by “color/texture”.
Riya. Assert that they can search by colour, texture and shape. Here we may search similar people (by faces), similar objects. Riya offers to use it’s “face recognition and text recognition, to search your personal photos”.
XCavator and the video about it. Search by example and sketch, by dominant colour and colour sets with their percentage, by spatial colour information, but first they find by keywords.
Piximilar. Visual similarity search for large image collections, it can be used in combination with keywords to refine searches on extremely large collections. “Piximilar’s visual similarity technology uses sophisticated algorithms to analyze hundreds of image attributes such as colour, shape, texture, luminosity, complexity, objects and regions.” Also here you can choose a set of colours with their percentage.
PixID. Identify editorial images in print and online publications. “PixID is a comprehensive, automated image monitoring service that uses advanced image identification algorithms to identify where your images are being used in print publications and the Internet.”
Recogmission, Picollator and their blog (in Russian). They use face recognition technology to search similar photos and organize photo archives. They assert that they can find objects on images (but nowadays only faces – it’s a great lack). They can cluster and classify image collection, search by visual example, reveal the infringement of copyright. There is also piFilter (the Web engine for automatic adult images detection). They offer Picollator.mobi for mobile search and funny variant of marking results (by tomato-icon).

Use Flickr results:

MUFIN. First search by keywords, then click on “Visually similar” link.
Flickr Color Selectr. We can choose the dominant colour.
Squared Circle Colr Pickr. We can also choose the dominant colour, the quantity of colours is much more.
Multicolr Search Lab. You can choose a set of colours with their percentage (by means of clicking the same colour several times).

Wang Research Group and the projects:

Wavelet Image Search Engine. Wavelet based images matching (by colour and texture). They also have wavelet based pornography elimination.
Simplicity and the same
Simplicity for Satellite Imagery and the same
CLUE (Cluster-based Image Retrieval)
Advancing Digital Imagery Technologies for Art and Cultural Heritages
Alip (Automatic Linguistic Indexing of Pictures)
Alipr (Automatic Photo Tagging and Visual Image Search). Fusion of visual search and text tagging – auto-tagging based on visual similarity.
Airplanes.net Image Similarity Search. Use Wang Simplicity technology.

Written by Sergey Egorov

No tags for this post.

Do web search giants and smaller ones neglect image retrieval?

Ignoring image retrieval, based on visual characteristics, is, at least, short-sighted search policy. Of course, direct search by image example in large databases is very time-expensive process. Most web engines offer different instruments for more precise (visual similar) search in hits, received after text query.

Google has it’s own Google Similar Images Labs. So we can search similar to our example images. Also we can filter results by image size, by content (news, faces, clip art, line drawings, photo content), by dominant colour (red, orange, yellow, green, teal, blue, purple, pink, white, gray, black, brown). All these opportunities are embedded to Google search machine. Unfortunately, we can search similar not for all images, only for some.

Yandex offers us the similar opportunities: filters by size, by dominant colour (red, orange, yellow, green, cyan, blue, violet, white, black), we can choose portraits and wallpapers. Yandex has interesting connexion between keywords and colour scale (for example, query “purple”).

Yahoo has embedded porno filter, filters by size, by content (wallpapers), by colour spectrum (”Black & White”, “Color”).

Ask is similar to Yahoo: filters by size, content and colour spectrum.

Rambler can filter porno (levels – temperate, strict, disable), filters by size and content (wallpapers).

Nigma, a dynamically evolutionary search system, offers only filters by size and content (wallpapers).

GoGo, a search engine from Mail company, asserts, that duplicates are removed from result hits. But this mechanism is not transparent and far from ideal.

Bing, a new web search engine from Microsoft Corporation, offers us some interesting opportunities. It has porno filter (also with 3 levels, as Rambler). Here we can filter by size, by model (square, wide, high), by colour spectrum, by style (photos, illustrations), by human beings presence (faces, head and shoulders). Also when you direct mouse to any image, you can click on “Show similar” link and gain more relevant hits. Of course, it is not ideal, but in most cases you can receive more precise results for any image (in contrast to Google).

Picsearch. Search by text and precise by size, colour spectrum, simple categories (animation, pictures). Picsearch has a family friendliness that allows children to surf in safety as all offensive material is filtered out by filtering systems.

Nevertheless, all this engines haven’t a satisfactory duplicates detection mechanism. Examples (query “mona lisa”): in Google, in Yandex, in Yahoo, in Ask, in Rambler, in Nigma, in GoGo, in Bing (and one more), in Picsearch. GoGo, you can notice, much better than other engines in duplicates detection, however the results are evidently less relevant. Opposite, Bing has the diversity of relevant images, but, at the same time, it has many duplicates.

Thus, as we could notice, any modern search engine simply can’t permit of neglecting multimedia retrieval. They should, at least, have embedded porno filter, filters by size and colour spectrum, by content and style, by dominant colour and, of course, opportunity of visual similarity searching. The nearest future – clusterization and classification of images collection (by colours, by content – portraits, monuments, sunsets, vehicles, landscapes and etc., by style – wallpapers, photos, illustrations, line drawings and etc.), search by outlines and rough sketches (for example, bright orange spot on less bright orange background – sunset), search simple geometric shapes (such as circles, ellipses, squares, rectangles, stars, crosses, triangles, hearts and etc.). Nowadays it is necessary to have good and visually convenient duplicates detection mechanism and opportunity of search by set of colours with their percentage (like QBIC).

Written by Sergey Egorov

No tags for this post.

In solving what tasks can CBIR aid?

Here the most actual applications of CBIR are presented:

  • detection of objectionable content (porno, erotics and so on);
  • person identification (by means of face detection and recognition techniques), matching with celebrities (like Picitup Celebrity Match);
  • criminal identification (fingerprint, face, iris);
  • search by image sample, outline and sketch (by colour, set of colours, texture, shape, spatial layout and other features);
  • search of similar parts and fragments in images (sub-image retrieval);
  • images duplicates detection, clusterization (with similar images in each cluster) and classification of images collection;
  • organization of photo archives and multimedia encyclopedias;
  • visual shopping (examples: Like.com, Picitup, Etsy, Visuvi);
  • geographic information systems, processing of different maps, satellite and aerial photos;
  • in fashion and textile industry (texture analysis);
  • in absolutely different spheres (stenography, astronomy, ecology and environment and etc.);
  • online applications for art and art history (examples: QBIC Project and Wang Project);
  • medical image archives and processing, pathology detection and aid in diagnostics (examples: IRMA Project, Visuvi);
  • trademarks and logos databases and revealing the infringement of copyright (like PixID);
  • the similar applications for mobile phones (like Picollator.mobi) and iPhones (like Apple);
  • the similar applications for video retrieval (search of similar persons by faces and similar scenes by fragment, sample, sketch, spatial arrangement of colours and regions and etc.), example – TV news processing;
  • document image retrieval – revealing the text in images, signatures, stamps, handwritings and words matching, search by text query without any character recognition.

Of course, it is far from being the complete applications list!

Written by Sergey Egorov

No tags for this post.

What is Content-based image retrieval?

Wikipedia:

Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR) is the application of computer vision to the image retrieval problem, that is, the problem of searching for digital images in large databases

Ok. It is absolutely correct.

All web search engines leaders, such as Google, Yandex, Yahoo, Ask and etc., find multimedia content by means of text descriptions. Billions of images are tagged manually by keywords, so we have got used to search multimedia files by text queries.
But this approach have two cardinal and critical drawbacks:

  • text descriptions are very subjective and therefore far from perfection, the same image can be depicted by absolutely opposite keywords;
  • tagging multimedia files is very hard, time-taking and staff-resources-taking work.

And what is CBIR? It is a combination of different areas of knowledge, such as pattern recognition, object matching, machine learning, wavelet filtering and so on. CBIR is devoted to understanding visual characteristics of images without any text descriptions.

Try to think it over! The major part of information we perceive through our eyes, the major part of information we perceive at first several seconds, and even at fractions of a second. Most of us are lazy and don’t like to read texts, especially foreign texts. We like books and journals with illustrations, we prefer catalogues of goods with their photos, we choose the pleasure resort for our vacation looking at photos of hotels and sightseeings. You may continue this list infinitely! Oh, even small kids and children are able to distinguish basic colours and simple objects. They can’t read and write, but they perceive everything around them through their eyes.

Modern techniques and algorithms are able to understand images content even nowadays, it isn’t fantastics. It is reality and it is our nearest future!

Written by Sergey Egorov

No tags for this post.