Image Ranking by global feature estimation

Having an automated image ranking process would be very beneficial to companies such as 500px or Gettyimages where 1000's of images are ingested/uploaded every day and are traditionally ranked manually by an editor or by crowdsourcing. This human intervention can be temporarily avoided if images can be ranked by estimating it's quality.

To build an image quality estimator, following image quality properties were used for this proof-of-concept:

• Blur Estimation:
     Basically, this technique estimates the proportion of blurred pixels. Results are in the range 0-1. A higher number implies a sharper image.
  Example code: Blur-detection
• Sharpness:
     Sharpness measures the clarity and level of detail of an image.
  Example code: Estimation-of-Image-Sharpness
• Colorfulness:
     Though there is no real way to estimate the colorfulness of an image but there are various algorithms to measure the quality of image based on compression. One such algorithm is from Hasler and Susstruck's colorfulness metrics. Paper
• Naturalness:
     Naturalness is basically a single valued summary of how natural the colours in an image are. One such algorithm is Color Naturalness index (CNI) defined by Huang, Qiao & Wu. Paper
• Image Contrast:
     Estimating the contrast of an image.
• Colour Contrast:
     This is basically weighted average of the average colour difference of all the segments in the image.
• Brightness:
     Extract the average brightness of an image.

In this following examples, I have 3 ranks. Rank 1 = Good quality, Rank 2 = Medium and Rank 3 = Bad Quality images. Following is a snippet of my rank calculation.

Image Rank 1 Examples
"blur":0.9660969387755102 "sharpness":0.5067756918923018 "color":0.1462474560577043 "quality":0.8798513615119794 "contrast":0.08234799394638759 "colorContrast":8.328080618896797 "brightness":0.14739056340426107	"blur":0.966530612244898 "sharpness":0.49986904977578045 "color":0.19059606130699966 "quality":0.8476687344507642 "contrast":0.08834899629937157 "colorContrast":25.384664271394108 "brightness":0.24995158193079098	"blur":0.7220025510204081 "sharpness":0.15089643141301262 "color":0.4003952223119604 "quality":0.9609951671164367 "contrast":0.2292792429036637 "colorContrast":32.053295457355276 "brightness":0.4115245849574555
Image Rank 2 Examples
"blur":0.8221428571428572 "sharpness":0.15935783910386503 "color":0.04443692154503701 "quality":0.5085070823398357 "contrast":0.14823618660855284 "colorContrast":26.781796490365295 "brightness":0.27380289202677777	"blur":0.9705484693877551 "sharpness":0.41994123003434913 "color":0.19996393727671855 "quality":0.3153617863913573 "contrast":0.10908353919667178 "colorContrast":43.63807505087183 "brightness":0.18082135350839887	"blur":0.8633290816326531 "sharpness":0.2271008743069167 "color":0.12817043260208896 "quality":0.6680113666484762 "contrast":0.05242860949574584 "colorContrast":44.47249174965709 "brightness":0.222476585047196

From the above results, an simple image ranking system can be automatically performed by estimating the global image features values. More complex algorithms can be used to further improve the image quality estimation process such as Bokeh Estimator which can be used to detect background blur and camera focus[link].
Note: source code to follow soon at my github

Image processing Benchmarks

For this benchmark the following most widely used image processing libraries were considered.

- imagemagick [http://www.imagemagick.org/script/index.php]
- graphicsmagick [http://www.graphicsmagick.org/]
- epeg [https://github.com/mattes/epeg]
- opencv [http://opencv.org/]
- vips

Test environment:
Memory: 5.8 GB
Processor: Intel Xeon CPU W3530 @ 2.80Ghz x 4 Core
OS: Ubuntu 13.04 / 64 bit
Graphics: Gallium 0.4 on AMD Redwood

Original Image - 350KB - 3168x3168 pixels | Resized to 640x480
imagemagick x 3.69 ops/sec ±2.27% (23 runs sampled)
gm x 5.03 ops/sec ±0.68% (29 runs sampled)
opencv x 19.18 ops/sec ±1.27% (49 runs sampled)
epeg x 35.49 ops/sec ±1.16% (60 runs sampled)
vips x 40.62 ops/sec ±5.01% (69 runs sampled)

Original Image - 1 MB - 3000x2000 | Resized to 640x480
imagemagick x 4.97 ops/sec ±2.35% (29 runs sampled)
gm x 5.00 ops/sec ±0.54% (29 runs sampled)
opencv x 15.15 ops/sec ±1.36% (41 runs sampled)
epeg x 27.47 ops/sec ±0.98% (69 runs sampled)
vips x 36.26 ops/sec ±6.05% (89 runs sampled)

Original Image - 15MB - 5382x6254 pixels | Resized to 640x480
imagemagick x 0.87 ops/sec ±1.20% (9 runs sampled)
gm x 0.87 ops/sec ±0.66% (9 runs sampled)
vips x 1.74 ops/sec ±0.43% (13 runs sampled)
opencv x 1.88 ops/sec ±4.09% (9 runs sampled)
epeg x 3.87 ops/sec ±0.78% (14 runs sampled)

From the above results, VIPS seems to be the fastest among all followed by epeg and opencv. But one thing to consider is the features provided vs performance. Libraries such as VIPS and EPEG are more optimized towards image resizing and image cropping while opencv, graphicsmagick and imagemagick provides a slew of image processing and analysis features.

Code snippet for benchmarking: https://gist.github.com/karthik20522/7605083

Event Viewer - Image Search

"Event Viewer" is an yet another attempt to visualize images similar to my {re}Search Timeline project.
Demo at : http://karthik20522.github.io/EventViewer



The whole point of this proof-of-concept project is to visualize the images from the perspective of the events rather than just displaying an grid of images. For example, a search on GettyImages.com website basically displays a list of images in a tabular fashion which provides no sense of association between individual images being displayed. But having them grouped together as part of an event provides a sense of association and correlation between images.

Displaying images is always a tricky business. A dominant color filter technique could probably provide an alternate way of scanning thru images as user might be more interested in images of particular color than the fine details of an image.

From a technology standpoint for building this project was nothing special.

• ASP.NET MVC 4 - Razor
• Amazon SQS - for event scrapping from GettyImages
• Connect API for event and image detail lookup
• MongoDB
• Dominant Color Extraction

Source code at: https://github.com/karthik20522/EventViewer

{Re}Search timeline - How I Built This

Demo site coming soon..

Another proof of concept and another interesting way of viewing images. Checkout my previous yearinimages.com post. The idea behind Yearinimages was to provide a very eye candy way of representing the best images of 2011. But unfortunately mosaic layout provides no sense of navigation. {Re}search project was my attempt to avoid this pitfall of navigation while still holding the visual way of presenting data with images.



The idea behind this POC is to provide the user a timeline based navigation of events. For example, when you search Japanese Earthquake or Oscar awards which when viewed as a timeline of events, would provide more visual perspective than just displaying a grid of images.

So how did I build this? Well, I used Gettyimages connect API to extract event and image information and Timeline plugin by verite for building the timeline. Addition to above api's and plugins there were quite a few hacks that was required to make this happen. Following is how I did it:

Extracting the event information:
Thou gettyimages connect API provides the search by personality , keyword and search phrases unfortunately it only provides limited refinement options on events. Basically it's not possible to get a list of events similar to the one the gettyimages website provides like following screenshot:



I do believe that the refinement can be customized per customer basis with connect API but I am not sure. Anyways, for extracting the list of events for a particular search I screen scrapped from gettyimages.com website. Following is code snippet of screen scraping using WATIN ui automation framework:



Now that we have list of event ids screen scrapped we can use gettyimages connect to get event details for displaying on UI. Following is code snippet to call connect API using my connect API library (its nothing but a helper library to call the connect API rest service)



Setting up timeline ui:
Now that we have list of events we need to convert it to the format the timeline plugin expects. Timeline plugin takes json as its data source. I used json2csharp.com site to convert the sample json format that timeline github page provided to c# data model that can be used code behind. I used json.net to convert the timeline c# object to json string. Code snippet for converting event list to timeline object.



Note that the hard-coded html in timeline.text property is to provide a autosuggest capability for user. The code above is just for demo purpose's to show that timeline plugin is capable of taking in Html as part of it's json data. Html rendering of the above code:



At my 1st iteration, I hooked up the events to default timeline functionality. But what I would like is to allow users to click on an event and display all images associated to that event. Since this is a custom functionality some hacks to timeline code had to be done. Like the following:
- Associating eventID to the event timeline div so when timeline is clicked the code can determine which eventID it's referencing too
- Adding a Ajax loading gif to every event
- Upon sliding to an event, do Ajax call to server to get images associated to that event
- Display the images when Ajax call is completed
- Show image information when clicked

Framework, plugins, API used:
- jQuery 1.7.1
- jQuery light box
- wookmark jQuery plugin
- Timeline by Verite.Co
- .net framework 4.0/mvc3
- Newtonsoft json.net
- WATIN ui framework
- json2csharp.com for converting the template json to c# object
- gettyimages connect API

YearInImages.com - How I built this?

Image mosaic source at https://github.com/karthik20522/ImageMosaic



Following post is a very high level technical overview of how the images were constructed and stored and how Getty Images Connect API was used to make this site possible. Site link: yearinimages.com

Connect API story:
All editorial images in Getty images are associated to an event. An event could be anything, Oscar awards, Olympics etc with a bunch of images associated to it. Every event has a numerical id that makes the event unique in our system. The years in focus 2011 images are categorized into 4 main categories (News, Sports, Entertainment, Archive). Each of these categories has many Events and with some simple scripting the EventId’s of each Event was scraped from the site and the image associated with these events was queried using Connect API. Following is a code snippet of querying Getty Connect API:
More technical information of GettyImages Connect API can be found at https://api.gettyimages.com/apis

C# wrapper to call GettyImages connect API: https://github.com/karthik20522/GettyConnect

Note that Connect API only returns back maximum of 75 images per search, so you would have to loop if an event has more than 75 images. The image metadata that Connect API provides has enough information to describe an image but you could also query GetImageDetail endpoint to get complete metadata details of an image. Once the image information was queried and saved (yearinimage uses mongodb as datastore), the image thumbnails were then downloaded for building image mosaic using following code snippet:

Image mosaic story:
The idea behind building image mosaic is to take the original image and split it to a grid say 20x20 pixels (depending on the size of the image) and compute the average RGB within each of the blocks.



Now unfortunately the thumbnail images that were downloaded from GettyImages are not uniform in size (landscape vs portrait). Since we are splitting the image into square Grid, the thumbnails had to be resized to be uniform. Once the thumbnails were resized, their average RGB values were calculated. Average color code snippet:

The next step was to iterate through each image and find the closest match to each block on the image. Since the image grid is basically a matrix of n x m, this can be serialized and stored in file/db. YearinImage used the following class model to store the matrix:


Upon page load, the browser builds a table/grid and places the image in the appropriate location (x,y) to make it look like a single mosaic. With client side scripting (JavaScript), user interaction with each of the mosaic image was possible like displaying image information when a user hovers over the image. This image information (caption) is fetched from the server using ajax calls. The ajax calls by themselves are relatively fast (less than 100ms) but when a user hovers over many images really fast these ajax calls get’s queued up and the user experiences slowness as the browser is waiting for queued ajax calls to complete. This delay can be avoided by aborting the previous ajax call to the server when a user hovers away from the image. Example of aborting an ajax call is like follows:


As a side note, since the browser displays 100’s of images for the mosaic, this potentially could lead to memory build up on the browser. This can be handled by using revokeObjectURL method. Code snippet:

GettyImages Connect API

Read my blog post on GettyImages Tech blog about how to leverage GettyImages Connect API:

http://blog.gettyimages.com/2012/05/17/leveraging-our-connect-api-oh-the-things-you-can-do

Programming Images: Using WIC to extract metadata

Read my blog post on GettyImages Tech blog about how to use WIC to extract Image Metadata (XMP/IPTC):

http://blog.gettyimages.com/2012/03/27/programming-images-using-wic-to-extract-metadata