Image Ranking by global feature estimation

Having an automated image ranking process would be very beneficial to companies such as 500px or Gettyimages where 1000's of images are ingested/uploaded every day and are traditionally ranked manually by an editor or by crowdsourcing. This human intervention can be temporarily avoided if images can be ranked by estimating it's quality.

To build an image quality estimator, following image quality properties were used for this proof-of-concept:

• Blur Estimation:
     Basically, this technique estimates the proportion of blurred pixels. Results are in the range 0-1. A higher number implies a sharper image.
  Example code: Blur-detection
• Sharpness:
     Sharpness measures the clarity and level of detail of an image.
  Example code: Estimation-of-Image-Sharpness
• Colorfulness:
     Though there is no real way to estimate the colorfulness of an image but there are various algorithms to measure the quality of image based on compression. One such algorithm is from Hasler and Susstruck's colorfulness metrics. Paper
• Naturalness:
     Naturalness is basically a single valued summary of how natural the colours in an image are. One such algorithm is Color Naturalness index (CNI) defined by Huang, Qiao & Wu. Paper
• Image Contrast:
     Estimating the contrast of an image.
• Colour Contrast:
     This is basically weighted average of the average colour difference of all the segments in the image.
• Brightness:
     Extract the average brightness of an image.

In this following examples, I have 3 ranks. Rank 1 = Good quality, Rank 2 = Medium and Rank 3 = Bad Quality images. Following is a snippet of my rank calculation.

Image Rank 1 Examples
"blur":0.9660969387755102 "sharpness":0.5067756918923018 "color":0.1462474560577043 "quality":0.8798513615119794 "contrast":0.08234799394638759 "colorContrast":8.328080618896797 "brightness":0.14739056340426107	"blur":0.966530612244898 "sharpness":0.49986904977578045 "color":0.19059606130699966 "quality":0.8476687344507642 "contrast":0.08834899629937157 "colorContrast":25.384664271394108 "brightness":0.24995158193079098	"blur":0.7220025510204081 "sharpness":0.15089643141301262 "color":0.4003952223119604 "quality":0.9609951671164367 "contrast":0.2292792429036637 "colorContrast":32.053295457355276 "brightness":0.4115245849574555
Image Rank 2 Examples
"blur":0.8221428571428572 "sharpness":0.15935783910386503 "color":0.04443692154503701 "quality":0.5085070823398357 "contrast":0.14823618660855284 "colorContrast":26.781796490365295 "brightness":0.27380289202677777	"blur":0.9705484693877551 "sharpness":0.41994123003434913 "color":0.19996393727671855 "quality":0.3153617863913573 "contrast":0.10908353919667178 "colorContrast":43.63807505087183 "brightness":0.18082135350839887	"blur":0.8633290816326531 "sharpness":0.2271008743069167 "color":0.12817043260208896 "quality":0.6680113666484762 "contrast":0.05242860949574584 "colorContrast":44.47249174965709 "brightness":0.222476585047196

From the above results, an simple image ranking system can be automatically performed by estimating the global image features values. More complex algorithms can be used to further improve the image quality estimation process such as Bokeh Estimator which can be used to detect background blur and camera focus[link].
Note: source code to follow soon at my github

Image processing Benchmarks

For this benchmark the following most widely used image processing libraries were considered.

- imagemagick [http://www.imagemagick.org/script/index.php]
- graphicsmagick [http://www.graphicsmagick.org/]
- epeg [https://github.com/mattes/epeg]
- opencv [http://opencv.org/]
- vips

Test environment:
Memory: 5.8 GB
Processor: Intel Xeon CPU W3530 @ 2.80Ghz x 4 Core
OS: Ubuntu 13.04 / 64 bit
Graphics: Gallium 0.4 on AMD Redwood

Original Image - 350KB - 3168x3168 pixels | Resized to 640x480
imagemagick x 3.69 ops/sec ±2.27% (23 runs sampled)
gm x 5.03 ops/sec ±0.68% (29 runs sampled)
opencv x 19.18 ops/sec ±1.27% (49 runs sampled)
epeg x 35.49 ops/sec ±1.16% (60 runs sampled)
vips x 40.62 ops/sec ±5.01% (69 runs sampled)

Original Image - 1 MB - 3000x2000 | Resized to 640x480
imagemagick x 4.97 ops/sec ±2.35% (29 runs sampled)
gm x 5.00 ops/sec ±0.54% (29 runs sampled)
opencv x 15.15 ops/sec ±1.36% (41 runs sampled)
epeg x 27.47 ops/sec ±0.98% (69 runs sampled)
vips x 36.26 ops/sec ±6.05% (89 runs sampled)

Original Image - 15MB - 5382x6254 pixels | Resized to 640x480
imagemagick x 0.87 ops/sec ±1.20% (9 runs sampled)
gm x 0.87 ops/sec ±0.66% (9 runs sampled)
vips x 1.74 ops/sec ±0.43% (13 runs sampled)
opencv x 1.88 ops/sec ±4.09% (9 runs sampled)
epeg x 3.87 ops/sec ±0.78% (14 runs sampled)

From the above results, VIPS seems to be the fastest among all followed by epeg and opencv. But one thing to consider is the features provided vs performance. Libraries such as VIPS and EPEG are more optimized towards image resizing and image cropping while opencv, graphicsmagick and imagemagick provides a slew of image processing and analysis features.

Code snippet for benchmarking: https://gist.github.com/karthik20522/7605083

Smart Thumbnail Cropping

Scaling an image down to a thumbnail size is a common practice when hosting it on websites to reduce page load time and save bandwidth and so on.. But very little has been done in optimization of those thumbnails from a human view-ability point of view. Human view-ability, what? Take a large image where background covers the major part of the image and shrink it down to a thumbnail size (say 192 px) and notice that the details of the image is subdued by the background.

To solve this problem of smart cropping, I am using a variation of descriptors and image processing tricks to extract only the most feature rich part of the image and preserve the aspect ratio while cropping the image. Following are the test results of algorithm used:

Sample 1
Original Thumbnail	Feature Extraction	Cropped Thumbnail
Sample 2
Original Thumbnail	Feature Extraction	Cropped Thumbnail
Sample 3
Original Thumbnail	Feature Extraction	Cropped Thumbnail

So what's in the pipeline:

• Open source the image processing code
• Build a http handler (ASP.NET http handler) for dynamic cropping

Thoughts?

Event Viewer - Image Search

"Event Viewer" is an yet another attempt to visualize images similar to my {re}Search Timeline project.
Demo at : http://karthik20522.github.io/EventViewer



The whole point of this proof-of-concept project is to visualize the images from the perspective of the events rather than just displaying an grid of images. For example, a search on GettyImages.com website basically displays a list of images in a tabular fashion which provides no sense of association between individual images being displayed. But having them grouped together as part of an event provides a sense of association and correlation between images.

Displaying images is always a tricky business. A dominant color filter technique could probably provide an alternate way of scanning thru images as user might be more interested in images of particular color than the fine details of an image.

From a technology standpoint for building this project was nothing special.

• ASP.NET MVC 4 - Razor
• Amazon SQS - for event scrapping from GettyImages
• Connect API for event and image detail lookup
• MongoDB
• Dominant Color Extraction

Source code at: https://github.com/karthik20522/EventViewer

Development Stack and stuff

For the past year or two, I had been dabbling with different technologies, frameworks looking for an ideal combination of frontend, backend and development tools. Following are what I tend to use and recommend for both personal and consultancy projects.
I usually customize most of these open-source projects as per my needs

Project Management

• Asana - https://asana.com

Database

• MongoDB

CMS

• Calipso - http://calip.so/

Bulletin Board

• NodeBB - http://nodebb.org/

Configuration Management

• Etcd - https://github.com/coreos/etcd

Logging and Analysis

• Logstash - http://logstash.net
• Kibana - http://www.elasticsearch.org/overview/kibana
• Log.io - http://logio.org

Search

• ElasticSearch - http://www.elasticsearch.org
• Algolia - http://www.algolia.com

Source Control

• Github – http://github.com

AWS – Amazon Web Services

• SQS – Simple Queue service
• SES – Simple Email Service

Hosting

• Rackspace / Amazon
• Digital Ocean - https://www.digitalocean.com

Image processing

• GraphicsMagick with Node API - http://aheckmann.github.io/gm/

Development Tools/Frameworks/Languages

• .NET
  • ASP.NET MVC
  • ASP.NET WebAPI for REST services - http://www.asp.net/web-api
  • Dapper.NET or Simple.Data for SQL ORM’s and MongoDB CSharp Driver for MongoDB access
  • FluentSecurity for user authentication and roles http://www.fluentsecurity.net/
• NodeJS
  • NodeProxy for proxy and load balancing service
  • Node Cluster for Multi Node process
  • Node Express Boilerplate - https://github.com/mape/node-express-boilerplate

Misc Libraries and websites

• ASP.NET Razor Helpers - http://www.mikepope.com/blog/documents/WebHelpersAPI.html
• OverAPI - http://overapi.com/
• JQuery Plugins - http://www.unheap.com/
• MicroJs - http://microjs.com/

Design and Me

My attempts and my progress of design skills :p

Cutting Board: my entry at my company (GettyImages) internal hackathon

---

Clip The Trip: Mobile

---

61 Degree:



Pad Devices


Mobile

---

Research Timeline:

---

OpenSays.me

---

Year In Images: http://yearinimages.com

---

Headhunter: my entry at my company (GettyImages) internal hackathon

---

GetNitty

Project Frame - How I Built This

The whole idea behind Project Frame was to provide the user the ability to download a part of the video, basically a online video clipper.
Note: I built this just for fun and its just a proof of concept.

Tools used:
- FFMPEG - http://www.ffmpeg.org
- ASP.NET 4.0
- GettyImages Connect API
- JQuery videoBG
- jQuery Wookmark plugin
- jQuery imagesLoaded plugin
- simplePagination.js
- Chardinjs
- jQuery pageSlide
- soundmanager 2
- SLY https://github.com/Darsain/sly

Homepage:



The page is pretty simple page except for the background video. I used videoBG plugin to play video. So how did i build the video? FFMpeg has a way to superimpose multiple video in a grid layout. An example for generating grid layout can be found at https://ffmpeg.org/trac/ffmpeg/wiki/FilteringGuide#multipleinputoverlayin4x4grid. Using this as my starting point, I downloaded 16 videos from GettyImages.com and generated four 4x4 videos and took the final 4 videos and superimposed on top of each other to generate 1 large video with 16videos embedded within in. But off course the final video was gigantic, so I resized the final video to web size.

FFMpeg script code to generate the superimposed video: FFMpeg script code to resize the video:
Search Result Page:

	Move along nothing fancy here. Connect API was used to search videos and jQuery Wookmark in combination with ImageLoader was used to display the result in pinterest style fashion.

Video Edit Page:



Few things happen here when the page is loaded:
1) Download the video file to a temp. directory
2) Generate thumbnails using ffmpeg

	This generates thumbnails of 96x72 dimensions and with files as out{number}.jpg. Folder structure something like the following:

3) There is Video metadata information view for which I used jQuery Imageslide.



Following server side code snippet for generating the image information table which jquery pageslider uses for rendering.


4) Once the user selects the preferred video range, following code is executed on the server side to clip the video using ffmpeg.



Off-course this solution of downloading the video file and generating the thumbnail is not a viable solution for a real world scenario. But we could always pre-process the thumbnails and store the thumbnails are an image sprite (saving as an image sprite would save number of files the OS would have to handle; off-course this sprite technique was ment for real world scenario). Since the entire code based (server side) is all async/TPL based, multiple users using the tool works quite well. But there is always an alternate solution for video processing like Animoto or Amazon Transcoding service.

Google first, Code second and Learn later

In my mind, there are 2 types of programmers, someone who can write code and others who cannot (but offcourse there are 5 types http://stevenbenner.com/2010/07/the-5-types-of-programmers/). Nevertheless how many types there are I am slowly turning out to be programmer who can write code without understanding!! I am not talking about code that I am writing (like business logic) but rather the underlying technologies, frameworks, code snippets that abstracts the implementation. Why?

Well almost every technical challenge may have been encountered by someone else and they probably shared their thoughts and their code to the world in the form of blog post, open source projects, stackoverflow questions etc. Keeping this in mind reinventing the wheel is a waste of time but rather reusing the shared code and modifying to your problem statement is much smarter, faster and better use of your time.

I do believe it is true that best programmers know how to use google pretty well (http://www.reddit.com/r/programming/comments/197n65/the_best_programmers_are_the_quickest_to_google/). Lets take a problem that I had to solve

Problem statement: was to build a imaging library to perform image manipulation (resizing, watermark & captioning) and metadata processing (both read and write from and to the images).

Following is what I ended up doing to get the task done:
Note: I am only discussing about how I went about getting the code from Internet and adjusting to fit my solution.

• Since I was restricted from using libraries like Aforge, ImageMagick for performance reasons, so I started with looking for generic image processing code in C# at google but ended up at koders.com for partial~complete solutions.
• Found few hundred implementations but drilled down to two components: GDI based processing and WIC (Windows Imaging Component) based processing
• Wrote test cases to verify the open source code/snippet met all functional requirements of the problem statement like resizing, watermarking etc
• Once the functionality verified, I reorganized the code to fit the programming standards like naming conventions, reusability etc.
  • Note that I only moved the code around but left the original core code untouched like the real image processing section (calculations, memory handling etc)
• Now that I had image processing part done, I looked for Metadata processing solutions online as we had to write custom metadata (IPTC, XMP) into images
• This metadata took a while since in real world, software's like photoshop etc wrote metadata differently and the code had to be smart enough to read any metadata format.
  • This is the part where I had to learn the open sourced code to figure out rest of the implementation
• Then more testing, testing and testing (unit testing) to verify the implementation and stability of the code
• Once all the business requirement was completed, this was the stage where I got to learn/read thru the original code and as part of refactoring, I rewrote the entire business logic/image processing layer from scratch but with all the edge case considerations and imaging algorithms that the open source code provided.

So how did I fare? Well, if I had to write the image processing code from scratch it would have taken months and probably missed tons of edge cases that would have haunted me if the self-written code went to deal with real world images. To recap, first I google'd already written image processing code that was closest solution to my problem and then I eventually learnt the code logic and finally as part of refactoring strategy, I rewrote the entire code (with my new understanding of the open source code) and verified it by using the test cases that was originally implemented to verify the functionality of the open sourced code.

Is this cheating? YES but when you are trying to solve complex problems like image processing, it is better to pick where other ppl have left off. Smart, maybe! Lazy, most likely.

How about this; what if i asked you to implement a TRIE data structure to search a collection of strings? Would you go about implementing from scratch or would you rather just use an pre-existing implementation? Like this one http://geekyisawesome.blogspot.com/2010/07/c-trie.html? So are you cheating or just being smart?

So my friends, learn to google first and then code. It definitely saved me a ton.

Clipthetrip.com v2 - what to expect

Quite a lot infact,

• Brand new UI design
  Note: following screenshots are very early mocks
  
  

  Homepage	Search Page	Clip Page

• Yelp integration
  
• Completely redone code base - async calls using TPL
• New technologies/frameworks like
  • Fluent Security [ http://www.fluentsecurity.net/ ]
  • ElasticSearch [http://www.elasticsearch.org/]
  • Semantic Keyword Extraction [http://developer.yahoo.com/contentanalysis/]
  • SOA based architecture
  • Webapi. Completely rest based implementation
  • Knockoutjs [http://knockoutjs.com/]
• Mobile version of the site and possibly a native mobile app for v2.1

Don't be the 1 Percenters of programming

Don't be or atleast try to be the 1 Percenters of programming but rather be the 3 to 5 Percenters. Who the hell are these guys? Well from my perspective,

1 Percenters are the elite programmers who end up building stuff like databases, operating systems, programming languages that we program on, writing code for satellites etc etc you know the best of the best type. Think about this, it is said there are 9 million Java programmers but probably few hundreds if not thousands work on building the Java language or few hundred that end up building database engines like neo4j, lucene etc. Only the elites.

2 Percenters I believe are the guys/girls who understand the various components that our 1 Percenters build. These guys are the architects, principle engineers of sorts.. The high level understanding type of guys.

3 to 5 Percenters are the guys you and me should strive to be. Why? I believe these are the programmers who get stuff done. Duck tape programmers, polyglot programmers of sorts and most importantly programmers with imagination. If you were do a startup, you probably would be first looking for programmers who can hack things together to get stuff done. I am quite sure Mark Zuckerberg would have probably got duck tape programmers during the initial stages of Facebook and once FB blew up that’s when he might have started poaching the elites to build custom databases and server farms etc.

So what about the rest of the 95 Percenters, well code monkeys! So what about me!! Well aggressively perusing the 5 Percenter status :)

Evolution of my Resume - Karthik Srinivasan

My Resume over the years:

NOTE: I AM ONLY DISPLAYING PAGE 1 OF MY RESUMES

First there was this: college days



Then there was this: while looking to move from first job (Albany International)



Then there was this: while looking to move from second job (Travelocity)

Single page resume	Two page resume (second page isn't shown)

Now there is this coded resume as a full time programmer (not planning to move anymore thou, but keeping resume updated is probably good)


Page 2 is not shown

ClipTheTrip.com - How I Built This

Clipthetrip.com is a fun, free crowd sourced destination guides. From a data modeling perspective it has a pretty simple object interaction. There is Location object, there is CLIP object, there is photo object and a user object. MongoDB was used as a data store so objects were quite denormalized instead of a referential model.

Some of the technologies and API used for building the site is as follows:

• MongoDB
• .NET MVC3.0
• .NET 4.0 web optimization library for Js and CSS minimizing and combining
• jQuery Plugins
  • Wookmark
  • jQuery Lightbox
  • qTips
  • GMap3 for google maps
  • jQuery Fullscreen for homepage background
  • Imagesloaded
  • Autosuggest for location suggestion
  • Plupload for uploading images
  • Tablesorter
  • CookieControl from the pesky Europe cookie law
  • jQuery Reject
• External API's
  • Weather information using Accuweather
  • Currency using Webservicex service
  • Business search using Bing
• Image processing using SimpleDotImage library
• Generating offline PDF's using wkhtmltopdf
• HtmlAgilityPack for screen scrapping
• Amazon SES for email

As part of this article I am sharing code snippets of how I extracted information from external API's and other smaller code snippets. Since most part of the site is just CRUD operations, I feel it would be a waste to explain the code structure or architecture (considering architecture is quite simple!)

Currency data from Webservicex.net
Example page: http://www.clipthetrip.com/city/london
Business information from Bing
Weather information from Accuweather
Example page: http://www.clipthetrip.com/city/london
Image Resizing using SimpleDotImage
Javascript and CSS bundling using Web.Optimization

{Re}Search timeline - How I Built This

Demo site coming soon..

Another proof of concept and another interesting way of viewing images. Checkout my previous yearinimages.com post. The idea behind Yearinimages was to provide a very eye candy way of representing the best images of 2011. But unfortunately mosaic layout provides no sense of navigation. {Re}search project was my attempt to avoid this pitfall of navigation while still holding the visual way of presenting data with images.



The idea behind this POC is to provide the user a timeline based navigation of events. For example, when you search Japanese Earthquake or Oscar awards which when viewed as a timeline of events, would provide more visual perspective than just displaying a grid of images.

So how did I build this? Well, I used Gettyimages connect API to extract event and image information and Timeline plugin by verite for building the timeline. Addition to above api's and plugins there were quite a few hacks that was required to make this happen. Following is how I did it:

Extracting the event information:
Thou gettyimages connect API provides the search by personality , keyword and search phrases unfortunately it only provides limited refinement options on events. Basically it's not possible to get a list of events similar to the one the gettyimages website provides like following screenshot:



I do believe that the refinement can be customized per customer basis with connect API but I am not sure. Anyways, for extracting the list of events for a particular search I screen scrapped from gettyimages.com website. Following is code snippet of screen scraping using WATIN ui automation framework:



Now that we have list of event ids screen scrapped we can use gettyimages connect to get event details for displaying on UI. Following is code snippet to call connect API using my connect API library (its nothing but a helper library to call the connect API rest service)



Setting up timeline ui:
Now that we have list of events we need to convert it to the format the timeline plugin expects. Timeline plugin takes json as its data source. I used json2csharp.com site to convert the sample json format that timeline github page provided to c# data model that can be used code behind. I used json.net to convert the timeline c# object to json string. Code snippet for converting event list to timeline object.



Note that the hard-coded html in timeline.text property is to provide a autosuggest capability for user. The code above is just for demo purpose's to show that timeline plugin is capable of taking in Html as part of it's json data. Html rendering of the above code:



At my 1st iteration, I hooked up the events to default timeline functionality. But what I would like is to allow users to click on an event and display all images associated to that event. Since this is a custom functionality some hacks to timeline code had to be done. Like the following:
- Associating eventID to the event timeline div so when timeline is clicked the code can determine which eventID it's referencing too
- Adding a Ajax loading gif to every event
- Upon sliding to an event, do Ajax call to server to get images associated to that event
- Display the images when Ajax call is completed
- Show image information when clicked

Framework, plugins, API used:
- jQuery 1.7.1
- jQuery light box
- wookmark jQuery plugin
- Timeline by Verite.Co
- .net framework 4.0/mvc3
- Newtonsoft json.net
- WATIN ui framework
- json2csharp.com for converting the template json to c# object
- gettyimages connect API