About Me

Curriculum Vitae

A brief list of my current skill set

Bloggybits

Automatically Cropping Images is Hard
Monday, 21st October 2013, 19:00

But maybe we can use face detection?

The Git Cheat Sheet
Friday, 6th September 2013, 11:30

for github, bitbucket, that kinda stuff

CoffeeScript and TypeScript are Stupid
Saturday, 17th August 2013, 11:21

Don't use them!

Changing the Order of the jQuery Event Queue
Wednesday, 3rd July 2013, 20:27

It's just a push to the left

How Do Spammers Get My Email Address?
Wednesday, 15th May 2013, 18:03

I think these days I have a pretty good idea

XSLT, node.js 0.10 and a Fun Two Days of Native Modules and Memory Leaks
Thursday, 25th April 2013, 17:14

documentation makes things less cryptic, so lets not write much of it

Fixing CentOS high cpu usage when running as a virtual machine under VirtualBox
Sunday, 21st April 2013, 20:28

innotek rocks! I mean Sun... I mean Oracle...

Repairing a dK'Tronics Keyboard and Scoping Out a ZX Spectrum 48k - Part One
Sunday, 17th March 2013, 23:51

What signals inside of it actually look like

Projects and Sillyness

MAME Cabinet Diary

How I built my own arcade cabinet

Loading Screen Simulator

I don't miss the ZX Spectrum, I still use it!

The Little Guy Chat Room

It's a Pitfall inspired chat room

GPMad MP3

A fully featured MP3 player what I wrote

GP Space Invaders

My first little emulator

GP32 Development Page

Some info and links about this cute little handheld

Disney Nasties

Uncensored images, you must be 18 to view them

Minna's World

The cuttest fluffy puppy in the world

Diary of a Hamster

Learn about how hamsters think, first hand

Utilities

Time Calculator

A simple little online utility for working out how many hours to bill a client

A Few Links

Automatically Cropping Images is Hard
Monday, 21st October 2013, 19:00

Over at MyReviewer.com, on the home pages we use automatically cropped images to promote reviews, news items and articles. Layout wise everything works well enough, the images are done with CSS backgrounds and always have a fixed height, but a variable width that always makes the image fill the box.

Obviously this is achieved with some JavaScript that adds and removes columns as space becomes available or goes away. The actual images are wide enough to accommodate these boxes at their maximum width, it's not complicated, or rocket surgery as the old World of Warcraft joke goes.

On the server side, images are cropped to this special thumbnail size in very simplistic ways, re-sized to match the correct width, the tops and bottoms are just chopped off. Which leads to a problem I've been trying to address automatically for a while, so many images on MyReviewer are Blu-ray or DVD covers and these are portrait squares being banged into a landscape thumbnail hole.

Occasionally, things all work out and this kind of automation works just fine, like in the following example of The Rocketeer:

Quite often though, it doesn't, Looking for Hortense looks okay, but at the same time offers nothing visual of any real interest or specifically instantly recognisable from the original cover.

And as for poor Alan Davies, typical of many covers which contain a big title at the top, all that is left of the QI regular is his hair.

Solution Options

I've been considering a few options for a while, the first has been a very unfavourable but probably solid enough one, when uploading get the up-loader to drag around the image in some kind of UI and position it nicely for promos. But since almost any image can potentially be a promo if it turns up in the right places, that is a lot of work to expect from our review team.

Another is to do allow some way of fixing ones that look bad, but again that requires a level of interaction, a UI to do it, and someone to notice it on the homepage before too many visitors have been put off by the eyesore.

Both have smelt like a lot of work, and require too much user interaction for my tastes. I've really wanted to add some kind of automated solution, one that would work most of the time adequately without any extra effort on anyone's part.

The avenue I've been occasionally googling to see if something out there would allow me to do this, has been face recognition. The idea is quite simple, scan an image for faces, then try to fit them as pleasantly as possible inside the cropping area.

Another option I also considered was the sort of dynamic resizing you can do in Photoshop, where it removes parts of the image in order of least interest, but having tried that on a few covers it just looked awful.

Face Recognition With node.js

I have no idea why I never found this before, but the open source centre of the object recognition universe appears to be OpenCV, and once I found that out it wasn't long before I came across bindings for this library that work with node.js, courtesy of Peter Braden. And the first example he includes in the read-me is none other than exactly what I was after.

Off to experiment I shoe-horned this into my image delivery server, firstly using the default face detection rule-set. Now I'm no expert on this, so forgive me if I get things wrong here. Everything seems to revolve around Haar Cascade files, which are rule-sets for detecting Haar features.

The default one seems to detect some odd things as faces, totally miss a few of them, and get a few right. So it's a good place to start, but alas not good enough for my purposes. Included in node-opencv are various data files for other rule-sets that I gave a go too, and I found that the ones for detecting eyes worked surprisingly well with very little false positives.

So I decided to combine the two, and here are examples of Alan, Kristen, Doctor Who and the scary lady in Carved. When the face detection misses, the eye detection seems to pick up the slack.

You can see with Looking for Hortense, that it has picked out the eyes of Kristin Scott Thomas, and both an eye from the larger Jean-Pierre Bacri, and his face in the full body smaller image.

Carved lady isn't detected at all with the face recognition, but one of her eyes is picked up making that better than nothing.

The three new Doctor Whos are only detected as faces, with Matt Smith not even being picked up at all. I did find one of the rule-sets picked up all three, but unfortunately it had so many false-positives on other images it was a no go.

Finally Alan not only gets his face picked up but also his left eye. All in all a lot more accurate than just cropping the centre to make a thumbnail.

Logic wise, the cropping algorithm goes like this:

  1. If there are no faces or eyes, just centre as before
  2. If there is just one face or eye, centre it inside the cropped area
  3. If there are more, try and fit all of them within the cropped area
  4. If you can't fit them all, find the largest feature and centre that

The results work quite well, so I'm gently pleased for the moment anyway with the improvement. Here is a before and after of the home page showing the promo images, many are just the same as before, but of those that have been changed, most of them are for the better.

The above is before, below is after.

Future Work

Pattern recognition like this really needs training with example images, unfortunately this is a lot of work so I can't really justify doing it at present. But in the future, maybe I'll build my own recognition data-sets to scan and build a rule-set from.

I may also just check every few months to see if any rule-sets from other people are working better than the default ones I'm using. Until then, what we have now is at least an improvement.

Comments

Add Your Own Comment