Automatically Cropping Images is Hard
Monday, 21st October 2013, 19:00
Over at MyReviewer.com, on the home pages we use automatically cropped images to promote reviews, news items and articles. Layout wise everything works well enough, the images are done with CSS backgrounds and always have a fixed height, but a variable width that always makes the image fill the box.
Obviously this is achieved with some JavaScript that adds and removes columns as space becomes available or goes away. The actual images are wide enough to accommodate these boxes at their maximum width, it's not complicated, or rocket surgery as the old World of Warcraft joke goes.
On the server side, images are cropped to this special thumbnail size in very simplistic ways, re-sized to match the correct width, the tops and bottoms are just chopped off. Which leads to a problem I've been trying to address automatically for a while, so many images on MyReviewer are Blu-ray or DVD covers and these are portrait squares being banged into a landscape thumbnail hole.
Occasionally, things all work out and this kind of automation works just fine, like in the following example of The Rocketeer:
Quite often though, it doesn't, Looking for Hortense looks okay, but at the same time offers nothing visual of any real interest or specifically instantly recognisable from the original cover.
And as for poor Alan Davies, typical of many covers which contain a big title at the top, all that is left of the QI regular is his hair.
Solution Options
I've been considering a few options for a while, the first has been a very unfavourable but probably solid enough one, when uploading get the up-loader to drag around the image in some kind of UI and position it nicely for promos. But since almost any image can potentially be a promo if it turns up in the right places, that is a lot of work to expect from our review team.
Another is to do allow some way of fixing ones that look bad, but again that requires a level of interaction, a UI to do it, and someone to notice it on the homepage before too many visitors have been put off by the eyesore.
Both have smelt like a lot of work, and require too much user interaction for my tastes. I've really wanted to add some kind of automated solution, one that would work most of the time adequately without any extra effort on anyone's part.
The avenue I've been occasionally googling to see if something out there would allow me to do this, has been face recognition. The idea is quite simple, scan an image for faces, then try to fit them as pleasantly as possible inside the cropping area.
Another option I also considered was the sort of dynamic resizing you can do in Photoshop, where it removes parts of the image in order of least interest, but having tried that on a few covers it just looked awful.
Face Recognition With node.js
I have no idea why I never found this before, but the open source centre of the object recognition universe appears to be OpenCV, and once I found that out it wasn't long before I came across bindings for this library that work with node.js, courtesy of Peter Braden. And the first example he includes in the read-me is none other than exactly what I was after.
Off to experiment I shoe-horned this into my image delivery server, firstly using the default face detection rule-set. Now I'm no expert on this, so forgive me if I get things wrong here. Everything seems to revolve around Haar Cascade files, which are rule-sets for detecting Haar features.
The default one seems to detect some odd things as faces, totally miss a few of them, and get a few right. So it's a good place to start, but alas not good enough for my purposes. Included in node-opencv are various data files for other rule-sets that I gave a go too, and I found that the ones for detecting eyes worked surprisingly well with very little false positives.
So I decided to combine the two, and here are examples of Alan, Kristen, Doctor Who and the scary lady in Carved. When the face detection misses, the eye detection seems to pick up the slack.
You can see with Looking for Hortense, that it has picked out the eyes of Kristin Scott Thomas, and both an eye from the larger Jean-Pierre Bacri, and his face in the full body smaller image.
Carved lady isn't detected at all with the face recognition, but one of her eyes is picked up making that better than nothing.
The three new Doctor Whos are only detected as faces, with Matt Smith not even being picked up at all. I did find one of the rule-sets picked up all three, but unfortunately it had so many false-positives on other images it was a no go.
Finally Alan not only gets his face picked up but also his left eye. All in all a lot more accurate than just cropping the centre to make a thumbnail.
Logic wise, the cropping algorithm goes like this:
- If there are no faces or eyes, just centre as before
- If there is just one face or eye, centre it inside the cropped area
- If there are more, try and fit all of them within the cropped area
- If you can't fit them all, find the largest feature and centre that
The results work quite well, so I'm gently pleased for the moment anyway with the improvement. Here is a before and after of the home page showing the promo images, many are just the same as before, but of those that have been changed, most of them are for the better.
The above is before, below is after.
Future Work
Pattern recognition like this really needs training with example images, unfortunately this is a lot of work so I can't really justify doing it at present. But in the future, maybe I'll build my own recognition data-sets to scan and build a rule-set from.
I may also just check every few months to see if any rule-sets from other people are working better than the default ones I'm using. Until then, what we have now is at least an improvement.