Google+ chose to Auto Enhance this prosaic shot of potential firewood stacked in the garden.

Following on from recent posts about online software that will automatically arrange your photographs, I’ve been finding what else is going on out there, and discoveries include some odd features as well as the next stage of automation – story construction. This won’t be news for the tech savvy but there may be a few people reading this post who share the same low level of technical comprehension that I have and might find these topics entertaining, if not useful.

What intrigues me about this subject is that somewhere behind all the algorithms there must be teams of human beings involved in identifying and agreeing criteria on which the code is then based. I know little about how this process works (pleased to hear some explanations) and the only personal experience that has any relationship to this goes back to 2000 when I collaborated briefly with a computer science researcher who, for his PhD thesis was working in the field of metadata for image databases and was investigating classification, indexing and retrieval of images.

I’d accumulated several hundred images on colour negative film after photographing a small woodland over an entire year, and (incidental to my aim) found I’d created an archive of the place. This body of images lent itself well to the research so I scanned all the photographs, logged the dates and the locations in the wood of where each image was taken. Then working towards the image recognition and retrieval goal we explored ways to classify and categorise the pictures on the basis of their formal properties – eg close-up or long shot; colour and tonal factors; recurring textures such as water or leaves; strong vertical or horizontal elements etc.

All this info was entered into a database and became, I suppose, a manually created version of, and prequel to the EXIF data which emerged a couple of years later and is now embedded into every digital image. Experiencing a completely different approach to assessing images provided me with a valuable new tool but for various reasons we didn’t take the research much further. However, zip ahead to 2014 and as far as I understand many of the strands of that technology are now omnipresent and used, often unwittingly, whenever you post photos to Facebook, flickr or Google+.

So somewhere, in California, or China, or maybe even Bristol, I’m guessing there are people discussing what the ideal texture of skin should look like in order to create an algorithm that will automatically enhance your girlfriend’s/husband’s spotty/wrinkly face into the smooth blandness of a Barbie/Ken doll for Google+; or they’ve assessed hundreds of perfectly lit mountain/lake/beach scenes to arrive at a solution to provide some zing for your poorly exposed holiday photos. And someone’s created an presumptuous bit of code to decide which of your photos has the merits to be included under the ‘highlights’ heading.

Here are a few of the discoveries I’ve made about how algorithms and photographs are getting together (I’ll move on to the story element in a future post) but first, the derivation of the word itself is pretty interesting; according to Wikipedia it comes from either Algaurizin or Algoritmi  which are Latin versions of the name of 2ndC CE Persian mathematician, astronomer and geographer al-Khwārizmī, who, incidentally, was an orthodox Muslim who also wrote a treatise on the Jewish calendar.

So, what might al-Khwārizmī have made of Google’s Auto Awesome algorithms? If you use an Android phone (maybe it happens with iPhones too) and Google+, you can enable auto-back up and once they’ve uploaded your photos are then subject to (though you can turn it off) all manner of ‘enhancements’ – Google explain the feature like this :

“Sometimes we’ll create a brand new image based on a set of photos in your library. For example: if you upload a sequence of photos, we’ll try and animate them automatically. Or if you send us a few family portraits, we’ll find everyone’s best smile, and stitch them together into a single shot. Likewise with panoramas, filmstrips, and a whole lot more. We call these kinds of enhancements Auto Awesome.”

‘Best smile’ ? Creating creepily homogeneous happy family memories – will the one person who just won’t smile in any of the group photos be eradicated? Or will someone else’s smile be borrowed and pasted on? It’s tempting to play with and try to subvert the feature but all I’ve tried so far is taking a sequence of photos of the morning glory flowers in our greenhouse, rotating the phone at each shot, and, as I anticipated, a giff was automatically created from them.

Google+ also decided to make a dull panorama for me from some indifferent shots I’d taken on my phone simply to record that after 15 years of neglect grass mowing was happening in nearby fields.

Auto Enhance is another feature you’ll find on Google+ (see photo pair at top of page), here’s a bit about it from Social Times:

‘Auto Enhance – Imagine Google’s new enhancement feature as a filter you can turn on or off (or as Picassa with a different name). Don’t like wrinkles? Keep the enhancements on. Just simply uploading your photos to Google will get you better brightness, contrast, saturation, and much more.  The changes are made with special algorithms that can detect faces, eyes and skin. No more wrinkles, blemishes, or red eyes. No Photoshop skills required.’

And there’s Auto Highlight

“Sifting through vacation photos to assemble the perfect album can take hours. Auto Highlight helps you find your favorites faster by de-emphasizing duplicates, blurry images and poor exposures, and focusing instead on pictures with the people you care about, landmarks, and other positive attributes. Simply visit the Photos page, and you’ll see your Highlights ready to share.”

What might “Positive attributes” be, who’s decided they are either ‘positive’ or ‘attributes’? Google also offer occasional, time-limited ‘special’ enhancements that include the option to have your group photos celebrity bombed or to have a national flag of your choice painted on the face of someone in your photo; what fun – no chance of that being contentious is there?

Then there are people who work for flickr on developing the ‘Interestingness’ algorithm, the following is from the FAQs page of Big Huge Labs:

‘Photos are automatically selected by computer according to a secret algorithm called Interestingness. Interestingness is what Flickr calls the criteria used for selecting which photos are shown in Explore. All photos are given an Interestingness “score” that can also be used to sort any image search on Flickr. The top 500 photos ranked by Interestingness are shown in Explore. Interestingness rankings are calculated automatically by a secret computer algorithm. The algorithm is often referred to by name as the Interestingness algorithm. Although the algorithm is secret, Flickr has stated that many factors go into calculating Interestingness including: a photo’s tags, how many groups the photo is in, views, favorites, where click-throughs are coming from, who comments on a photo and when, and more. The velocity of any of those components is a key factor. For example, getting 20 comments in an hour counts much higher than getting 20 comments in a week.

Lots of secrets then, but a few clues emerge about the factors involved although it’s more to do with data mining the context of the photo than about the content. But here’s a novel idea that’s very much about the content of the image. Science Daily report that a team from Brown University have developed a computer algorithm that:

‘enables users to instantly change the weather, time of day, season, or other features in outdoor photos with simple text commands. Machine learning and a clever database make it possible. To start the project, James Hays, Manning Assistant Professor of Computer Science at Brown University, and his team defined a list of transient attributes that users might want to edit. They settled on 40 attributes that range from the simple – cloudy, sunny, snowy, rainy, or foggy – to the subjective – gloomy, bright, sentimental, mysterious, or calm.
The next step was to teach the algorithm what these attributes look like. To do that, the researchers compiled a database consisting of thousands of photos taken by 101 stationary webcams around the world. The cameras took pictures of the same scenes in varying of conditions – different times of day, different seasons and in all kinds of weather.

The researchers then asked workers on Mechanical Turk – a crowdsourcing marketplace operated by Amazon – to annotate more than 8,000 photos according to which of the 40 attributes are present in each. Those annotated photos were then fed through a machine learning algorithm.

Four Seasons. A new algorithm developed by Brown computer scientists allows users to change the season and other “transient attributes” in outdoor photos.
Credit: Image courtesy of Brown University

“Now the computer has data to learn what it means to be sunset or what it means to be summer or what it means to be rainy – or at least what it means to be perceived as being those things,” Hays explained.

Once an entire year has been recreated from one of your photos, a celebrity or two inserted, smiles reconfigured, sequences animated and stitched together and your collection of images has been edited to ‘highlights’ it looks like some kind of story could emerge largely by itself too – that’s the next post.