Automatic Clustering of Geo-tagged Images. Part 1: using multi-dimensional histograms

There's a lot of geo-tagged images on the web. Sometimes the image coordinate is slightly wrong, or the scene isn't quite what you expect. It's therefore useful to have a way to automatically download images from a given place (specified by a coordinate) from the web, and automatically classify them according to content/scene.

In this post I describe a pythonic way to:
1) automatically download images based on input coordinate (lat, long)
2) extract a set features from each image
3) classify each image into groups
4) display the results as a dendrogram

This first example, 2) is achieved using a very simple means, namely the image histogram of image values. This doesn't take into account any texture similarities or connected components, etc. Nonetheless, it does a reasonably good job at classifying the images into a number of connected groups, as we shall see.

In subsequent posts I'm going to play with different ways to cluster images based on similarity, so watch this space!

User inputs: give a lat and long of a location, a path where you want the geo-tagged photos, and the number of images to download

	#I tried this out on Horseshoe Bend, a much-photographed site in Glen Canyon, AZ. The approximate coordinates are
	lt=36.879444
	ln= -111.513889
	#I wanted to limit the analysis to 37 images:
	numimages=37
	#Here I'm just setting the path to be the present working directory:
	path=os.getcwd()
	#Get the images from panoramio. I've hard-wired it to search for images within lt + lt+0.1 and ln + ln+0.01, but you can specify any search radius you like.
	url = "http://www.panoramio.com/map/get_panoramas.php?order=popularity&set=public&from=0&to=%s&minx=%s&miny=%s&maxx=%s&maxy=%s&size=medium" % (str(numimages),str(ln),str(lt),str(ln+.01),str(lt+.1))

view raw geotag.py hosted with ❤ by GitHub

First, import the libraries you'll need. Then interrogates the website and downloads the images.

	try: import simplejson as json
	except ImportError: import json
	import os
	import urllib, urlparse
	import glob
	import numpy as np
	import Image
	from itertools import combinations
	import simplejson as json
	import pylab as mpl
	c = urllib.urlopen(url)
	j =json.loads(c.read())
	imurls=[]
	for im in j['photos']:
	imurls.append(im['photo_file_url'])
	for url in imurls:
	image = urllib.URLopener()
	image.retrieve(url,os.path.basename(urlparse.urlparse(url).path))

view raw auto-class 1.py hosted with ❤ by GitHub

As you can see the resulting images are a mixed bag. There's images of the river bend, the road, the desert, Lake Powell and other random stuff. My task is to automatically classify these images so it's easier to pull out just the images of the river bend

The following compiles a list of these images. The last part is to sort which is not necessary but doing so converts the list into a numpy array which is.  Then clustering of the images is achieved using Jan Erik Solem's rather wonderful book 'Programming Computer Vision with Python'. The examples from which can be downloaded here. Download this one, then this bit of code does the clustering:

	imlist=[]
	for infile in glob.glob( os.path.join(path, '*.jpg') ):
	imlist.append(infile)
	imlist=np.sort(imlist)
	features = np.zeros([len(imlist),512])
	for i,f in enumerate(imlist):
	im = np.array(Image.open(f))
	h, edges = np.histogramdd(im.reshape(-1,3),8,normed=True, range=[(0,255),(0,255),(0,255)])
	features[i] = h.flatten()
	import hcluster
	tree = hcluster(features)

view raw auto-class 2.py hosted with ❤ by GitHub

The approach taken is to use hierarchical clustering using a simple euclidean distance function. This bit of code does the dendogram plot of the images:

draw_dendrogram(tree,imlist,filename='horseshoe.png')

view raw dendro.py hosted with ❤ by GitHub

which in my case looks like this (click on the image to see the full extent):

It's a little small, but you can just about see (if you scroll to the right) that it does a good job at clustering images of the river bend which look similar. The single-clustered, or really un-clustered, images to the left are those of the rim, road, side walls, etc which don't look anything like the river bend. 

Next, reorder the image list in terms of similarity distance (which increases in both the x and y directions of the dendrogram above)

	order = tree.get_cluster_elements()
	slist=imlist[order]
	#If i'm only interested in images of the river bend, you can see at a glance that I can remove the final cluster of 4 images, and then take the last 2 sub-clusters (comprising 10 images) so this gives me those images:
	a=4
	slist=slist[:len(order)-a]
	a=10
	slist=slist[-a:]
	#Finally, I create a plot of only those final 2 clusters
	mpl.figure()
	for i in range(10):
	im=Image.open(slist[i])
	mpl.subplot(2,5,i+1)
	mpl.imshow(np.array(im))
	mpl.axis('equal')
	mpl.axis('off')
	mpl.savefig('riverbend.png')
	mpl.close()

view raw geotag2.py hosted with ❤ by GitHub

Which gives me:

As you can see they are all images of the river bend, except the 2nd from the left on the top row, which is a picture of a shrub. Interestingly, the pattern of a circular shrub surrounded by a strip of sand is visually similar to the horse shoe bend!!

However, we don't want to include it with images of the river, which is why a more sophisticated method that image histogram is required to classify and group similar image ... the subject of a later post.