Craigslist Part 2: using matlab to plot your search results in Google Maps

In 'Craigslist Part 1' I demonstrated a way to use matlab to automatically search craigslist for properties with certain attributes. In this post I show how to use matlab and python to create a kml file for plotting the results in Google Earth. Download the matlab googleearth toolbox from here Add google earth toolbox to path. Import search results data and get map links (location strings). Loop through each map location and string the address from the url, then geocode to obtain coordinates. The function 'geocode' writes and executes a python script to geocode the addresses (turn the address strings into longitude and latitude coordinates). The python module may be downloaded here Once we have the coordinates, we then need to get rid of nans and outliers (badly converted coordinates due to unreadable address strings). Use the google earth toolbox to build the kml file. Finally, run google earth and open the kml file using a system command: The above is a simple scatter plot which only shows the location of the properties and not any information about them. Next shows a more complicated example where the points are plotted with labels (the asking price) and text details (the google map links) in pop-up boxes First each coordinate pair is packaged with the map and name tags. Concatenate the strings for each coordinate and make a kml file. Finally, run google earth and open the kml file using a system command:

Broadcom wireless on Ubuntu 12.04

I just upgraded to Ubuntu 12.04 and my broadcom wireless card was not being picked up. It took me some time to figure out how to do this, and the forums give a lot of tips which didn't work for me, so I thought I'd give it a quick mention.

Then disconnect your wired connection and reboot. I found it is crucial that you disconnect any wired connections.

 Upon reboot, go to System Settings - Hardware - Additional Drivers

 It should then pick up the broadcom proprietary driver. Install it, then reboot, and you should be back up and running!

Craigslist Part 1: Using matlab to search according to your criteria and retrieve posting urls

I have a certain fondness for using matlab for obtaining data from websites. It simply involves understanding the way in which urls are constructed and a healthy does of regexp. In this example, I've written a matlab script to search craiglist for rental properties in Flagstaff with the following criteria: 1) 2 bedrooms 2) has pictures 3) within the price range 500 to 1250 dollars a month 4) allows pets 5) are not part of a 'community' housing development. This script will get their urls, prices, and google maps links, and write the results to a csv file. First, get the url and find indices of the hyperlinks only, then get only the indices of urls which are properties. Then get only the indices of urls which have a price in the advert description; find the prices; strip the advert urls from the string; sort the prices lowest to highest; and get rid of urls which mention a community housing development. Finally, get the links to google maps in the adverts if there are some, and write the filtered urls, maps, and prices to csv file. Save and execute in matlab, or perhaps add this line to your crontab for a daily update!

/usr/local/MATLAB/R2011b/bin/matlab -nodesktop -nosplash -r "craigslist_search;quit;"

Using cron to backup Thunderbird emails

My bash script (which I've saved as backup_emails.sh in my home folder and made executable) looks a little like this:

 set -x   
  x=`date +%y%m%d.%H%M`   
  tar zcf thunderb-mail-${x}.tgz ~/.mozilla-thunderbird   
  mv thunderb-mail-${x}.tgz /media/MY_HARDDRIVE/email_backups/   

Then invoke in cron using:

 crontab -e   

add something like:

 30 4 * * * ~/./backup_emails.sh  

which does the backup at 04.30 every day

Tweetdeck on ubuntu 64-bit

Right, the whole world's into twitter. The website is a little difficult to manage when you have a lot of people to follow. The software solution (isn't there always one!) is TweetDeck - unfortunately it runs under Adobe Air, which (surprise, surprise) is available for Linux but 64-bit architectures. So your computer requires a little bit of persuading ...

... you need a few libraries - issue the following commands at the terminal:

$ sudo apt-get lib32asound2 lib32gcc1 lib32ncurses5 lib32stdc++6 lib32z1 libc6 libc6-i386
$ sudo getlibs -l libnss3.so.1d
$ sudo getlibs -l libnssutil3.so.1d
$ sudo getlibs -l libsmime3.so.1d
$ sudo getlibs -l libssl3.so.1d
$ sudo getlibs -l libnspr4.so.0d
$ sudo getlibs -l libplc4.so.0d
$ sudo getlibs -l libplds4.so.0d
$ sudo getlibs -l libgnome-keyring.so
$ sudo getlibs -l libgnome-keyring.so.0
$ sudo getlibs -l libgnome-keyring.so.0.1.1
$ sudo ldconfig

Then install Adobe Air from here

download, then move to your home directory,

right click - properties - make executable

and type

$ chmod +x AdobeAIRInstaller.bin
$ sudo ./AdobeAIRInstaller.bin

Then grab the 'air' file from here, and install using Adobe Air

Happy tweeting!

Youtube karaoke

Ok - here's how to do karaoke to your favourite video track, and publish the results on your blog

First, you need to get the video. Youtube will have it, guaranteed. There's a neat little linux package you can install to download youtube videos:

$ sudo apt-get install youtube-dl

Then go to youtube, and find the url. I chose 'people everyday' by arrested development - an oldy, but a classic, and its something I knew the words to. Then get it:

$ youtube-dl http://uk.youtube.com/watch?v=r9OjoPskf_c

It will download a 'flv' (flash) format video called the last part of the url - in this case 'r9OjoPskf_c.flv'

This next part isn't strictly necessary, but for ease of future access you may want to rename it, and also get it into some format understood by pretty much everything, e.g:

$ ffmpeg -i r9OjoPskf_c.flv people_everyday.avi

Then you need to record your soundtrack. I found the easiest way to do this was to use a linux tool called 'sound recorder'. Applications - Sound/Video - Sound Recorder. just open a new file and press record - simple!

Oh, but its a good idea to open your video exactly when you press 'record' - to get background and timings

Then when you finish press 'stop'. save it - e.g. 'people_everyday.mp3'

Then use mencoder to mux the mp3 and avi file together, like so:

$ mencoder people_everyday.avi -o output.avi -ovc copy -oac copy -audiofile people_everyday.mp3

Your output file 'output.avi' is ready to go onto your blog - like so!

Happy thanksgiving!

The ups and downs of skyping with Linux

Originally posted in The Daily Daniel
----------------------------------------

Having recently moved to the USA, my need to communicate cheaply with people back home has meant that I have subscribed to Skype which allows me to call people through computers for free, or to landline or mobile phones at discounted rate.

I use linux because my computer is my toy and, in many respects, my livelihood. Linux is a much more fun and much more powerful operating system because it grants you, at least in principle, more control over the operation of your computer (which is usually unseen and unreachable on windows systems) through commands made in a terminal. Scientific computing is easier and much wider in scope if you know a few basic linux commands. Almost everything in linux is obfuscating, but learning its ways is enormously satisfying. All linux computing is completely free, as it should be.

Being a linux user, however, means that new software like skype are either generally unavailable because they are proprietary, or because the software developers don't choose to make them available to linux users because they represent a small fraction of computer users. Not only was I very glad to hear, then, that skype was available for linux, but that it even dealt with my 64-bit architecture (which has given me all sorts of problems installing drivers and other software, but I won't go into that ...). So I installed skype with no problems:

$ sudo su | sudo apt-get install skype

And it worked fine. So the next thing I did was buy a webcam. I had one delivered through amazon. None of the cheap ones said they were linux compatible, but i through caution to the wind, and got myself one for 31 dollars.

Of course, it came with a cd with windows drivers, and nothing else. After a couple of hours trawling through the web I sort-of semi installed the drivers for it, using the usual linux user forums. Initially, I couldn't get anything to work properly with it - I tried numerous programs (cheese, camorama, XawTV, MoviePlayer, VLC ...). Eventually I got VLC and camorama do work with it. Don't even ask me how now - i remember just applying my usual tenacity to it, and slamming my monkey fists into the keyboard until i hit the right combination of letters and symbols which would tell the program I had a camera, where it was, and what drivers it needed. It probably looked a little something like this (or equally as hideous)

$ sudo jhas -jh 837afm$^&qeiuhn>>KOUUIG4n we8f-&hcjku8hujbn ok?

Right, so after I wiped the blood of the monitor I turned to skype. Crap - skype for linux ''currently doesn't support webcams'' -aaaargh!!! Ok, fine, I can still call and I can text message? No. Skype for linux "currently doesn't support sms messaging". Great - so I can't even txt message. All windows and mac skype users can txt message. Macs run on a LINUX KERNEL and they can txt message!!! The skype developers obviously just cant be bothered to write the code necessary. Or maybe they think that all linux users only have nerdy friends, who are all messaging each other over the internet because thats where they spent 99% of their lives. Erm, no comment

There's nothing quite like skype thats free and open source. So, I went back to the forums, going on the assumption that it must be possible, and that some wonderful person would have either figured a way to crack into the program and unlock the functionality to txt message, or written a new program to interface with skype and allow it to be able to send text messages.

Unlikely as it sounds, and the reason why I make this post, is because not only someone has, but it works. IT WORKS!!! Ha ha ha ha ha in your face! I know this is sad but I'm so happy with this result - i feel like the little orphan boy born with no opportunities and no hopes, who happens to stray across a wadge of bank notes, unsighted, in the dirty street. They have fallen out of a passing carriage. There's a massive grin across my grubby little urchin face. IThere's another eason for my post - to sing the praises of those geeks who toil for hours writing these impenetrable programs, and upload them onto the web for everyone to use.

Next time you see a computer nerd, give him a little squeeze for me.

Don't get me wrong, the solution isnt perfect. Its a little clunky and temperamental, but its all we have and i found it and it works! Ok, so this is what you do: there are two things to download and install. The first is a program language written in python which allows links to skype to be made, and interfacing from the command line. You download it from here. Downlaod the tarball named 'Skype4Py-1.0.29.0.tar.gz', untar it into a directory somewhere (like your desktop), then in the terminal make that directory your working directory and type:

$ sudo su | python setup.py install

It basically allows you to, within python, talk to the skype program and, if your clever enough, get it to do things for you. I tried to write a couple of rudimentary python scripts to do stuff with it, but it would have basically meant learning a new coding language, which I'm not really prepared to do just to send a few txt messages! Turns out I dont have to because someone called Rafael Laboissiere has already done it. As explained here, you install the 'skysentials' program easily:

$ wget http://alioth.debian.org/~rafael/skype/skysentials_1.0.1-1_all.deb
$ sudo dpkg -i skysentials_1.0.1-1_all.deb

Then, open a konsole (if you havent got konsole type # sudo apt-get update, then #sudo apt-get install konsole) by typing

$ konsole

Then open 2 windows. As superuser, type # skype in one - skype appears, and you sign in as per usual. In the other window, type #skysentials. A gui appears which allows you to send texts via skype, working out the cost automatically. Brilliant!

to skype software developers - is this capability latent within skype for linux, and if so why hasnt it been unlocked for users?

to rafael and the developers of skype4Py - you really are clever and unsung, thank you

and to everyone else, goodnight and happy texting, whatever OS you use ...

LaTeX equations in Blogger posts

Originally posted in The Daily Daniel
___________________________


Well, I know you are burning to know this one (!) Fetch the code directly from CodeCogs site and use in your website following this link

It uses beautiful

typesetting to create an image, which you embed in your blog posting. For example, the continuity equation for water waves:



Voila! Hope that's clear

A tour through home 'pageranking' - try it yerself!

Originally posted in The Daily Daniel
_____________________________

Now I live in the foothills of silicon valley, I thought I'd write a piece on one of its most famous and useful and revolutionary spawns, Google, and specifically it's PageRank algorithm. This is the piece of code which Google Search engines use to find webpages that are most relevant to your enquiry, and rank them so the most relevant are near the top. Its a bloody marvel, as we all know (remember the bad old days using old lycos and yahoo searches etc - pah!)

Instead of writing stuff on it, which has been done elsewhere numerous times, I thought I would show you how to do it using Matlab

First, you need to get the code. Cleve Moler at Mathworks has written a library which contains a primitive attempt at the proprietary algorithm. You need to get it, unzip it and untar it. This can all be done within Matlab, like so:

>> url='http://www.mathworks.com/moler/ncm.tar.gz';
>> gunzip(url,'ncm')
>> untar('ncm/ncm.tar','ncm')

Next, cd to where you unzipped it to, which is probably -

>> cd([pwd,filesep,'ncm'])

Then run the surfer algorithm on a website of your choosing, this one for example, and tell it to search for n nodes (in this case, 200 - warning, it takes a while)

>> [U,G]=surfer('http://www.thedailydanielblog.blogspot.com',200);

U is a list of websites connected to yours, arranged in matrix G. My website surfer matrix looks a little like this:




I was interested in what these sites were so I wrote a bit of code to print them to a text file:

>> fid=fopen('dansweb.txt','wt');
>> for i=1:size(char(U),1), fprintf(fid,'%s\n',char(U(i,:))); end
>> fclose(fid)

dansweb.txt then contains a list of 200 websites

The pagerank algorithm was then applied to U and G to find

>> pagerank(U,G)

The results are unsurprising - it charts links i have made, blogspot pages associated with my labels, and widgets i have used. There is even a bar chart output:
which looks slightly better if you add the websites to it

page-rank in out url
190 0.0432 9 2 http://www.usgs.gov
191 0.0383 4 2 http://www.usgs.gov/ask
192 0.0383 4 2 http://search.usgs.gov
72 0.0137 21 0 http://www.blogger.com
79 0.0131 40 0 http://purl.org/syndication/thread/1.0
71 0.0129 20 0 http://www.blogger.com/profile/
184 0.0098 5 0 http://planet.ubuntu.com
5 0.0092 60 6 http://www.blogger.com/profile/02981038739002302942
189 0.0092 1 0 http://wiki.octave.org
193 0.0091 6 5 http://marine.usgs.gov
10 0.0087 61 0 http://ims.ucsc.edu
3 0.0086 59 117 http://thedailydanielblog.blogspot.com/feeds/posts/default
4 0.0085 58 118 http://www.blogger.com/feeds/1936258002310927863/posts/default
8 0.0082 57 1 http://www.gnu.org/software/octave
11 0.0082 57 0 http://twitter.com
12 0.0082 57 0 http://hypem.com
63 0.0082 57 7 http://www.ourblogtemplates.com
21 0.0081 56 54 http://thedailydanielblog.blogspot.com/search/label/photos
24 0.0081 56 50 http://thedailydanielblog.blogspot.com/search/label/santa%20cruz
23 0.0081 56 49 http://thedailydanielblog.blogspot.com/search/label/san%20francisco
26 0.0081 56 46 http://thedailydanielblog.blogspot.com/search/label/travel
22 0.0081 56 43 http://thedailydanielblog.blogspot.com/search/label/preparations
25 0.0081 56 42 http://thedailydanielblog.blogspot.com/search/label/santa%20news
14 0.0080 56 36 http://thedailydanielblog.blogspot.com/search/label/internet
15 0.0080 56 36 http://thedailydanielblog.blogspot.com/search/label/job
17 0.0080 56 36 http://thedailydanielblog.blogspot.com/search/label/map
18 0.0080 56 36 http://thedailydanielblog.blogspot.com/search/label/matlab
20 0.0080 56 35 http://thedailydanielblog.blogspot.com/search/label/musings
16 0.0080 56 34 http://thedailydanielblog.blogspot.com/search/label/linux
27 0.0080 56 34 http://thedailydanielblog.blogspot.com/search/label/work
19 0.0080 56 33 http://thedailydanielblog.blogspot.com/search/label/move
6 0.0078 21 0 http://www.blogger.com/openid-server.g
67 0.0069 20 0 http://www.blogger.com/openid-server.g\42 /\76\n
196 0.0057 3 4 http://walrus.wr.usgs.gov/infobank
197 0.0057 3 2 http://mrib.usgs.gov
183 0.0053 4 1 http://fridge.ubuntu.com
194 0.0052 3 0 http://vineyard.er.usgs.gov/query.html
66 0.0052 17 0 http://www.blogger.com/profile/02981038739002302942\42 /\76\n

So, I have a pagerank of 0 (more or less) which is the lowest possible. The highest is 10, so this isn't a great example, admittedly, but I'll let interested readers have a play.

I discovered that facebook has a 9 and plymouth.ac.uk has a 7. Now, before you comment, i know that there are already loads of web-based pagerank checkers out there, but this is more fun, and here you get a breakdown of what sites contribute to each other's scores, OK?

Said in a Neil Buchanon from Art Attack (ITV since the 1990s) - 'try it yerself'!!

>> clear, clc, exit

Geeky things i found

Originally posted on 'The DailyDaniel'
------------------------------------------

Linux is 17 years old. In May 2008, 1.95% of the OS market was shared by Linux distributions (compared to 91.11% Windows). This depresses me.

Check out Ubuntu at http://www.ubuntu.com/. It's just a great operating system.

%%%%%%%%%%%%%%%%%%%
Did you know that the Matlab logo (the first eigenfunction of the L-shaped membrane) is on a “sponsor a highway” sign on Route 9 in Natick,Massachusetts. Cleve Moler, creator of Matlab, likened its shape to "a beach towel blowing in the wind, constrained by a picnic basket on one-fourth of the towel".
%%%%%%%%%%%%%%%%%%

A map of the internet!!! Now this IS cool

http://www.touchgraph.com/TGGoogleBrowser.php?start=digg.com

%%%%%%%%%%%%

and finally a very funny and clever map from the makers of the Onion

http://www.theonion.com/content/atlas/


Good night!