Create a diary for your daily workflow

You can modify your ~./bash_history file so it doens't store duplicates and stores more lines than the default (only 1000):



However, sometimes you want to go back to your history file and see what commands you issued on a particular day .... one solution is to make a periodic backup with a timestamp using cron.

First, make a directory to put your files in, and make a script which is going to do the work:



Open the shell script and copy the following inside



close, then in the terminal issue


admittedly hourly is a little excessive - go here is you are unfamiliar with cron scheduling definitions and modify as you see fit

Installing Mb-System on Fedora 17

MB-System is the only fully open-source software I am aware of for swath sonar data including multibeam and sidescan backscatter. As such, I'm really hoping using it works out for me!

Compiling it on a fresh install of Fedora 17 wasn't that straight-forward (took me about 5 hours to figure all this out), so I thought I'd put together a little how-to for the benefit of myself, and hopefully you.

I've broken it down into a few stages so some sort of semblance of order can be imposed on this random collection of installation notes.

Step 1: download the software from ftp server here

In the terminal:

For me this created a folder called 'mbsystem-5.3.1982'

Step 2: install pre-requisites

A) Generic Mapping Tools (GMT). You need to go to the Downloads tab, then click on the INSTALL FORM link. This will take you to a page where you can download the installation script, and fill in an online form of your parameter settings which you can submit and save the resulting page as a textfile which becomes the input to the bash script. Sounds confusing, but it's not - the instructions on the page are adequate. I opted to allow GMT to install netCDF for me. Then in the terminal I did this:

B) Perl parallel forkmanager:

C) X11

D) openmotif

E) fftw

F) ghostview - I had to install this indirectly using kde3:

(Note to Fedora - why oh why oh why are B, C, and F above not installed by default!?)

G)  OTPSnc tidal prediction software: download from here

untar, and cd to the directory

first double check that ncdump and ncgen are installed (which ncdump ncgen)

then edit the makefile so it reads:

then in the terminal issue:

Hopefully this compiles without errors, then I moved them to a executable directory:

H) openGL libraries (again I had to do this indirectly, this time through mesa):

Step 3: prepare mbsystem makefiles

cd mbsystem-5.3.1982/

You have to go in and point install_makefiles to where all your libraries are. This is time-consuming and involves a lot of ls, which, and whereis!

Here's a copy of the lines I edited in my install_makefiles parameters:

Then in the terminal

Step 3: install mbsystem

first I had to do this (but you may not need to)

I then updated my ~/.bashrc so the computer can find all these lovely new files:

Test the install by issuing



you should see the manual pages come up. Right, now down to the business of multibeam analysis!

em1 to ethX on Fedora 17 (how to install matlab if you don't have eth)

I have just moved from Ubuntu 12.04 to Fedora 17. No reason other than necessity for my new job. I'm still happy with Ubuntu and even don't mind Unity!

First thing I wanted to do, of course, was install matlab. It turns out I had a bit of work to do to get it installed on my Dell Precision machine! So here's a post to help out anyone who has the same problem.

If you don't have some eth listed when you ifconfig (I had the NIC listed as em1), matlab will install but you can't activate it. The problem is described on their website  

Basically, Fedora udev names NICs (network cards) embedded on the motherboard as em1, em2 (etc) rather then eth0, eth1 (etc). More details here

The two solutions listed on the Mathworks for changing the names back to eth do not work in my Fedora install. The reason why the first one doesn't work is because there is no /etc/iftab file. the reason why the second one works is because udev rules are not listed in etc (they are in lib instead and have different names and structures)

Thankfully, Fedora has built in a mechanism to modify the names of NICs post install. You do it by editing the following file

/lib/udev/rules.d/71-biosdevname.rules

so the line which reads

GOTO="netdevicename_end"

commented by default, is uncommented

Then you need to edit
/etc/default/grub

by adding the following line to the bottom:

biosdevname=0

Then I backed up my ifcfg em1 network script by issuing

cp /etc/sysconfig/network-scripts/ifcfg-em1 /etc/sysconfig/network-scripts/ifcfg-em1.old

then I renamed it eth0 (the number can be anything, I believe, between 0 and 9, and matlab will figure it out when it activates):

mv /etc/sysconfig/network-scripts/ifcfg-em1 /etc/sysconfig/network-scripts/ifcfg-eth0

and edited ifcfg-eth0 so the line which read

DEVICE="em1"

to read:

DEVICE="eth0"

This is all the information the machine needs, when it boots up, to change the naming convention from em1 back over to eth0.

Upon reboot, ifconfig shows eth0 where em1 was before (dmesg | grep eth would also show the change), and matlab installs perfectly.

A satisfyingly geeky end to a Tuesday!

Downloading Photobucket albums with Matlab and wget

Here's a script to take the pain out of downloading an album of photos from photobucket.

It uses matlab to search a particular url for photo links (actually it looks for the thumbnail links and modifies the string to give you the full res photo link), then uses a system command call to wget to download the photos All in 8 lines of code! 

Ok to start you need the full url for your album. The '?start=all' bit at the end is important because this will show you all the thumbnails on a single page. Then simply use regexp to find the instances of thumbnail urls. Then loop through each link, strip the unimportant bits out, remove the 'th_' from the string which indicates the thumbnail version of the photo you want, and call wget to download the photo. Easy-peasy!

Alternative to bwconvhull

Matlab's bwconvhull is a fantastic new addition to the image processing library. It is a function which, in its full capacity, will return either the convex hull of the entire suite of objects in a binary image, or alternatively the convex hull of each individual object. I use it, in the latter capacity, for filling multiple large holes (holes large enough that imfill has no effect) in binary images containing segmented objects 

For those who do not have a newer Matlab version with bwconvhull included, I have written this short function designed to be an alternative to the usage

 P= bwconvhull(BW,'objects');

and here it is:



It uses the IP toolbox function regionprops, around for some time, to find the convex hull images of objects present, then inscribes them onto the output image using the bounding box coordinates. Simples!

Craigslist Part 2: using matlab to plot your search results in Google Maps

In 'Craigslist Part 1' I demonstrated a way to use matlab to automatically search craigslist for properties with certain attributes. In this post I show how to use matlab and python to create a kml file for plotting the results in Google Earth. Download the matlab googleearth toolbox from here Add google earth toolbox to path. Import search results data and get map links (location strings). Loop through each map location and string the address from the url, then geocode to obtain coordinates. The function 'geocode' writes and executes a python script to geocode the addresses (turn the address strings into longitude and latitude coordinates). The python module may be downloaded here Once we have the coordinates, we then need to get rid of nans and outliers (badly converted coordinates due to unreadable address strings). Use the google earth toolbox to build the kml file. Finally, run google earth and open the kml file using a system command: The above is a simple scatter plot which only shows the location of the properties and not any information about them. Next shows a more complicated example where the points are plotted with labels (the asking price) and text details (the google map links) in pop-up boxes First each coordinate pair is packaged with the map and name tags. Concatenate the strings for each coordinate and make a kml file. Finally, run google earth and open the kml file using a system command:

Broadcom wireless on Ubuntu 12.04

I just upgraded to Ubuntu 12.04 and my broadcom wireless card was not being picked up. It took me some time to figure out how to do this, and the forums give a lot of tips which didn't work for me, so I thought I'd give it a quick mention.

Then disconnect your wired connection and reboot. I found it is crucial that you disconnect any wired connections.

 Upon reboot, go to System Settings - Hardware - Additional Drivers

 It should then pick up the broadcom proprietary driver. Install it, then reboot, and you should be back up and running!

Craigslist Part 1: Using matlab to search according to your criteria and retrieve posting urls

I have a certain fondness for using matlab for obtaining data from websites. It simply involves understanding the way in which urls are constructed and a healthy does of regexp. In this example, I've written a matlab script to search craiglist for rental properties in Flagstaff with the following criteria: 1) 2 bedrooms 2) has pictures 3) within the price range 500 to 1250 dollars a month 4) allows pets 5) are not part of a 'community' housing development. This script will get their urls, prices, and google maps links, and write the results to a csv file. First, get the url and find indices of the hyperlinks only, then get only the indices of urls which are properties. Then get only the indices of urls which have a price in the advert description; find the prices; strip the advert urls from the string; sort the prices lowest to highest; and get rid of urls which mention a community housing development. Finally, get the links to google maps in the adverts if there are some, and write the filtered urls, maps, and prices to csv file. Save and execute in matlab, or perhaps add this line to your crontab for a daily update!

/usr/local/MATLAB/R2011b/bin/matlab -nodesktop -nosplash -r "craigslist_search;quit;"

Reduce file size/quality of pdf using ghostscript

Write and execute a Matlab script without opening Matlab

Here's an example of using cat to write your m-file and then execute from the command line. This particular example will just plot and print 10 random numbers.

SmartGit on Ubuntu 10.04

SmartGit is a graphical user interface for managing and creating git repositories for software version control

First, uninstall any git you already have (or thought you had)

 sudo apt-get remove git  

Then, get the latest git

 sudo aptitude build-dep git-core  
 wget http://git-core.googlecode.com/files/git-1.7.9.1.tar.gz  
 tar xvzf git-1.7.9.1.tar.gz  
 cd git-1.7.9.1  
 ./configure  
 make  
 sudo make install  
 cd ..  
 rm -r git-1.7.9.1 git-1.7.9.1.tar.gz  

Next, you need Java Runtime Environment 7 which you can download from here

and then follow the installation instructions here

Download SmartGit from here, then

 tar xvzf smartgit-generic-2_1_7.tar.gz  
 cd smartgit-2_1_7  
 cd bin  

Then edit the smartgit shell script

 gedit smartgit.sh  

updating SMARTGIT_JAVA_HOME to where JRE is installed (e.g. /usr/lib/jvm/jrel1.7.0_03). Now you're ready to run

 ./smartgit.sh  

As part of the installation you will be prompted for the location of git, which you can find out by issuing in another terminal:

 whereis git  

Ellipse Fitting to 2D points, part 3: Matlab

The last of 3 posts presenting algorithms for the ellipsoid method of Khachiyan for 2D point clouds. This version in Matlab will return: 1) the ellipse radius in x, 2) the ellipse radius in y, 3) the centre in x, 4) the centre in y, and 5) orientation of the ellipse, and coordinates of the ellipse

Ellipse Fitting to 2D points, part 2: Python

The second of 3 posts presenting algorithms for the ellipsoid method of Khachiyan for 2D point clouds. This version in python will return: 1) the ellipse radius in x, 2) the ellipse radius in y x is a 2xN array [x,y] points

Ellipse Fitting to 2D points, part 1: R

The first of 3 posts presenting algorithms for the ellipsoid method of Khachiyan for 2D point clouds. This version in R will return: 1) the ellipse radius in x, 2) the ellipse radius in y, 3) the centre in x, 4) the centre in y, and 5) orientation of the ellipse, and coordinates of the ellipse

Non-linear least squares fit forced through a point

Sometimes you want a least-squares polynomial fit to data but to constrain it so it passes through a certain point. Well ... Inputs: x, y (data); x0 and y0 are the coordinates of the point you want it to pass through, and n is the degree of polynomial

Matlab function to generate a homogeneous 3-D Poisson spatial distribution

This is based on Computational Statistics Toolbox by W. L. and A. R. Martinez, which has a similar algorithm in 2D. It uses John d'Errico's inhull function Inputs: N = number of data points which you want to be uniformly distributed over XP, YP, ZP which represent the vertices of the region in 3 dimensions Outputs: x,y,z are the coordinates of generated points

3D spectral noise in Matlab with specified autocorrelation

Here is a function to generate a 3D matrix of 1/frequency noise Inputs are 1)rows, 2) cols (self explanatory I hope), 3) dep (the 3rd dimension - number of images in the stack) and 4) factor, which is the degree of autocorrelation in the result. The larger this value, the more autocorrelation an the bigger the 'blobs' in 3D Enjoy!

Computational geometry with Matlab and MPT

For computational geometry problems there is a rather splendid tool called the Multi Parametric Toolbox for Matlab Here are a few examples of using this toolbox for a variety of purposes using 3D convex polyhedra represented as polytope arrays. In the following examples, P is always a domain bounded 3D polytope array

Restoring Nvidia graphics on Ubuntu

If you're an idiot like me and do this:

sudo apt-get --purge remove nvidia-*

You'll soon realise your display looks a little crappy, because you've turned off your graphics card drivers. Not to worry! You restore like this:

sudp apt-get install nvidia-*

then go into /etc/modprobe.d/blacklist.conf and remove any 'blacklists' to nvidia drivers, then restart the x-server using

sudo /etc/init.d/gdm restart

Normal service resumed, and you are free to make a new balls-up!

Wrapping and interfacing compiled code into more flexible and user-friendly programs using Zenity and Bash

Subtitles include:

1. How to reconstruct the three-dimensional shape of an object in a scene in a very inexpensive way using shadow-casting
2. How to write and execute a matlab script from a bash script

I've written a bash script - shadowscan.sh - which is a wrapper for a set of programs for the reconstruction 3D surface of an object in a series of photographs using the ingenious shadow-casting algorithm of Jean-Yves Bouguet and Pietro Perona

The script relies heavily on the GUI-style interfacing tools of Zenity and acts as an interface and wrapper for the algorithm as implemented by these guys. It will also write and execute a matlab script for visualising the outputs.

This script should live in a directory with the following compiled programs, all available here:
1. ccalib (performs the camera calibration to retrieve the intrinsic parameters of camera and desk)
2. find_light (performs the light calibration to find light-source coordinates)
3. desk_scan (does the 3d reconstruction)
4. merge (merges 2 3d reconstructions, e.g. from different angles)

and the following matlab script:
1. selectdata.m (available here)

When the makefile is run, it creates the compiled programs in the ./bin directory. It's important to consult the README in the main folder as well as within the 'desk_scan' folder to have a clear idea of what the program does and what the input parameters mean.

My program will check if the camera calibration and light calibration files exist in the folder and if not will carry them out accordingly then several reconstructions are carried out, and the outputs merged. Finally matlab is called from the command line to run 'p_lightscan_data.m' to visualize the results.

when the script finishes there will be the following output files:
1. xdim.txt - max dimension to scan to
2. params - camera calibration parameters
3. data - camera calibration data
4. light.txt - light calibration parameters
5. sample.pts - [x,y,z] of object, in coordinates relative to light source
6. sample_lightscan1.tif - filled colour contour plot of reconstructed object
7. sample_lightscan2.tif - mesh plot of reconstructed object

I've tried to keep it flexible. It will run only the elements it needs to. For example, if the camera or lighting has been carried out before for a different object it will pick up on that. Also, the program can be invoked by passing it two arguments (path to folder where sample images are, and what sequence of contrasts to use for the reconstruction), which avoids having to input with the graphical menus. Example usage:
1. bash shadowscan.sh
2. bash shadowscan.sh /home/daniel/Desktop/shadow_scanning/sample/bottle 30.35.40.45

Program for spatial correlogram of an image, in C

This program will read an image, convert it to greyscale, crop it to a square, and calculate it's spatial (1D) autocorrelation function (correlogram), printing it to screen.

It uses OpenCV image processing libraries - see here and here

Compilation instructions (in Ubuntu):

 sudo apt-get install libcv-dev libcv1 libcvaux-dev libcvaux1 libhighgui-dev libhighgui1  
 pkg-config --libs opencv; pkg-config --cflags opencv  
 PKG_CONFIG_PATH=/usr/lib/pkgconfig/:${PKG_CONFIG_PATH}  
 export PKG_CONFIG_PATH  
 gcc `pkg-config --cflags --libs opencv` -o Rspat Rspat.c  

Usage in the terminal:

 ./Rspat IMG_0208.JPG   

The program:

Controlling a serial device with Python

This program to sends a command to serial device, receives data from the device, converts it to numeric format, and graphs the data in real time. It is written in Python using wxPython for the display.

The device is connected to /dev/ttyS0 with a baud rate of 19200, 8-bit data, no parity, and 1 stopbit. You need to edit the line 'serial.Serial' with the relevant parameters to use this code for your device. Editing 'DATA_LENGTH' will change the rate at which the graph scrolls forward. Also, you'd need to change the 'command' that the device understands as 'send me data' (here it is the s character - line 'input'). Knowledge of the format of the returned data is also essential (here I turn ascii into 2 integers) line 'struct.unpack' - see here for help). Otherwise this should work with any device. Tested on Ubuntu 9.10 and 10.4 using serial port and USB-serial converters.

Using cron to backup Thunderbird emails

My bash script (which I've saved as backup_emails.sh in my home folder and made executable) looks a little like this:

 set -x   
  x=`date +%y%m%d.%H%M`   
  tar zcf thunderb-mail-${x}.tgz ~/.mozilla-thunderbird   
  mv thunderb-mail-${x}.tgz /media/MY_HARDDRIVE/email_backups/   

Then invoke in cron using:

 crontab -e   

add something like:

 30 4 * * * ~/./backup_emails.sh  

which does the backup at 04.30 every day

Make Google Earth Faster on Ubuntu

In terminal type:

 export LIBGL_ALWAYS_INDIRECT=true  

Batch GIMP script for auto-sharpen, white-balance and colour enhance

Like you, my point-and-shoot camera in auto-ISO mode often, if not always, gives me pictures which need a little adjustment to realise their full potential.

In GIMP, I usually use the 'auto white balance' option which usually gives great results. Often, I also use the 'auto color enhance' option and, occasionally, I use a simple filter to sharpen the features in the image.

Here's a GIMP script which will automate the process for a folder of pictures There are four input parameters. The first is the 'pattern' to find the files, e.g. "*.tiff" gives you all the tiff files in the directory. The next three inputs refer only to the sharpening filter. I use the (confusingly named) 'unsharp' filter which works by comparing using the difference of the image and a gaussian-blurred version of the image.

'radius' (pixels) of the blur (>1, suggested 5)
'amount' refers to the strength of the effect (suggested 0.5)
'threshold' refers to the pixel value beyond which the effects are applied (0 - 255, suggested 0)

These require a little bit of trial and error, depending on the type of picture. In the command line, the script is called like this:

 gimp -i -b '(batch-auto-fix "*.JPG" 5.0 0.5 0)' -b '(gimp-quit 0)'  

An obligatory example. Before:




and after:






Enjoy!

Convert movies to be compatible with ZEN X-Fi media player

Creative - another company which doesn't provide support for non-Windows users. Get with the times, losers!

The ZEN player needs a specific format of movie (bit-rate and aspect ratio). Here's how to do it with mencoder (available for free on all platforms)

 mencoder 'MyMovie.avi' -oac mp3lame -lameopts cbr:mode=2:br=96 -af resample=44100 -srate 44100 -ofps 20 -ovc lavc -lavcopts vcodec=mpeg4:mbd=2:cbp:trell:vbitrate=300 -vf scale=320:240 -ffourcc XVID -o MyMovie4zen.avi  

Granular Therapy

Here's a relaxing movie I just made of spinning modelled sand grains:



I'm not going to divulge the details of the model of how to create the sand grains (a current research project of mine). I can tell you I used these rather excellent functions for MATLAB:

Volume interpolation Crust algorithm, described more here

A toolbox for Multi-Parametric Optimisation

I used 'recordmydesktop' to capture the grains in my MATLAB figure window. It created an OGV format video (which I renamed to 'beautiful_grains.ogv'). I then used the following code to decode the video into jpegs

 mkdir out_pics  
 mplayer -ao null -ss 00:00:03 -endpos 50 beautiful_grains.ogv -vo jpeg:outdir=out_pics  

I needed to crop the pictures to just the grains, then turn the cropped pictures back into a video. I used this:

 cd out_pics  
 for image in *.jpg  
 do  
 # crop the image  
 convert $image -crop 600x600+350+135 grains%05d.tiff  
 done  
 # make movie from the cropped tiffs  
 ffmpeg -r 25 -i grains%05d.tiff -y -an spinning_grains_small.avi  

et voila! Now sit back with a cuppa and relax to those spinning grains.

Updated Power Spectrum from an Image

A while back I posted on using ImageMagick to do a full frequency-decomposition on an image, and create a power spectrum. It had several inaccuracies. Here's an update with how to do it better

First, you need to install a version of IM which is HDRI enabled (see here)

# test for valid version of Image Magick

 im_version=`convert -list configure | \  
  sed '/^LIB_VERSION_NUMBER /!d; s//,/; s/,/,0/g; s/,0*\([0-9][0-9]\)/\1/g'`  
 [ "$im_version" -lt "06050407" ] && errMsg "--- REQUIRES IM VERSION 6.5.4-7 OR HIGHER ---"  

# test for hdri enabled

 hdri_on=`convert -list configure | grep "enable-hdri"`  
 [ "$hdri_on" = "" ] && errMsg "--- REQUIRES HDRI ENABLED IN IM COMPILE ---"  

#read $file, load and converting to greyscale

 convert $file -colorspace gray in_grey.tiff  

# crop image to smallest dimension

 min=`convert in_grey.tiff -format "%[fx: min(w,h)]" info:`  

# crop image, and convert to png (why?)

 convert in_grey.tiff -background white -gravity center -extent ${min}x${min} in_grey_crop.png  

# get data, find mean as proportion, convert mean to percent, and de-mean image

 data=`convert in_grey_crop.png -verbose info:`  
 mean=`echo "$data" | sed -n '/^.*[Mm]ean:.*[(]\([0-9.]*\).*$/{ s//\1/; p; q; }'`  
 m=`echo "scale=1; $mean *100 / 1" | bc`  
 convert in_grey_crop.png -evaluate subtract ${m}% in_grey_crop_dm.png  

# pad to power of 2

 p2=`convert in_grey_crop_dm.png -format "%[fx:2^(ceil(log(max(w,h))/log(2)))]" info:`  
 convert in_grey_crop_dm.png -background white -gravity center -extent ${p2}x${p2} in_grey_crop_dm_pad.png  

# fft

 convert in_grey_crop_dm_pad.png -fft in_ft.png  

# get spectrum

 convert in_ft-0.png -contrast-stretch 0 -evaluate log 10000 in_spec.png  

... but it's still not quite right. Any tips?

Turn your OGV movie section into a cropped animated GIF

An example from a recent project

 # make directory for output jpg  
 mkdir out_pics   
 # section to the ogv video file into jpegs  
 mplayer -ao null -ss 00:00:03 -endpos 50 beautiful_movie.ogv -vo jpeg:outdir=out_pics  
 cd out_pics  
 # loop over each image and crop into desired rectangle  
 for image in *.jpg  
 do  
 convert $image -crop 600x600+350+135 "$image crop.tiff"  
 done  
 # turn into an animated gif  
 convert -page 600x600+0+0 -coalesce -loop 0 *.tiff beautiful_animation.gif  

Miscellaneous useful bash

For my benefit more than anything - things I use over and over again, but do not have the mental capacity to remember

# number of files in current directory

 ls -1R | grep .*.<file extension> | wc -l  

# check if you have a program installed

 aptitude show <name of program> | grep State  

# only list the (sub-)directories in your current directory

 ls -l | grep ^d  

# remove backup files

 find ./ -name '*~' | xargs rm  

# use zenity get the directory where the file you want is

 zenity --info --title="My Program" --text="Select  
 which directory contains the file(s) you want"  
 export target_dir=`zenity --file-selection --directory`  

# list all txt files in this directory

 x=$( ls $target_dir | grep .*.txt |sort|zenity --list --title="Select" --column="coordinate file" )  

# get this directory

 export raw_dir=`zenity --file-selection --directory`  

# removes .txt (for example, for later use as start of filename)

 str="$x"  
 Var1=`echo $str | awk -F\. '{print $1}'`  

# gets the rest of the file name

 str="$Var1"  
 Var2=`echo $str | awk -F\- '{print $2}'`   

# turn string into (indexable) array

 e=$(echo $mystring | sed 's/\(.\)/\1 /g')  
 b=($e)  

# get max and min dimensions of an image

 min=`convert image1.tiff -format "%[fx: min(w,h)]" info:`  
 max=`convert image1.tiff -format "%[fx: max(w,h)]" info:`  

# take away 1 from both

 min=$((min-1))   
 max=$((max-1))  

Unlock the power of ImageMagick to HDRI

My primary motivation to do this has been to use ImageMagick for Fast Fourier Transforms, which have a myriad of applicability in image processing and filtering

First, you have to install perlmagick. Using apt,

 sudo apt-get install perlmagick  

Then the perl libraries

 sudo apt-get install libperl-dev  

Install the rather magnificent fftw libraries

 sudo apt-get install libfftw3-dev fftw-dev  

wget the source code, untar, move it, cd home

 wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz   
 tar xvfz ImageMagick.tar.gz   
 mv ImageMagick-6.6.0-8 $HOME  
 cd $HOME/ImageMagick-6.6.0-8   

Install (see here for instructions)

 make distclean   
 ./configure --enable-hdri LDFLAGS='-L/usr/local/lib -R/usr/local/lib'  

see also here for a bug fix, and also see here for instructions

make & sudo make install
make check


To see whether the fft commands are working properly, you could issue the following:

 if [ -s lena.png] #check to see if the file exists  
 then  
 # do fft and ifft roundtrip  
 convert lena.png -fft -ift lena_mp_roundtrip.png  
 else  
 #first download  
 wget http://optipng.sourceforge.net/pngtech/img/lena.png  
 #then convert  
 convert lena.png -fft -ift lena_mp_roundtrip.png  
 fi  

see here for further information on fft transforms with, and watch your world spin faster

Installing NCVIEW for Ubuntu

Recently I ran into problems installing NCVIEW, which is a tool for efficiently viewing the contents of a NetCDF data file. The problem was something to do with the X terminal. So here's how to install it from source:

 wget ftp://cirrus.ucsd.edu/pub/ncview/ncview-1.93g.tar.gz  
 sudo apt-get install libxaw7-dev  
 tar -xvvf ncview-1.93g.tar.gz  
 cd ncview-1.93g/  
 ./configure --x-libraries=/usr/X11R6/lib  
 make  
 make install  

Quick tips for Ubuntu 9.04

One of the beauties of starting your OS afresh is, well, you get a new and better OS. Ubuntu 9.4 (or 'jaunty jackalope') is a slick version of my previous Ubuntu 8.04 ('hardy heron'). So far, I've ascertained that it looks better, it has better support for hardware, and it boots really fast.

One of the pains of starting your OS afresh is having to reinstall all your software and set your preferences. I thought I'd put a little 'as I do it' guide which may take some of the pain away. This list has a few of the 'biggies' and maybe a couple of items more suited to my personal taste, but what the hey!

1. configure the firewall
sudo apt-get install gufw

2. install wine (windows emulator, for those windows programs you cant live without - an ever decreasing number horray!)
sudo apt-get install wine

3. This one is crucial if you want to run java - it takes ages, and at the end when it installs java, you need to be able to use the tab and 'pg dwn' keys to get to the end of the license agreement
sudo apt-get install ubuntu-restricted-extras

4. VLC is still the best media player around
sudo apt-get install vlc

5. skype -
sudo wget http://www.medibuntu.org/sources.list.d/jaunty.list -O /etc/apt/sources.list.d/medibuntu.list

wget -q http://packages.medibuntu.org/medibuntu-key.gpg -O- | sudo apt-key add - && sudo apt-get update

sudo aptitude install skype

6. Google Earth (version 5) - install the binary from here. Right click - Properties - Permissions - make executable. Then double click, and it will install - easy! It will create a desktop icon which, by default, you cannot see because you do not have permission. In the terminal type (here, replace 'daniel' with your username):

cd ./Desktop
sudo chown daniel:daniel Google-googlearth.desktop

7. Expand the capabilities of nautilus file browser

sudo apt-get install nautilus-actions nautilus-gksu nautilus-image-converter nautilus-open-terminal nautilus-script-audio-convert nautilus-script-collection-svn nautilus-script-manager nautilus-sendto nautilus-share nautilus-wallpaper

8. Adobe AIR

wget http://airdownload.adobe.com/air/lin/download/latest/AdobeAIRInstaller.bin
chmod +x ./AdobeAIRInstaller.bin
./AdobeAIRInstaller.bin

9. Adobe acrobat reader

sudo gedit /etc/apt/sources.list

Add the below line in sources.list

deb http://archive.canonical.com/ubuntu jaunty partner

sudo apt-get update
sudo apt-get install acroread

G'night!

Never do this

 sudo mv filename.file /bin/sh  

Just don't mess with your bin/sh, and secondly don't attempt to restart your computer after you mess with your /bin/sh (which is the symbolic link to /bin/dash - low level crazy stuff which ordinarily you shouldnt ever know or care about, but which ensures your computer can do some vital things, like initialise).

This I discovered to my cost, and now I'm having to spend my weekend recovering all my files

This is a worrying and heartbreaking exercise, as I'm sure you'll understand - my computer is my livelihood and most of my hobbies, and I havent backed stuff up for ages, and recently I've been doing some cool stuff which exists only on the computer which now needs a complete OS overhaul!!

I'm writing this for instruction and warning rather than pity, you understand. You can't reboot, not even in any safe mode. I did a bit of internet forum research, and there was little guidance (mostly, instructions for windows users to use Ubuntu to recover their files, but not Ubuntu users to recover their files)

It turns out it's thankfully simple.

1. download an Ubuntu distro iso, and burn it to CD (instructions here and here)
2. place in optical drive of bummed out computer, and use BIOS to instruct it to boot from disk

3. select option to 'try Ubuntu without installing it on your computer' (or whatever the exact wording is - should be the first option in most versions)

4. then wait for it to boot - the OS running from the CD can recognise your hard-drive, but won't let you automatically access it. Open a terminal and type the following (assuming you want to access drive sda1 - you may want sdb1 or something):

 sudo mkdir /media/ubuntu  
 sudo mount -t ext3 /dev/sda1 /media/ubuntu  

for ease of copying and later access, change the permissions to all directories so anyone can have read/write privileges:

 sudo su  
 chmod 777 -R /media/ubuntu  

this will take a while if you have a lot of files and directories, but when it's done you're ready to plug in a large external hard drive, and save your precious files away - exactly what I'm doing now! The next step will be to reinstall ubuntu (I'm going to go for the latest - 9.04) and spend weeks reinstalling all my software and getting it to my specification - weeks I really don't have!

Flower Bed

Recently I investigated Apollonian gaskets (a type of space-filling fractal composed of ever-repeating circles) for use as 'pore-space fillers' in a model I am creating for synthetic sediment beds. I came across a most excellent script here:

http://www.mathworks.com/matlabcentral/fileexchange/15987

In my Friday afternoon boredom, I have just modified some aspects of the program to create a function which simulates a pretty flower bed Some example outputs below - enjoy! Right, now back to work ...

Frequency Transforms on Images

As part of my work, I have to do a lot of spectral analysis with images. Traditionally I have used MATLAB to do this - it is exceptionally easy and powerful, and works just fine. As a simple example, if I want a power-spectrum of an image, treated as a 2D matrix of 8-bit numbers, the sequence of commands is this:

 %------------------- MATLAB  

 % read image in, convert to greyscale and double-precision  

 image=double(rgb2gray(imread('myimage.jpg')));  

 % crop to smallest dimension  

 [m,n]=size(im);  

 im=im(1:min(m,n),1:min(m,n));  

 % find image mean  

 imMean=mean(im(:));  

 % and subtract mean from the image  

 demean=im-imMean;  

 % find the 2D fourier transform  

 ft=fft2(demean);  

 % power spectrum is the element-by-element square of the fourier transform  

 spectrum=abs(ft).^2;  

Seven lines of commands. Couldn't be simpler. That's the beauty of MATLAB. The snag is that it costs several hundreds to thousands of dollars for the program, which makes it difficult to share your code with other people. So recently I've thought about coding up some things I want to share with other people in Python.

Python is a cross-platform, open-source programming language and environment. It is not too dissimilar to MATLAB in many respects, especially when you install the right libraries. In order to do the same in Python as above, as easy as possible, I needed to install Python Imaging Library (PIL),  NumPy and SciPy. Then the sequence of operations is as follows:

 # ----------------PYTHON  

 #-----------------------  

 # declare libraries to use  

 import Image  

 import scipy.fftpack  

 from pylab import*  

 import cmath  

 from numpy import*  

 # load image, convert to greyscale  

 im=Image.open('myimage.jpg').convert("L")  

 # crop to smallest dimension  

 box=(0,0,im.ny,im.nx)  

 region=im.crop(box)  

 # find mean  

 imMean=mean(asarray(region))  

 # 2D fourier transform of demeaned image  

 ft=fft2(region-imMean)  

 # power spectrum  

 spectrum=pow(abs(ft),2)  

Same answer. Not bad - not harder, just different. And completely free. Oh, and this'll work on UNIX, Linux, Mac, and PC when Python is installed (with a couple of extra libraries). Great! 

Hmm, I thought, I wonder if it is possible to do the same in a piece of non-numerical software. I tried ImageMagick because its free and ubiquitous on Linux and Mac, and also available for PC, and it has a massive user-community. It is incredibly powerful as a command-line graphics package - even more powerful than PhotoShop and Illustrator, and like all UNIX-based applications, infinitely flexible. It turns out it is a little more tricky to calculate the power spectrum in ImageMagick, but here's how.

First, you need the right tools - ImageMagick is usually already present on most Linux distributions - if not, if a debian/Ubuntu user use sudo apt-get install imagemagick, or download from the webiste hyperlinked above. Then you'll need an open-source C-program for calculation of FFTs: this is provided by FTW - download fftw-3.2.1.tar.gz, untar, cd to that directory, then install the binary by issuing the usual commands:

 ./configure  

 sudo make && make install  

This C-program is no good on its own, so the last ingredient is a wrapper for ImageMagick. Fortunately - nay, crucially, this has been provided by Sean Burke and Fred Weinhaus - introducing IMFFT! The program can be downloaded and installed from here. And here is an in-depth tutorial of how to use it (I'm only just getting to grips with it myself). For some reason the compiled program is called 'demo'. I move it to my /bin directory so my computer can see it wherever I am working, and in the process change the name to something more suitable - fft:

 cp demo ./fft && sudo mv ./fft /bin  

Right, it's a little bit more involved, but here is the bash shell script for the power spectrum in ImageMagick:

 #----------------------------- IMAGEMAGICK  

 #-------------------------------------------------  

 #!/bin/bash  

 # read image and convert to greyscale  

 convert myimage.tif -colorspace gray input_grey.tiff  

 # find min dimension for cropping  

 min=`convert input_grey.tiff -format "%[fx: min(w,h)]" info:`  

 # crop image  

 convert input_grey.tiff -background white -gravity center -extent ${min}x${min} input_grey_crop.tiff  

 # de-mean image  

 # get data  

 data=`convert input_grey_crop.tiff -verbose info:`  

 # find mean as proportion  

 mean=`echo "$data" | sed -n '/^.*[Mm]ean:.*[(]\([0-9.]*\).*$/{ s//\1/; p; q; }'`  

 # convert mean to percent  

 m=`echo "scale=1; $mean *100 / 1" | bc`  

 # de-mean image  

 convert input_grey_crop.tiff -evaluate subtract ${m}% ggc_dm.tiff  

 # pad to power of 2  

 p2=`convert ggc_dm.tiff -format "%[fx:2^(ceil(log(max(w,h))/log(2)))]" info:`  

 convert ggc_dm.tiff -background white -gravity center -extent ${p2}x${p2} ggc_dm_pad.tiff  

 # forward fast fourier transform  

 fft f ggc_dm_pad.tiff  

 # creates ggc_dm_F_Q16.tiff, on which get spectrum  

 convert ggc_dm_F_Q16.tiff -contrast-stretch 0% -gamma .5 -fill "rgb(1,1,1)" -opaque black ggc_dm_F_Q16_spec.tiff  

Voila! The spectrum is the image 'ggc_dm_F_Q16_spec.tiff'. I haven't yet worked out how to keep all the data inside the program, to eliminate the need (and cumulative errors associated with) writing the outputs at every stage to file, and eading them back in again. This means I can't use the spectrum for any scientific application. But in theory it should be possible. Any suggestions welcome!

Shannon's Box

Claude Shannon, father of information theory, apparently once invented a box whose sole purpose was to, when switched on, turn itself off again. Recently I've got into scripting with CHDK (Canon Hackers Development Kit) - a firmware enhancement for Canon cameras. It allows you to turn your cheap point-and-shoot camera into something better, for free. Long exposures, very rapid shots, motion detection - you name it. I can tell I'm at the start of a beautiful friendship ... look at this!

You can write and upload scripts to automate processes. The language is uBASIC - really intuitive and easy. My first script is pretty basic - it has been to tell the camera to, when switched on, automatically take 3 photos and then turn itself off! I therefore called it Shannon's Box:

 rem Author Daniel Buscombe  
 @title Shannon's Box  
 @param a Shoot Count  
 @default a 3  
 shoot full  
 for n=2 to a  
 shoot full  
 if n=a then goto "shutdown"  
 next n  
 end  
 :shutdown  
 shut_down  

More, I'm sure, on CHDK in the following months.

Ubuntu Beautification

Tip #1:

It's amazing really - most fonts contain within them “hints” laid down by their designer about how they should look on-screen. Artistic meta-data! However, Ubuntu ignores them and uses a system called 'autohinting'.

It works well, but you might also want to try bytecode hinting. This uses the hinting built into the fonts. To activate bytecode hinting, open a terminal window and type:

$ sudo dpkg-reconfigure fontconfig-config

Using the cursor keys, select Native from the menu and then hit Enter.

On the next screen you’ll be asked if you want to activate subpixel rendering. This is good for TFT screens, so make the choice (or just select Automatic). Next you’ll be asked if you want to activate bitmap fonts, which are non-true type fonts good for use at low point levels. There’s no harm in using them, so select yes. The program will quit when it’s finished. Once that’s happened, type the following to write the changes to the system and update files:

$ sudo dpkg-reconfigure fontconfig
$ sudo dpkg-reconfigure defoma

System - Quit - Logout, and then log back in again. Letters will appear more rounded and the antialiasing will appear better.

Tip #2:

465 fonts of Brian Kent (http://www.aenigmafonts.com) are available to Ubuntu users. To install the fonts, you’ll need to add a new software repository:

System - Administration - Software Sources
- Third-Party Software tab - Add

type the following into the dialog box that appears:

deb http://ppa.launchpad.net/corenominal/ubuntu hardy main

Click 'Add Sources', then Close and, when prompted, agree to reload the package lists. Then use Synaptic to search for and install the 'ttf-aenigma' package. Once installed the fonts will be available for use straight away in all applications.

Enjoy your beautified Ubuntu!

How to get on youtube, twitter and myspace quicker

Well, the easiest way is save time when you boot your computer!

Whenever Ubuntu boots it runs several scripts that start necessary background services. By default these are set to run one-by-one but if you have a processor with more than one core, such as Intel’s CoreDuo series or AMD’s Athlon X2, you can configure Ubuntu to run the scripts
in parallel. This way all the cores are utilized and quite a bit of time can be saved at each boot. To make the change, type the following to open the necessary configuration file in Gedit:

$ gksu gedit /etc/init.d/rc

Look for the line that reads CONCURRENCY=none and change it so it reads CONCURRENCY=shell.

Then save the file and reboot your computer. This should save you several seconds of time every time you boot up, and you can go on being a handsome speedy devil

Every pixel tells a story

Recently the Daily Daniel ran a story (hark at me! I'm not The Times!) with a single photograph - a mosaic - each 'pixel' containing a separate photo. I've finally got round to explaining how it is done, if only to show that I didn't arrange the pictures manually - she is beautiful, admittedly, but I gotta work to eat!

Right, the Linux program is called 'metapixel' and is downloaded from here. If you've enough photographs, you can produce some incredible results

You need to issue 2 commands - one prepares the photos, then the other makes a mosaic out of them. Create a folder somewhere to store the prepared photos called, say, 'recast'

$ metapixel-preparee -r /directory/where/photos/are /directory/recast --width=40 --height=40

The larger the width and height, the longer it takes. Be sure to get the aspect ratio of the photos right - it doesnt check or automatically correct. Choose a single photo that you want to make a mosaic out of (e.g. /directory/in.png), then (be warned, this may/will take several hours for a big photo!):

$ metapixel /directory/in.png --library /directory/recast --scale=5 --distance=5

The scale tells it how big you want the output (i.e. here 5 x size of input image), and the distance tells it how far to look for matches - it matches by colour so its a really good idea to scale by the range of inputs. It will give you an error if the distance parameter is too large. The results can be mixed, but once you hone the inputs and parameters down, and you have a LOT of patience, you can get some really great mosaics!

Tweetdeck on ubuntu 64-bit

Right, the whole world's into twitter. The website is a little difficult to manage when you have a lot of people to follow. The software solution (isn't there always one!) is TweetDeck - unfortunately it runs under Adobe Air, which (surprise, surprise) is available for Linux but 64-bit architectures. So your computer requires a little bit of persuading ...

... you need a few libraries - issue the following commands at the terminal:

$ sudo apt-get lib32asound2 lib32gcc1 lib32ncurses5 lib32stdc++6 lib32z1 libc6 libc6-i386
$ sudo getlibs -l libnss3.so.1d
$ sudo getlibs -l libnssutil3.so.1d
$ sudo getlibs -l libsmime3.so.1d
$ sudo getlibs -l libssl3.so.1d
$ sudo getlibs -l libnspr4.so.0d
$ sudo getlibs -l libplc4.so.0d
$ sudo getlibs -l libplds4.so.0d
$ sudo getlibs -l libgnome-keyring.so
$ sudo getlibs -l libgnome-keyring.so.0
$ sudo getlibs -l libgnome-keyring.so.0.1.1
$ sudo ldconfig

Then install Adobe Air from here

download, then move to your home directory,

right click - properties - make executable

and type

$ chmod +x AdobeAIRInstaller.bin
$ sudo ./AdobeAIRInstaller.bin

Then grab the 'air' file from here, and install using Adobe Air

Happy tweeting!

Miscellaneous stuff of use

The documentation on the use of 'wget' is a little sketchy on one of its most powerful uses - automatically downloading loads of files from internet servers. Let's say you need the 1000 tiff files sitting at http://www.server.com/hypothetical_directory

Apparently wildcards are out, such as:

$ wget http://www.server.com/hypothetical_directory/*.tiff

because html does not support 'globbing'. So here's the solution:

$ wget -r -1l --no-parent -A.tiff http://www.server.com/hypothetical_directory

%----------------------------------
Another thing I've needed to do recently is get ALL the metadata from an image file - you wouldn't believe how much information is recorded - channel statistics, histogram of shades, date/time, camera make, resolution - you name it, its recorded. It's a wonderful thing about using images for your research as well - you can keep track of all your metadata in the data files! Right, well I discovered that no amount of 'right-clicking' on the image - in any operating system - will tell you everything you need to know. The suitable command is:

$ identify -verbose myimage.jpg

%------------------------------------
Finally, for now, a couple of sites that will help you if you need to make a purchase of hardware for your linux machine:

This one keeps a database of usb-devices which are known to be linux-friendly, and the following two help you out when choosing a compatible printer and scanner: here and here.

Keep on truckin'

How to read and search your blog page with the imaginative use of the 'regexp' in matlab

Just a quickie - there may be some application for this (?)

You start with your blog url:

>> url='http://thezestyblogfarmer.blogspot.com/';

read it in, then you can start searching through its contents:

>> text=urlread(url);

for example, I can list all the unix commands I have mentioned - its easy because they all start with a dollar sign

>> text_linux=regexp(text,'>\$[^<\r\n]*<','match')' which gives me the output:

'>$ sudo apt-get update<' '>$ sudo apt-get upgrade<' '>$ sudo apt-get clean<' '>$ sudo apt-get autoclean<' '>$ sudo apt-get -f install<' '>$ sudo badblocks -v /dev/sda1<' '>$ sudo aptitude install debsums<' '>$ debsums <' '>$ sudo apt-get install firestarter<' '>$ sudo apt-get install youtube-dl<' '>$ youtube-dl http://uk.youtube.com/watch?v=r9OjoPskf_c<' '>$ ffmpeg -i r9OjoPskf_c.flv people_everyday.avi<' [1x97 char] '>$ sudo su | sudo apt-get install skype<' '>$ sudo jhas -jh 837afm$^&qeiuhn>>KOUUIG4n we8f-&hcjku8hujbn ok?<' '>$ sudo su | python setup.py install<' [1x75 char] '>$ sudo dpkg -i skysentials_1.0.1-1_all.deb<' '>$ konsole<' or maybe the matlab commands which all start with 2 > signs:

>> text_matlab=regexp(text,'>\>>[^<\r\n]*<','match')'

'>>> cf=0; %current frame<' '>> for k=10:50 % identity matrix from [10,10] to [50,50]<' '>> clf;<' '>> plot(fft(eye(k))) %plot<' '>> axis equal; axis off; axis([-1 1 -1 1]); % sort out axes<' '>> pause(0.01) %take a break<' '>> cf=cf+1; %update current frame<' [1x87 char] '>> end<' [1x94 char] '>>KOUUIG4n we8f-&hcjku8hujbn ok?<' '>>> convert my_image.jpg -resize 200% my_new_image.jpg<' '>>> direc=dir([pwd,filesep,'*.','jpeg']);<' '>>> filenames={};<' '>>> [filenames{1:length(direc),1}] = deal(direc.name);<' '>>> filenames=sortrows(char(filenames{:}));<' '>>> mkdir([pwd,filesep,'james_and_the_giant_peach'])<' '>>> mkdir([pwd,filesep,'thumblina'])<' '>>> for i=1:size(filenames,1)<' '>>> system(['convert ',deblank(filenames(i,:))...<' '>>> system(['convert ',deblank(filenames(i,:))...<' '>>> end<' '>>> url='http://www.mathworks.com/moler/ncm.tar.gz';<' '>>> gunzip(url,'ncm')<' '>>> untar('ncm/ncm.tar','ncm')<' '>>> cd([pwd,filesep,'ncm'])<' '>>> [U,G]=surfer('http://www.thedailydanielblog.blogspot.com',200);<' '>>> fid=fopen('dansweb.txt','wt');<' '>>> for i=1:size(char(U),1), fprintf(fid,'%s\n',char(U(i,:))); end<' '>>> fclose(fid)<' '>>> pagerank(U,G)<' and finally how to list the websites you refer to - I can't post the code into my blog because it involves html code which blogger doesn't like - replace '>\>>' with '< [the first letter of the alphabet] href'


enjoy!

The big clean

One of the beauties of a Linux system is that it is so easy to keep current. The software's free, and it tells you automatically if there are upgrades available. I probably upgrade my ubuntu hardy heron (8.04) OS every couple of days!

It's just as easy to clean, so there isn't loads of junk lying about everywhere - a problem which just kills windows machines after a while. I've had this OS since it came out (on the 4th month of 2008, hence 8.04) so I thought it was high time to have a bit of, well considering the decent run of weather we've been having here, a spring clean

First, how to upgrade and clean up the software packages. This is a crucial step for someone like me who spends a lot of time downloading and checking out new software

$ sudo apt-get update

$ sudo apt-get upgrade (the only thing I installed was 'replacement guidance-backends' - whatever they are!)

$ sudo apt-get clean - this removes cached packages, to free up disk space

$ sudo apt-get autoclean - this deletes partially downloaded packages and various bits of computer detritus

$ sudo apt-get -f install - this does a check for packages which might have broken whilst you tried to install them

This next bit takes much longer - it checks for 'bad blocks' on your hard drive - its like the 'disk defragmenter' tool in newer versions of Windows. Basically, it checks your disk (mine is /dev/sda1 but yours might be something else - run 'fdisk' to find out) for physical defects

$ sudo badblocks -v /dev/sda1

Now I don't know about you but I have some software which crashes a lot, or just doesn't seem to work at all. Run these commands if you're worried about malicious software or corrupt pacakges

$ sudo aptitude install debsums - this will install the necessary package 'debsums'

$ debsums - this will run the check (warning - if your computer is as full as mine, this will also take a very long time - allow an hour or so)

Linux doesn't suffer heavily from internet viruses and malware/spyware, but it is known for even linux machines to become infected and it is becoming more common. So here's how to install a firewall (on ubuntu):

$ sudo apt-get install firestarter - firestarter is a fancy interface to the more general linux 'iptables' program. To see whether your firewall is configured type sudo iptables -L into a shell. If the output doesn't have a list of rows starting with 'ACCEPT', it is not configured.

Then go to System>Administration>Firestarter - follow the options easy peasy! Make sure when it first starts to select the option, under 'Preferences', 'minimise to tray on close'

The firewall will run in the background and monitor and do what it does when you're surfing the web. To activate the firewall, type 'firestarter' into a shell before you browse

Ok - cleaned, defragmented, firewalled. And drought over! Man, that was rather professional!