Routine BASH memory aid

Some common things I do in BASH

Coordinate system conversion using cs2cs from PROJ

I hate geodesy. It's confusing, multifarious, and ever-changing (and I call myself a scientist!). But we all need to know where we, and more importantly our data, are. I have need of a single tool which painlessly does all my conversions, preferably without me having to think too much about it.

You could use an online tool, but when you're scripting (and I'm ALWAYS scripting) you want something a little less manual. Enter cs2cs from the PROJ.4 initiative. It will convert pretty much anything to anything else. Perfect.

In Ubuntu:
sudo apt-get install proj-bin

In Fedora I searched yum for proj-4

If the man page is a little too much too soon to take in, this wonderful page helps you decide which flags and parameters to use, with a rudimentary understanding of what you want

For example, if I want to convert coordinates in Arizona Central State Plane to WGS84 Latitude/Longitude, I use this:



(the State Plane Coordinate System is new to me, but my view of the whole field of geodesy is 'let's make things even more complicated, sorry, accurate!')

Let's decompose that a bit:



tells it I want decimal degrees with 6 values after the decimal point (you could leave this out if you want deg, min, sec)



says use the Transverse Mercator projection



are all parameters related to the input coordinates in Arizona Central State Plane (the 'to_meter' is a conversion from feet, the default unit, to metres)



are the parameters related to the output coordinates in Latitude/Longitude

infile is a two column list of points e.g.
2.19644131000e+005 6.11961823000e+005
2.17676234764e+005 6.11243565478e+005
2.19763457634e+005 6.11234534908e+005
2.19786524782e+005 6.11923555789e+005
2.19762476867e+005 6.11378246389e+005

outfile will be created with the results:
-111.846501    36.517758 0.000000
-111.868478    36.511296 0.000000
-111.845175    36.511202 0.000000
-111.844912    36.517412 0.000000
-111.845185    36.512498 0.000000

By default it will always give you the third dimension (height) even if you didn't ask for it, like above, so I tend to trim it by asking awk to give me just the first 2 columns separated by a tab:



-111.846501      36.517758
-111.868478      36.511296
-111.845175      36.511202
-111.844912      36.517412
-111.845185      36.512498

Raspberry pi launch program on boot, and from desktop

The following is an example of how to launch a program running in a terminal upon boot, in Raspbian wheezy:



and add:



This will allow the same program to be launched from the desktop upon double-click:



add

Compiling Clarkson's Hull on Fedora

A note on compiling Ken Clarkson's hull program for efficiently computing convex and concave hulls (alpha shapes) of point clouds, on newer Fedora

First, follow the fix described here which fixes the pasting errors given if using gcc compiler.

Then, go into hullmain.c and replace the line which says:



at the end of the function declarations (before the first while loop)

Finally, rewrite the makefile so it reads as below It differs significantly from the one provided.



Notice how the CFLAGS have been removed. Compile as root (sudo make)

Create Custom Launchers in Fedora

I just finished installed SmartGit on my new OS, like I did the last one

SmartGit is a simple bash script that I had to run from the command line. This time I decided I wanted a program launcher from my applications menu and my 'favourites bar' (or whatever that thing on the left hand side of the desktop is called!) 

So first I moved the smartgit folder from ~/Downloads/ to /usr/local/ and made the .sh script executable



Then to create a launcher I made a new .desktop file:



and added the following:

Of course the 'Icon' can be anything you like - I chose the one which comes with the program out of sheer non-inventiveness! In Applications it will appear under All and Accessories. Right click and 'Add to Favourites' and it will appear on your side bar.

Create a diary for your daily workflow

You can modify your ~./bash_history file so it doens't store duplicates and stores more lines than the default (only 1000):



However, sometimes you want to go back to your history file and see what commands you issued on a particular day .... one solution is to make a periodic backup with a timestamp using cron.

First, make a directory to put your files in, and make a script which is going to do the work:



Open the shell script and copy the following inside



close, then in the terminal issue


admittedly hourly is a little excessive - go here is you are unfamiliar with cron scheduling definitions and modify as you see fit

Installing Mb-System on Fedora 17

MB-System is the only fully open-source software I am aware of for swath sonar data including multibeam and sidescan backscatter. As such, I'm really hoping using it works out for me!

Compiling it on a fresh install of Fedora 17 wasn't that straight-forward (took me about 5 hours to figure all this out), so I thought I'd put together a little how-to for the benefit of myself, and hopefully you.

I've broken it down into a few stages so some sort of semblance of order can be imposed on this random collection of installation notes.

Step 1: download the software from ftp server here

In the terminal:

For me this created a folder called 'mbsystem-5.3.1982'

Step 2: install pre-requisites

A) Generic Mapping Tools (GMT). You need to go to the Downloads tab, then click on the INSTALL FORM link. This will take you to a page where you can download the installation script, and fill in an online form of your parameter settings which you can submit and save the resulting page as a textfile which becomes the input to the bash script. Sounds confusing, but it's not - the instructions on the page are adequate. I opted to allow GMT to install netCDF for me. Then in the terminal I did this:

B) Perl parallel forkmanager:

C) X11

D) openmotif

E) fftw

F) ghostview - I had to install this indirectly using kde3:

(Note to Fedora - why oh why oh why are B, C, and F above not installed by default!?)

G)  OTPSnc tidal prediction software: download from here

untar, and cd to the directory

first double check that ncdump and ncgen are installed (which ncdump ncgen)

then edit the makefile so it reads:

then in the terminal issue:

Hopefully this compiles without errors, then I moved them to a executable directory:

H) openGL libraries (again I had to do this indirectly, this time through mesa):

Step 3: prepare mbsystem makefiles

cd mbsystem-5.3.1982/

You have to go in and point install_makefiles to where all your libraries are. This is time-consuming and involves a lot of ls, which, and whereis!

Here's a copy of the lines I edited in my install_makefiles parameters:

Then in the terminal

Step 3: install mbsystem

first I had to do this (but you may not need to)

I then updated my ~/.bashrc so the computer can find all these lovely new files:

Test the install by issuing



you should see the manual pages come up. Right, now down to the business of multibeam analysis!

em1 to ethX on Fedora 17 (how to install matlab if you don't have eth)

I have just moved from Ubuntu 12.04 to Fedora 17. No reason other than necessity for my new job. I'm still happy with Ubuntu and even don't mind Unity!

First thing I wanted to do, of course, was install matlab. It turns out I had a bit of work to do to get it installed on my Dell Precision machine! So here's a post to help out anyone who has the same problem.

If you don't have some eth listed when you ifconfig (I had the NIC listed as em1), matlab will install but you can't activate it. The problem is described on their website  

Basically, Fedora udev names NICs (network cards) embedded on the motherboard as em1, em2 (etc) rather then eth0, eth1 (etc). More details here

The two solutions listed on the Mathworks for changing the names back to eth do not work in my Fedora install. The reason why the first one doesn't work is because there is no /etc/iftab file. the reason why the second one works is because udev rules are not listed in etc (they are in lib instead and have different names and structures)

Thankfully, Fedora has built in a mechanism to modify the names of NICs post install. You do it by editing the following file

/lib/udev/rules.d/71-biosdevname.rules

so the line which reads

GOTO="netdevicename_end"

commented by default, is uncommented

Then you need to edit
/etc/default/grub

by adding the following line to the bottom:

biosdevname=0

Then I backed up my ifcfg em1 network script by issuing

cp /etc/sysconfig/network-scripts/ifcfg-em1 /etc/sysconfig/network-scripts/ifcfg-em1.old

then I renamed it eth0 (the number can be anything, I believe, between 0 and 9, and matlab will figure it out when it activates):

mv /etc/sysconfig/network-scripts/ifcfg-em1 /etc/sysconfig/network-scripts/ifcfg-eth0

and edited ifcfg-eth0 so the line which read

DEVICE="em1"

to read:

DEVICE="eth0"

This is all the information the machine needs, when it boots up, to change the naming convention from em1 back over to eth0.

Upon reboot, ifconfig shows eth0 where em1 was before (dmesg | grep eth would also show the change), and matlab installs perfectly.

A satisfyingly geeky end to a Tuesday!

Restoring Nvidia graphics on Ubuntu

If you're an idiot like me and do this:

sudo apt-get --purge remove nvidia-*

You'll soon realise your display looks a little crappy, because you've turned off your graphics card drivers. Not to worry! You restore like this:

sudp apt-get install nvidia-*

then go into /etc/modprobe.d/blacklist.conf and remove any 'blacklists' to nvidia drivers, then restart the x-server using

sudo /etc/init.d/gdm restart

Normal service resumed, and you are free to make a new balls-up!

Wrapping and interfacing compiled code into more flexible and user-friendly programs using Zenity and Bash

Subtitles include:

1. How to reconstruct the three-dimensional shape of an object in a scene in a very inexpensive way using shadow-casting
2. How to write and execute a matlab script from a bash script

I've written a bash script - shadowscan.sh - which is a wrapper for a set of programs for the reconstruction 3D surface of an object in a series of photographs using the ingenious shadow-casting algorithm of Jean-Yves Bouguet and Pietro Perona

The script relies heavily on the GUI-style interfacing tools of Zenity and acts as an interface and wrapper for the algorithm as implemented by these guys. It will also write and execute a matlab script for visualising the outputs.

This script should live in a directory with the following compiled programs, all available here:
1. ccalib (performs the camera calibration to retrieve the intrinsic parameters of camera and desk)
2. find_light (performs the light calibration to find light-source coordinates)
3. desk_scan (does the 3d reconstruction)
4. merge (merges 2 3d reconstructions, e.g. from different angles)

and the following matlab script:
1. selectdata.m (available here)

When the makefile is run, it creates the compiled programs in the ./bin directory. It's important to consult the README in the main folder as well as within the 'desk_scan' folder to have a clear idea of what the program does and what the input parameters mean.

My program will check if the camera calibration and light calibration files exist in the folder and if not will carry them out accordingly then several reconstructions are carried out, and the outputs merged. Finally matlab is called from the command line to run 'p_lightscan_data.m' to visualize the results.

when the script finishes there will be the following output files:
1. xdim.txt - max dimension to scan to
2. params - camera calibration parameters
3. data - camera calibration data
4. light.txt - light calibration parameters
5. sample.pts - [x,y,z] of object, in coordinates relative to light source
6. sample_lightscan1.tif - filled colour contour plot of reconstructed object
7. sample_lightscan2.tif - mesh plot of reconstructed object

I've tried to keep it flexible. It will run only the elements it needs to. For example, if the camera or lighting has been carried out before for a different object it will pick up on that. Also, the program can be invoked by passing it two arguments (path to folder where sample images are, and what sequence of contrasts to use for the reconstruction), which avoids having to input with the graphical menus. Example usage:
1. bash shadowscan.sh
2. bash shadowscan.sh /home/daniel/Desktop/shadow_scanning/sample/bottle 30.35.40.45

Controlling a serial device with Python

This program to sends a command to serial device, receives data from the device, converts it to numeric format, and graphs the data in real time. It is written in Python using wxPython for the display.

The device is connected to /dev/ttyS0 with a baud rate of 19200, 8-bit data, no parity, and 1 stopbit. You need to edit the line 'serial.Serial' with the relevant parameters to use this code for your device. Editing 'DATA_LENGTH' will change the rate at which the graph scrolls forward. Also, you'd need to change the 'command' that the device understands as 'send me data' (here it is the s character - line 'input'). Knowledge of the format of the returned data is also essential (here I turn ascii into 2 integers) line 'struct.unpack' - see here for help). Otherwise this should work with any device. Tested on Ubuntu 9.10 and 10.4 using serial port and USB-serial converters.

Using cron to backup Thunderbird emails

My bash script (which I've saved as backup_emails.sh in my home folder and made executable) looks a little like this:

 set -x   
  x=`date +%y%m%d.%H%M`   
  tar zcf thunderb-mail-${x}.tgz ~/.mozilla-thunderbird   
  mv thunderb-mail-${x}.tgz /media/MY_HARDDRIVE/email_backups/   

Then invoke in cron using:

 crontab -e   

add something like:

 30 4 * * * ~/./backup_emails.sh  

which does the backup at 04.30 every day

Make Google Earth Faster on Ubuntu

In terminal type:

 export LIBGL_ALWAYS_INDIRECT=true  

Convert movies to be compatible with ZEN X-Fi media player

Creative - another company which doesn't provide support for non-Windows users. Get with the times, losers!

The ZEN player needs a specific format of movie (bit-rate and aspect ratio). Here's how to do it with mencoder (available for free on all platforms)

 mencoder 'MyMovie.avi' -oac mp3lame -lameopts cbr:mode=2:br=96 -af resample=44100 -srate 44100 -ofps 20 -ovc lavc -lavcopts vcodec=mpeg4:mbd=2:cbp:trell:vbitrate=300 -vf scale=320:240 -ffourcc XVID -o MyMovie4zen.avi  

Updated Power Spectrum from an Image

A while back I posted on using ImageMagick to do a full frequency-decomposition on an image, and create a power spectrum. It had several inaccuracies. Here's an update with how to do it better

First, you need to install a version of IM which is HDRI enabled (see here)

# test for valid version of Image Magick

 im_version=`convert -list configure | \  
  sed '/^LIB_VERSION_NUMBER /!d; s//,/; s/,/,0/g; s/,0*\([0-9][0-9]\)/\1/g'`  
 [ "$im_version" -lt "06050407" ] && errMsg "--- REQUIRES IM VERSION 6.5.4-7 OR HIGHER ---"  

# test for hdri enabled

 hdri_on=`convert -list configure | grep "enable-hdri"`  
 [ "$hdri_on" = "" ] && errMsg "--- REQUIRES HDRI ENABLED IN IM COMPILE ---"  

#read $file, load and converting to greyscale

 convert $file -colorspace gray in_grey.tiff  

# crop image to smallest dimension

 min=`convert in_grey.tiff -format "%[fx: min(w,h)]" info:`  

# crop image, and convert to png (why?)

 convert in_grey.tiff -background white -gravity center -extent ${min}x${min} in_grey_crop.png  

# get data, find mean as proportion, convert mean to percent, and de-mean image

 data=`convert in_grey_crop.png -verbose info:`  
 mean=`echo "$data" | sed -n '/^.*[Mm]ean:.*[(]\([0-9.]*\).*$/{ s//\1/; p; q; }'`  
 m=`echo "scale=1; $mean *100 / 1" | bc`  
 convert in_grey_crop.png -evaluate subtract ${m}% in_grey_crop_dm.png  

# pad to power of 2

 p2=`convert in_grey_crop_dm.png -format "%[fx:2^(ceil(log(max(w,h))/log(2)))]" info:`  
 convert in_grey_crop_dm.png -background white -gravity center -extent ${p2}x${p2} in_grey_crop_dm_pad.png  

# fft

 convert in_grey_crop_dm_pad.png -fft in_ft.png  

# get spectrum

 convert in_ft-0.png -contrast-stretch 0 -evaluate log 10000 in_spec.png  

... but it's still not quite right. Any tips?

Turn your OGV movie section into a cropped animated GIF

An example from a recent project

 # make directory for output jpg  
 mkdir out_pics   
 # section to the ogv video file into jpegs  
 mplayer -ao null -ss 00:00:03 -endpos 50 beautiful_movie.ogv -vo jpeg:outdir=out_pics  
 cd out_pics  
 # loop over each image and crop into desired rectangle  
 for image in *.jpg  
 do  
 convert $image -crop 600x600+350+135 "$image crop.tiff"  
 done  
 # turn into an animated gif  
 convert -page 600x600+0+0 -coalesce -loop 0 *.tiff beautiful_animation.gif  

Miscellaneous useful bash

For my benefit more than anything - things I use over and over again, but do not have the mental capacity to remember

# number of files in current directory

 ls -1R | grep .*.<file extension> | wc -l  

# check if you have a program installed

 aptitude show <name of program> | grep State  

# only list the (sub-)directories in your current directory

 ls -l | grep ^d  

# remove backup files

 find ./ -name '*~' | xargs rm  

# use zenity get the directory where the file you want is

 zenity --info --title="My Program" --text="Select  
 which directory contains the file(s) you want"  
 export target_dir=`zenity --file-selection --directory`  

# list all txt files in this directory

 x=$( ls $target_dir | grep .*.txt |sort|zenity --list --title="Select" --column="coordinate file" )  

# get this directory

 export raw_dir=`zenity --file-selection --directory`  

# removes .txt (for example, for later use as start of filename)

 str="$x"  
 Var1=`echo $str | awk -F\. '{print $1}'`  

# gets the rest of the file name

 str="$Var1"  
 Var2=`echo $str | awk -F\- '{print $2}'`   

# turn string into (indexable) array

 e=$(echo $mystring | sed 's/\(.\)/\1 /g')  
 b=($e)  

# get max and min dimensions of an image

 min=`convert image1.tiff -format "%[fx: min(w,h)]" info:`  
 max=`convert image1.tiff -format "%[fx: max(w,h)]" info:`  

# take away 1 from both

 min=$((min-1))   
 max=$((max-1))  

Quick tips for Ubuntu 9.04

One of the beauties of starting your OS afresh is, well, you get a new and better OS. Ubuntu 9.4 (or 'jaunty jackalope') is a slick version of my previous Ubuntu 8.04 ('hardy heron'). So far, I've ascertained that it looks better, it has better support for hardware, and it boots really fast.

One of the pains of starting your OS afresh is having to reinstall all your software and set your preferences. I thought I'd put a little 'as I do it' guide which may take some of the pain away. This list has a few of the 'biggies' and maybe a couple of items more suited to my personal taste, but what the hey!

1. configure the firewall
sudo apt-get install gufw

2. install wine (windows emulator, for those windows programs you cant live without - an ever decreasing number horray!)
sudo apt-get install wine

3. This one is crucial if you want to run java - it takes ages, and at the end when it installs java, you need to be able to use the tab and 'pg dwn' keys to get to the end of the license agreement
sudo apt-get install ubuntu-restricted-extras

4. VLC is still the best media player around
sudo apt-get install vlc

5. skype -
sudo wget http://www.medibuntu.org/sources.list.d/jaunty.list -O /etc/apt/sources.list.d/medibuntu.list

wget -q http://packages.medibuntu.org/medibuntu-key.gpg -O- | sudo apt-key add - && sudo apt-get update

sudo aptitude install skype

6. Google Earth (version 5) - install the binary from here. Right click - Properties - Permissions - make executable. Then double click, and it will install - easy! It will create a desktop icon which, by default, you cannot see because you do not have permission. In the terminal type (here, replace 'daniel' with your username):

cd ./Desktop
sudo chown daniel:daniel Google-googlearth.desktop

7. Expand the capabilities of nautilus file browser

sudo apt-get install nautilus-actions nautilus-gksu nautilus-image-converter nautilus-open-terminal nautilus-script-audio-convert nautilus-script-collection-svn nautilus-script-manager nautilus-sendto nautilus-share nautilus-wallpaper

8. Adobe AIR

wget http://airdownload.adobe.com/air/lin/download/latest/AdobeAIRInstaller.bin
chmod +x ./AdobeAIRInstaller.bin
./AdobeAIRInstaller.bin

9. Adobe acrobat reader

sudo gedit /etc/apt/sources.list

Add the below line in sources.list

deb http://archive.canonical.com/ubuntu jaunty partner

sudo apt-get update
sudo apt-get install acroread

G'night!

Ubuntu Beautification

Tip #1:

It's amazing really - most fonts contain within them “hints” laid down by their designer about how they should look on-screen. Artistic meta-data! However, Ubuntu ignores them and uses a system called 'autohinting'.

It works well, but you might also want to try bytecode hinting. This uses the hinting built into the fonts. To activate bytecode hinting, open a terminal window and type:

$ sudo dpkg-reconfigure fontconfig-config

Using the cursor keys, select Native from the menu and then hit Enter.

On the next screen you’ll be asked if you want to activate subpixel rendering. This is good for TFT screens, so make the choice (or just select Automatic). Next you’ll be asked if you want to activate bitmap fonts, which are non-true type fonts good for use at low point levels. There’s no harm in using them, so select yes. The program will quit when it’s finished. Once that’s happened, type the following to write the changes to the system and update files:

$ sudo dpkg-reconfigure fontconfig
$ sudo dpkg-reconfigure defoma

System - Quit - Logout, and then log back in again. Letters will appear more rounded and the antialiasing will appear better.

Tip #2:

465 fonts of Brian Kent (http://www.aenigmafonts.com) are available to Ubuntu users. To install the fonts, you’ll need to add a new software repository:

System - Administration - Software Sources
- Third-Party Software tab - Add

type the following into the dialog box that appears:

deb http://ppa.launchpad.net/corenominal/ubuntu hardy main

Click 'Add Sources', then Close and, when prompted, agree to reload the package lists. Then use Synaptic to search for and install the 'ttf-aenigma' package. Once installed the fonts will be available for use straight away in all applications.

Enjoy your beautified Ubuntu!

How to get on youtube, twitter and myspace quicker

Well, the easiest way is save time when you boot your computer!

Whenever Ubuntu boots it runs several scripts that start necessary background services. By default these are set to run one-by-one but if you have a processor with more than one core, such as Intel’s CoreDuo series or AMD’s Athlon X2, you can configure Ubuntu to run the scripts
in parallel. This way all the cores are utilized and quite a bit of time can be saved at each boot. To make the change, type the following to open the necessary configuration file in Gedit:

$ gksu gedit /etc/init.d/rc

Look for the line that reads CONCURRENCY=none and change it so it reads CONCURRENCY=shell.

Then save the file and reboot your computer. This should save you several seconds of time every time you boot up, and you can go on being a handsome speedy devil