Simple howtos

This collection of mini howtos is a dynamic copy of the Unix Toolbox. This page extracts the XML content directly from the original XHTML DOM and displays only the requested node. Source code here (use "save as").
Unix Toolbox revision 14.4
Copyright (c) 2007-2012 Colin Barschel. Some rights reserved under Creative Commons [Attribution - Share Alike]
System
Processes
File System
Network
SSH SCP
VPN with SSH
RSYNC
SUDO
Encrypt Files
Encrypt Partitions
SSL Certificates
CVS
SVN
Useful Commands
Install Software
Convert Media
Printing
Databases
Disk Quota
Shells
Scripting
Programming
Online Help

Scripting

Basics | Script example | awk | sed | Regular Expressions | useful commands

The Bourne shell (/bin/sh) is present on all Unix installations and scripts written in this language are (quite) portable; man 1 sh is a good reference.

Basics

Variables and arguments

Assign with variable=value and get content with $variable
MESSAGE="Hello World"                        # Assign a string
PI=3.1415                                    # Assign a decimal number
N=8
TWON=`expr $N * 2`                           # Arithmetic expression (only integers)
TWON=$(($N * 2))                             # Other syntax
TWOPI=`echo "$PI * 2" | bc -l`               # Use bc for floating point operations
ZERO=`echo "c($PI/4)-sqrt(2)/2" | bc -l`
The command line arguments are
$0, $1, $2, ...                              # $0 is the command itself 
$#                                           # The number of arguments
$*                                           # All arguments (also $@)

Special Variables

$$                                           # The current process ID
$?                                           # exit status of last command
  command
  if [ $? != 0 ]; then
    echo "command failed"
  fi
mypath=`pwd`
mypath=${mypath}/file.txt
echo ${mypath##*/}                           # Display the filename only
echo ${mypath%%.*}                           # Full path without extention
foo=/tmp/my.dir/filename.tar.gz
path = ${foo%/*}                             # Full path without extention
var2=${var:=string}                          # Use var if set, otherwise use string
                                             # assign string to var and then to var2.
size=$(stat -c%s "$file")                    # get file size in bourne script
filesize=${size:=-1}

Constructs

for file in `ls`
do
    echo $file
done

count=0
while [ $count -lt 5 ]; do
    echo $count
    sleep 1
    count=$(($count + 1))
done

myfunction() {
    find . -type f -name "*.$1" -print       # $1 is first argument of the function
}
myfunction "txt"

Generate a file

MYHOME=/home/colin
cat > testhome.sh << _EOF
# All of this goes into the file testhome.sh
if [ -d "$MYHOME" ] ; then
    echo $MYHOME exists
else
    echo $MYHOME does not exist
fi
_EOF
sh testhome.sh

Bourne script example

As a small example, the script used to create a PDF booklet from this xhtml document:
#!/bin/sh
# This script creates a book in pdf format ready to print on a duplex printer
if [ $# -ne 1 ]; then                        # Check the argument
  echo 1>&2 "Usage: $0 HtmlFile"
  exit 1                                     # non zero exit if error
fi

file=$1                                      # Assign the filename
fname=${file%.*}                             # Get the name of the file only
fext=${file#*.}                              # Get the extension of the file

prince $file -o $fname.pdf                   # from www.princexml.com
pdftops -paper A4 -noshrink $fname.pdf $fname.ps # create postscript booklet
cat $fname.ps |psbook|psnup -Pa4 -2 |pstops -b "2:0,1U(21cm,29.7cm)" > $fname.book.ps

ps2pdf13 -sPAPERSIZE=a4 -sAutoRotatePages=None $fname.book.ps $fname.book.pdf
                                             # use #a4 and #None on Windows!
exit 0                                       # exit 0 means successful

Some awk commands

Awk is useful for field stripping, like cut in a more powerful way. Search this document for other examples. See for example gnulamp.com and one-liners for awk for some nice examples.
awk '{ print $2, $1 }' file                  # Print and inverse first two columns
awk '{printf("%5d : %s\n", NR,$0)}' file     # Add line number left aligned
awk '{print FNR "\t" $0}' files              # Add line number right aligned
awk NF test.txt                              # remove blank lines (same as grep '.')
awk 'length > 80'                            # print line longer than 80 char)

Some sed commands

Here is the one liner gold minehttp://student.northpark.edu/pemente/sed/sed1line.txt. And a good introduction and tutorial to sedhttp://www.grymoire.com/Unix/Sed.html.
sed 's/string1/string2/g'                    # Replace string1 with string2
sed -i 's/wroong/wrong/g' *.txt              # Replace a recurring word with g
sed 's/\(.*\)1/\12/g'                        # Modify anystring1 to anystring2
sed '/<p>/,/<\/p>/d' t.xhtml                 # Delete lines that start with <p>
                                             # and end with </p>
sed '/ *#/d; /^ *$/d'                        # Remove comments and blank lines
sed 's/[ \t]*$//'                            # Remove trailing spaces (use tab as \t)
sed 's/^[ \t]*//;s/[ \t]*$//'                # Remove leading and trailing spaces
sed 's/[^*]/[&]/'                            # Enclose first char with [] top->[t]op
sed = file | sed 'N;s/\n/\t/' > file.num     # Number lines on a file

Regular Expressions

Some basic regular expression useful for sed too. See Basic Regex Syntaxhttp://www.regular-expressions.info/reference.html for a good primer.
[\^$.|?*+()                          # special characters any other will match themselves
\                                    # escapes special characters and treat as literal
*                                    # repeat the previous item zero or more times
.                                    # single character except line break characters
.*                                   # match zero or more characters
^                                    # match at the start of a line/string
$                                    # match at the end of a line/string
.$                                   # match a single character at the end of line/string
^ $                                  # match line with a single space
[^A-Z]                               # match any line beginning with any char from A to Z

Some useful commands

The following commands are useful to include in a script or as one liners.
sort -t. -k1,1n -k2,2n -k3,3n -k4,4n         # Sort IPv4 ip addresses
echo 'Test' | tr '[:lower:]' '[:upper:]'     # Case conversion
echo foo.bar | cut -d . -f 1                 # Returns foo
PID=$(ps | grep script.sh | grep bin | awk '{print $1}')    # PID of a running script
PID=$(ps axww | grep [p]ing | awk '{print $1}')             # PID of ping (w/o grep pid)
IP=$(ifconfig $INTERFACE | sed '/.*inet addr:/!d;s///;s/ .*//')   # Linux
IP=$(ifconfig $INTERFACE | sed '/.*inet /!d;s///;s/ .*//')        # FreeBSD
if [ `diff file1 file2 | wc -l` != 0 ]; then [...] fi       # File changed?
cat /etc/master.passwd | grep -v root | grep -v \*: | awk -F":" \ # Create http passwd
'{ printf("%s:%s\n", $1, $2) }' > /usr/local/etc/apache2/passwd

testuser=$(cat /usr/local/etc/apache2/passwd | grep -v \    # Check user in passwd
root | grep -v \*: | awk -F":" '{ printf("%s\n", $1) }' | grep ^user$)
:(){ :|:& };:                                # bash fork bomb. Will kill your machine
tail +2 file > file2                         # remove the first line from file
I use this little trick to change the file extension for many files at once. For example from .cxx to .cpp. Test it first without the | sh at the end. You can also do this with the command rename if installed. Or with bash builtins.
# ls *.cxx | awk -F. '{print "mv "$0" "$1".cpp"}' | sh
# ls *.c | sed "s/.*/cp & &.$(date "+%Y%m%d")/" | sh # e.g. copy *.c to *.c.20080401
# rename .cxx .cpp *.cxx                             # Rename all .cxx to cpp
# for i in *.cxx; do mv $i ${i%%.cxx}.cpp; done      # with bash builtins