Simple howtos

This collection of mini howtos is a dynamic copy of the Unix Toolbox. This page extracts the XML content directly from the original XHTML DOM and displays only the requested node. Source code here (use "save as").
Unix Toolbox revision 14.4
Copyright (c) 2007-2012 Colin Barschel. Some rights reserved under Creative Commons [Attribution - Share Alike]
System
Processes
File System
Network
SSH SCP
VPN with SSH
RSYNC
SUDO
Encrypt Files
Encrypt Partitions
SSL Certificates
CVS
SVN
Useful Commands
Install Software
Convert Media
Printing
Databases
Disk Quota
Shells
Scripting
Programming
Online Help

Useful Commands

less | vi | mail | tar | zip | dd | screen | find | Miscellaneous

less

The less command displays a text document on the console. It is present on most installation.
# less unixtoolbox.xhtml
Some important commands are (^N stands for [control]-[N]):

vi

Vi is present on ANY Linux/Unix installation (not gentoo?) and it is therefore useful to know some basic commands. There are two modes: command mode and insertion mode. The commands mode is accessed with [ESC], the insertion mode with i. Use : help if you are lost.
The editors nano and pico are usually available too and are easier (IMHO) to use.

Quit

Search and move

Delete copy paste text

mail

The mail command is a basic application to read and send email, it is usually installed. To send an email simply type "mail user@domain". The first line is the subject, then the mail content. Terminate and send the email with a single dot (.) in a new line. Example:
# mail c@cb.vu
Subject: Your text is full of typos
"For a moment, nothing happened. Then, after a second or so, 
nothing continued to happen."
.
EOT
#
This is also working with a pipe:
# echo "This is the mail body" | mail c@cb.vu
This is also a simple way to test the mail server.

tar

The command tar (tape archive) creates and extracts archives of file and directories. The archive .tar is uncompressed, a compressed archive has the extension .tgz or .tar.gz (zip) or .tbz (bzip2). Do not use absolute path when creating an archive, you probably want to unpack it somewhere else. Some typical commands are:

Create

# cd /
# tar -cf home.tar home/        # archive the whole /home directory (c for create)
# tar -czf home.tgz home/       # same with zip compression
# tar -cjf home.tbz home/       # same with bzip2 compression
Only include one (or two) directories from a tree, but keep the relative structure. For example archive /usr/local/etc and /usr/local/www and the first directory in the archive should be local/.
# tar -C /usr -czf local.tgz local/etc local/www
# tar -C /usr -xzf local.tgz    # To untar the local dir into /usr
# cd /usr; tar -xzf local.tgz   # Is the same as above

Extract

# tar -tzf home.tgz             # look inside the archive without extracting (list)
# tar -xf home.tar              # extract the archive here (x for extract)
# tar -xzf home.tgz             # same with zip compression (-xjf for bzip2 compression)
                                # remove leading path gallery2 and extract into gallery
# tar --strip-components 1 -zxvf gallery2.tgz -C gallery/ 
# tar -xjf home.tbz home/colin/file.txt    # Restore a single file
# tar -xOf home.tbz home/colin/file.txt    # Print file to stdout (no extraction)

More advanced

# tar c dir/ | gzip | ssh user@remote 'dd of=dir.tgz' # arch dir/ and store remotely.
# tar cvf - `find . -print` > backup.tar              # arch the current directory.
# tar -cf - -C /etc . | tar xpf - -C /backup/etc      # Copy directories
# tar -cf - -C /etc . | ssh user@remote tar xpf - -C /backup/etc      # Remote copy.
# tar -czf home.tgz --exclude '*.o' --exclude 'tmp/' home/

zip/unzip

Zip files can be easier to share with Windows.
# zip -r fileName.zip /path/to/dir                    # zip dir into file fileName.zip
# unzip fileName.zip                                  # uncompress zip file
# unzip -l fileName.zip                               # list files inside archive
# unzip -c fileName.zip fileinside.txt                # print one file to stdout (no extraction)
# unzip fileName.zip fileinside.txt                   # extract one file only

dd

The program dd (disk dump or destroy disk or see the meaning of dd) is used to copy partitions and disks and for other copy tricks. Typical usage:
# dd if=<source> of=<target> bs=<byte size> conv=<conversion>
# kill -INFO PID                                      # View dd progress (FreeBSD, OSX)
Important conv options: The default byte size is 512 (one block). The MBR, where the partition table is located, is on the first block, the first 63 blocks of a disk are empty. Larger byte sizes are faster to copy but require also more memory.

Backup and restore

# dd if=/dev/hda of=/dev/hdc bs=16065b                # Copy disk to disk (same size)
# dd if=/dev/sda7 of=/home/root.img bs=4096 conv=notrunc,noerror # Backup /
# dd if=/home/root.img of=/dev/sda7 bs=4096 conv=notrunc,noerror # Restore /
# dd bs=1M if=/dev/ad4s3e | gzip -c > ad4s3e.gz                  # Zip the backup
# gunzip -dc ad4s3e.gz | dd of=/dev/ad0s3e bs=1M                 # Restore the zip
# dd bs=1M if=/dev/ad4s3e | gzip | ssh eedcoba@fry 'dd of=ad4s3e.gz' # also remote
# gunzip -dc ad4s3e.gz | ssh eedcoba@host 'dd of=/dev/ad0s3e bs=1M'
# dd if=/dev/ad0 of=/dev/ad2 skip=1 seek=1 bs=4k conv=noerror    # Skip MBR
    # This is necessary if the destination (ad2) is smaller.
# dd if=/vm/FreeBSD-8.2-RELEASE-amd64-memstick.img of=/dev/disk1 bs=10240 conv=sync
    # Copy FreeBSD image to USB memory stick

Recover

The command dd will read every single block of the partition. In case of problems it is better to use the option conv=sync,noerror so dd will skip the bad block and write zeros at the destination. Accordingly it is important to set the block size equal or smaller than the disk block size. A 1k size seems safe, set it with bs=1k. If a disk has bad sectors and the data should be recovered from a partition, create an image file with dd, mount the image and copy the content to a new disk. With the option noerror, dd will skip the bad sectors and write zeros instead, thus only the data contained in the bad sectors will be lost.
# dd if=/dev/hda of=/dev/null bs=1m                   # Check for bad blocks
# dd bs=1k if=/dev/hda1 conv=sync,noerror,notrunc | gzip | ssh \ # Send to remote
root@fry 'dd of=hda1.gz bs=1k'
# dd bs=1k if=/dev/hda1 conv=sync,noerror,notrunc of=hda1.img    # Store into an image
# mount -o loop /hda1.img /mnt                        # Mount the image
# rsync -ax /mnt/ /newdisk/                           # Copy on a new disk
# dd if=/dev/hda of=/dev/hda                          # Refresh the magnetic state
  # The above is useful to refresh a disk. It is perfectly safe, but must be unmounted.

Delete

# dd if=/dev/zero of=/dev/hdc                         # Delete full disk
# dd if=/dev/urandom of=/dev/hdc                      # Delete full disk better
# kill -USR1 PID                                      # View dd progress (Linux)
# kill -INFO PID                                      # View dd progress (FreeBSD)

MBR tricks

The MBR contains the boot loader and the partition table and is 512 bytes small. The first 446 are for the boot loader, the bytes 446 to 512 are for the partition table.
# dd if=/dev/sda of=/mbr_sda.bak bs=512 count=1       # Backup the full MBR
# dd if=/dev/zero of=/dev/sda bs=512 count=1          # Delete MBR and partition table
# dd if=/mbr_sda.bak of=/dev/sda bs=512 count=1       # Restore the full MBR
# dd if=/mbr_sda.bak of=/dev/sda bs=446 count=1       # Restore only the boot loader
# dd if=/mbr_sda.bak of=/dev/sda bs=1 count=64 skip=446 seek=446 # Restore partition table

screen

Screen (a must have) has two main functionalities:

Short start example

start screen with:
# screen
Within the screen session we can start a long lasting program (like top).
# top
Now detach with Ctrl-a Ctrl-d. Reattach the terminal with:
# screen -R -D
In detail this means: If a session is running, then reattach. If necessary detach and logout remotely first. If it was not running create it and notify the user. Or:
# screen -x
Attach to a running screen in a multi display mode. The console is thus shared among multiple users. Very useful for team work/debug!

Screen commands (within screen)

All screen commands start with Ctrl-a. The screen session is terminated when the program within the running terminal is closed and you logout from the terminal.

Find

Some important options:
# find . -type f ! -perm -444        # Find files not readable by all
# find . -type d ! -perm -111        # Find dirs not accessible by all
# find /home/user/ -cmin 10 -print   # Files created or modified in the last 10 min.
# find . -name '*.[ch]' | xargs grep -E 'expr' # Search 'expr' in this dir and below.
# find / -name "*.core" | xargs rm   # Find core dumps and delete them (also try core.*)
# find / -name "*.core" -print -exec rm {} \;  # Other syntax
      # Find images and create an archive, iname is not case sensitive. -r for append
# find . \( -iname "*.png" -o -iname "*.jpg" \) -print -exec tar -rf images.tar {} \;
# find . -type f -name "*.txt" ! -name README.txt -print  # Exclude README.txt files
# find /var/ -size +10M -exec ls -lh {} \;     # Find large files > 10 MB
# find /var/ -size +10M -ls           # This is simpler
# find . -size +10M -size -50M -print
# find /usr/ports/ -name work -type d -print -exec rm -rf {} \;  # Clean the ports
      # Find files with SUID; those file are vulnerable and must be kept secure
# find / -type f -user root -perm -4000 -exec ls -l {} \;
# find flac/ -iname *.flac -print -size +500k -exec /Applications/Fluke.app/Contents/MacOS/Fluke {} \;
                                      # I use above to add flac files to iTunes on OSX
Be careful with xarg or exec as it might or might not honor quotings and can return wrong results when files or directories contain spaces. In doubt use "-print0 | xargs -0" instead of "| xargs". The option -print0 must be the last in the find command. See this nice mini tutorial for findhttp://www.hccfl.edu/pollock/Unix/FindCmd.htm.
# find . -type f | xargs ls -l       # Will not work with spaces in names
# find . -type f -print0 | xargs -0 ls -l  # Will work with spaces in names
# find . -type f -exec ls -l '{}' \; # Or use quotes '{}' with -exec
Duplicate directory tree:
# find . -type d -exec mkdir -p /tmp/new_dest/{} \;

Miscellaneous

# which command                      # Show full path name of command
# time command                       # See how long a command takes to execute
# time cat                           # Use time as stopwatch. Ctrl-c to stop
# set | grep $USER                   # List the current environment
# cal -3                             # Display a three month calendar
# date [-u|--utc|--universal] [MMDDhhmm[[CC]YY][.ss]]
# date 10022155                      # Set date and time
# whatis grep                        # Display a short info on the command or word
# whereis java                       # Search path and standard directories for word
# setenv varname value               # Set env. variable varname to value (csh/tcsh)
# export varname="value"             # set env. variable varname to value (sh/ksh/bash)
# pwd                                # Print working directory
# mkdir -p /path/to/dir              # no error if existing, make parent dirs as needed
# mkdir -p project/{bin,src,obj,doc/{html,man,pdf},debug/some/more/dirs}
# rmdir /path/to/dir                 # Remove directory
# rm -rf /path/to/dir                # Remove directory and its content (force)
# rm -- -badchar.txt                 # Remove file whitch starts with a dash (-)
# cp -la /dir1 /dir2                 # Archive and hard link files instead of copy
# cp -lpR /dir1 /dir2                # Same for FreeBSD
# cp unixtoolbox.xhtml{,.bak}        # Short way to copy the file with a new extension
# mv /dir1 /dir2                     # Rename a directory
# ls -1                              # list one file per line
# history | tail -50                 # Display the last 50 used commands
# cd -                               # cd to previous ($OLDPWD) directory
# /bin/ls| grep -v .py | xargs rm -r # pipe file names to rm with xargs
Check file hashes with openssl. This is a nice alternative to the commands md5sum or sha1sum (FreeBSD uses md5 and sha1) which are not always installed.
# openssl md5 file.tar.gz            # Generate an md5 checksum from file
# openssl sha1 file.tar.gz           # Generate an sha1 checksum from file
# openssl rmd160 file.tar.gz         # Generate a RIPEMD-160 checksum from file