DIRECTORY STRUCTURE IN UNIX
| Directory | Contains |
| bin/ | Required Boot-time binaries |
| boot/ | Boot configuration files for the OS loader and kernel image |
| dev/ | Device files |
| etc/ | System configuration files and scripts |
| home/ | User/Sub branch directories |
| lib/ | Main OS shared libraries and kernel modules |
| Lost+found/ | Storage directory for “recovered” files |
| mnt/ | Temporary point to connect devices to |
| proc/ | Pseudo directory structure containing information about the kernel, currently running processes and resource allocation |
| root/ | Linux (non-standard) home directory for the root user. Alternate location being the / directory itself |
| sbin/ | System administration binaries and tools |
| tmp/ | Location of temporary files |
| usr/ | Difficult to define – it contains almost everything else including local binaries, libraries, applications and packages (including X Windows) |
| var/ | Variable data, usually machine specific. Includes spool directories for mail and news |
GENERAL UNIX COMMANDS
TO SEE WHERE COMMANDS ARE LOCATED
echo $PATH
/bin:/usr/bin:/usr/local/bin:$HOME/bin
which ls
SEEING HELP COMMAND
date –help
info date
man date
whatis date
Extended Documents:=/usr/share/doc
TO CHANGE CONSOLE
Ctrl+alt+(F1-F7)
CHANGING DIRECTORY
cd /home/deb/work
cd ..[One level up]
LISTING DIRECTORIES AND FILES
ls /
ls –al [All hidden files]
ls –l [long listing]
ls –R[recurses through sub-directories]
ls -d[Listing directories only]
ls -ltr|grep ^d[Listing directories only]
List of command options
Option Action
-a list hidden files
-d list the name of the current directory
-F show directories with a trailing ’/’ executable
files with a trailing ’*’
-g show group ownership of file in long listing
-i print the inode number of each file
-l long listing giving details about files and direc-tories
-R list all subdirectories encountered
-t sort by time modified instead of name
COPYING FILES
cp source destination
cp –P source destination [Preserves permissions]
cp –R source destination [Recursively copies the files]
cp ~helper/tmp/for_john tmp/comments[To copy a file from another user’s directory using a relative
pathname.]
cp /usr/lib/more.help .[To copy a file from another directory to the current working
directory, preserving the file name:]
MOVING AND RENAMING
mv file1 file2 [ Renaming]
mv source destination [ Move]
REMOVING FILES AND DIRECTORIES
rm chap[1-2][wildcard is used]
rm -i chap1[Removing interactively]
rm -rf[Deleting forefully file or directories]
rmdir deb[removing empty directory]
TO CHECK FILE TYPES
file deb.sh
Displaying the first few lines of a file
head /usr/local/doc/vi.txt[To display the first 10 lines of a file]
head -5 *.xdh[To display the first 5 lines of several files]
Displaying the last part of a file
tail fred[To display the last 10 lines of a text file]
tail -50c help.txt[To display the last 50 characters in a file]
tail -c -512 foo[Copies last 512 bytes from foo]
tail -c +512 foo[Copies everything after skippiong 512 bytes]
HANDLING PRINTERS
lp deb.lis[Printing file]
lp -d laser deb.lis[If there are more printers]
lp -n3 deb.lis[Printing 3 copies]
lpstat[To view print queue]
cancel laser[To cancel printing of current printer]
cancel pr1-320[Cancel job with request id pr1-320]
lpq[To see print que job number]
lprm 31[Removes job number 31]
lpr -p laser deb.lis[Prints on printer laser]
lpstat -c[To see pending jobs]
COUNTING LINES
wc deb.txt
wc -c[No. of characters]
wc -l[No. of lines]
COMPRESSING FILES
compress deb.txt
uncompress deb.txt
gzip deb
gunzip deb.z
USING TAR COMMAND
# display contents of a file
tar tvf myfile.tar
# display contents of a diskette (Solaris)
volcheck
tar tvf /vol/dev/rdiskette0/unnamed_floppy
# copy files to a tar file
tar cvf myfile.tar *.sql
# format floppy, and copy files to it (Solaris)
fdformat -U -b floppy99
tar cvf /vol/dev/rdiskette0/floppy99 *.sql
# append files to a tar file
tar rvfn myfile.tar *.txt
# extract files from a tar filem to current dir
tar xvf myfile.tar
SORTING
sort /etc/passwd
sort in order based on the whole line
sort -r /etc/passwd
reverse the order of the sort
sort +2n -t: /etc/passwd
sort on third field, where field delimiter is : (skip the first two fields)
sort +2n -t: -n /etc/passwd
same sort but treat the field as numbers not ASCII characters
sort –o telnos telnos(sorts telnos and replaces its content with the sorted output)
DUPLICATE ROW BY UNIQUE COMMAND
uniq names
remove duplicate lines from names and display them on the screen
uniq names uniq.names
remove duplicates lines from names and put them into uniq.names
uniq -d names
display all duplicate lines
Using symbolic links
ln -s source linkname
ln -s reports/reportA publications/my_report
TO DETERMINE DISK USAGE
# display disk free, in KB
df -kt
# display disk usage, in KB for directory
du -k mydir
# display directory disk usage, sort by largest first
du -ahx .|sort -rh|head -20
FIND Command
Find files by text search
$ find ~ -name “*.txt” -print
To find all ‘*.txt’ files in the current directory you are in and any sub-directories:
$ find . -name “*.txt” -print
To find all filenames with at least one upper case character, in your current directory
and any sub-directories:
$ find . -name “[A-Z]*” -print
To find files in /etc directory that begin with host, and any other characters after
that:
$ find /etc -name “host*” -print
To find all files in your $HOME directory:
$ find ~ -name “*” -print or find . -print
To bring the system to zero per cent response time, start at the root level and drill
through all directories listing ‘all’ files. If you want to remain on good terms with
your sys admin be very careful with this option!
$ find / -name “*” -print
Find all files that begin with two lower case characters, followed by two numbers,
followed by .txt. The following find command could return a file called ax37.txt.
$ find . -name “[a-z][a-z][0–9][0–9].txt” -print
Find files by perm mode
To find files with a permission of 755 which is read, write and executable by
owner and read and execute by everyone else,
$ find . -perm 755 -print
To find files that are read, write and executable for everyone (watch out for these) use
this find command. Put a dash in front of the octal number. The perm stands for
permissions and 007 is based on the notation you use for the chmod (absolute) mode.
$ find . -perm -007 -print
Ignoring directories
To display the files in the apps directory when you do not want find to search
in /apps/bin:
$ find /apps -name “/apps/bin” -prune -o -print
Find files by user and nouser
To find files by owner, you supply the actual login name of that user. To find files
owned by user dave in your $HOME directory:
$ find ~ -user dave -print
To find files owned by uucp in /etc:
$ find /etc -user uucp -print
To find files of a user that has been deleted, use the -nouser option. This
finds files that have no valid userid in the /etc/passwd file. You do not have to
supply a user name with this option; find will go off and do all the work. To find
all files not owned by a current user in the /home directory tree:
$ find /home -nouser -print
Find files by group and nogroup
Like the user and nouser option, find has the same options for groups. To find all
files with the group membership of ‘accts’ that are in the /apps directory:
$ find /apps -group accts -print
To find all files that have no valid group membership just supply the -nogroup
option. Here find starts at the root of the file system.
$ find / -nogroup -print
Find files by modification times
To find all files that have been modified in the last five days:
$ find / -mtime -5 -print
To find files in /var/adm directory that have not been modified in the last three
days:
$ find /var/adm -mtime +3 -print
Finding files that are newer or older in days or minutes
$ find . -newer age.awk ! -newer belts.awk -exec ls -l {} \;
Delete old trace files from DIAG location of Oracle Database
find /u01/app/oracle -type f -name “*.trc” -mtime +1 -exec rm {} \;
find /u01/app/oracle -type f -name “*.trm” -mtime +0.5 -exec rm {} \;
find /u01/app/oracle -type f -name “*.aud” -mtime +0.5 -exec rm {} \;
VI EDITOR
List of insert commands
To do this … Command
Insert text after the cursor a
Insert text before the cursor i
Append text at the end of the current line A
Insert text at the start of the current line I
Open a new line above the current line o
Open a new line below the current line O
Recovering files after a system crash
vi -r help.xdh
Viewing a file
view filename
Moving the cursor along a line
Press the ESC key to enter command mode before using these com-mands.
To move to Do this …
next character l
previous character h
next word w
next n words wn
previous word b
previous n words bn
end of current word e
start of current line 0 (zero)
end of current line $
Moving the cursor between lines
Press the ESC key to enter command mode before using these com-mands.
To move to Do this …
next line down j
(same column)
start of next line down +
previous line k
(same column)
start of previous line
Deleting characters
Press the ESC key to enter command mode before using these com-mands.
To delete Do this
current character x
previous character dh
Deleting words and lines
Press the ESC key to enter command mode before using these com-mands.
To delete Do this
current word dw
previous word db
entire line dd
to end of line d$
to start of line d0 (zero)
next n lines ndd
To get back a word or line that you have just deleted enter the command:
p
Searching for text
Press the ESC key to enter command mode before using these com-mands.
To search Do this …
forward for a pattern /pattern
backward for a pattern ?pattern
repeat previous search n
repeat previous search in N
reverse direction
Examples of replacing text
To replace one word in the current line with another:
:s/that/which
To replace every instance of one word in the current line with
another:
:s/destroy/delete/g
This replaces every occurrence of the word “destroy” with the
word “delete” in the current line.
To replace every occurrence of a word throughout the file, with
another:
:g/insert/s//add/g
This replaces every occurrence of “insert” with “add”.
To replace ^M from text file
:%s/^v^M/ /g
UNIX Scheduler and Process
CRONTAB
1st column Minutes 1–59
2nd column Hour 1–23 (0 is midnight)
3rd column Day_of_month 1–31
4th column Month 1–12
5th column Weekday 0–6 (0 is Sunday)
6th column Command to run
Here are some examples of crontab entries:
30 21 * * * /apps/bin/cleanup.sh
runs a script called cleanup.sh in /apps/bin at 9:30 every night.
Cron and crontab 33
45 4 1,10,22 * * /apps/bin/backup.sh
runs a script called backup.sh in /apps/bin at 4:45 a.m., on the 1st, 10th and 22nd
of each month.
10 1 * * 6,0 /bin/find -name “core” -exec rm {} \;
runs a find command at 1:10 a.m. only on Saturday and Sunday.
0,30 18-23 * * * /apps/bin/dbcheck.sh
runs a script called dbcheck.sh in /apps/bin at every 30 minutes past the hour,
between 18:00 and 23:00.
0 23 * * 6 /apps/bin/qtrend.sh
runs a script called qtrend.sh in /apps/bin at 11:00 p.m. every Saturday.
The general format of the crontab is:
Crontab [-u user] -e -l -r
where:
-u is the user login name
-e edits the crontab file
-l lists the crontab file
-r deletes the crontab file
Submitting a command to the background
$ find /etc -name “srm.conf” -print >find.dt 2>&1 &
Checking the process using ps
$ ps x|grep 28305
28305 p1 S 0:00 sh /root/ps1
28350 p1 S 0:00 grep 28305
Killing a background job
kill -signal [process_number]
$ kill 28305
$ kill -9 28305
Submitting a job using nohup
nohup command >myout.file 2>&1
$ nohup ps1 &
CONTROLLING ACCESS TO FILE SYSTEM
1.To display the permissions on a single file:
ls -l file1
-rw-r–r– 2 unixjohn 3287 Apr 8 12:10 file1
2.This displays the following information about the file file1.
-rw-r–r– – access permissions
2 – number of links to this file
unixjohn – owner
3287 – size in bytes
Apr 8 12:10 – date and time last modified
3.There are three types of permissions:
r – read the file or list files in the directory
w – write to the file or directory
x – execute the file or search the directory
4.Each of these permissions can be set for any one of three types of
user:
u – the user who owns the file (usually you)
g – members of the group the owner belongs to
o – all other users
The access permissions for all three types of user can be given as a
string of nine characters:
user group others
r w x r w x r w x
5.Changing permission
- To give yourself permission to execute a file that you own:
chmod u+x file1
This gives you execute permission for the file file1.
- To give members of your group permission to read a file:
chmod g+r file2
This gives the group permission to read the file file2.
- To give read permission to everyone for a particular type of file:
chmod a+r *.pub
This gives everyone permission to read all files with the extension
.pub.
- To give the group write and execute permission:
chmod g+wx $HOME/SCCS
This gives all members of the group permission to place files in
5.Setting access permissions numerically
There is a shorthand way of setting permissions by using octal num-bers.
Read permission is given the value 4, write permission the value
2 and execute permission 1.
r w x
4 2 1
These values are added together for any one user category:
1 = execute only
2 = write only
3 = write and execute (1+2)
4 = read only
5 = read and execute (4+1)
6 = read and write (4+2)
7 = read and write and execute (4+2+1)
So any access permission can be expressed as three digits. For exam-ple:
user group others
chmod 640 file1 rw- r– —
chmod 754 file1 rwx r-x r–
chmod 664 file1 rw- rw- r–
6.Creating user and groups
useradd -u 21- -g dba -c “THE RDBMS” -d /home/oracle -s /bin/ksh -m oracle
groupadd -g 241 dba
Informations are stored in /etc/passwd annd /etc/shadow
7.Setting limits on file size
ulimit 234567
8.Modifying and removing users
usermod -s /bin/bash oracle
userdel oracle
9.Changing password
passwd
# change group to staff for this file
chgrp staff myfile
# change owner to jsmith for this file
chown jsmith myfile
Shell input and output
ECHO
$ echo “What is your name :\c”
$ read name
You can put variables within echo statements as well as escape codes. In this
example, the terminal bell is rung, your $HOME directory is displayed, and the
command tty is evaluated by the shell.
$ echo “\007your home directory is $HOME, you are connected on ‘tty‘”
your home directory is /home/dave, you are connected on /dev/ttyp1
If it’s LINUX then . . .
You have to put a ‘-n’ after the echo to suppress the new line:
$ echo -n “What is your name :”
You have to put a ‘-e’ after echo for the escape code to work:
$ echo -e “\007your home directory is $HOME, you are connected on
‘tty‘”
your home directory is /home/dave, you are connected on /dev/ttyp1
READ
read variable1 variable2 . . .
$ read name
Hello I am superman
$ echo $name
Hello I am superman
$ read name surname
John Doe
$ echo $name $surname
John Doe
$ pg var_test
#!/bin/sh
# var_test
echo “First Name :\c”
read 51
read name
echo “Middle Name :\c”
read middle
echo “Last name :\c”
read surname
If it’s LINUX then . . .
Remember to use the ‘-n’ echo option.
$ pg var_test
#!/bin/sh
# var_test
echo “First Name :\c”
read name
echo “Middle Name :\c”
read middle
echo “Last name :\c”
read surname
PIPES
$ who | awk ‘{print $1″\t”$2}’
matthew pts/0
louise pts/1
$ df -k | awk ‘{print $1}’| grep -v “Filesystem”
$ df -k | awk ‘{print $1}’| grep -v “Filesystem”|sed s’/\/dev\///g’
hda5
hda8
hda6
hdb5
hdb1
hda7
hda1
TEE
The tee command acts the way it is pronounced. It takes the input and sends
one copy to the standard output and another copy to a file. If you want to see
your output and save it to a file at the same time, then this is the command for
you.
The general format is:
tee -a files
$ who | tee who.out
louise pts/1 May 20 12:58 (193.132.90.9)
matthew pts/0 May 20 10:18 (193.132.90.1)
cat who.out
louise pts/1 May 20 12:58 (193.132.90.9)
matthew pts/0 May 20 10:18 (193.132.90.1)
$ find etc usr/local home -depth -print | cpio -ovC65536 -O \
$ echo ” myscript is now running, check out any errors…in
myscript.log” | tee -a myscript.log
$ myscript | tee -a myscript.log
$ echo “stand-by disk cleanup starting in 1 minute” | tee/dev/console
$ sort myfile | tee -a accounts.log
$ myscript | tee -a accounts.log
Standard input, output and errors
Redirecting standard output
$ cat passwd | awk -F: ‘{print $1}’| sort 1>sort.out
$ ls -l | grep ^d >>files.out
$ ls account* >> files.out
Redirecting standard input
sort < names.txt
sort <names.txt > names.out
$ cat >> myfile <<MAYDAY
> Hello there I am using a $TERM terminal
> and my user name is $LOGNAME
> bye…
> MAYDAY
$ pg myfile
Hello there I am using a vt100 terminal
and my user name is dave
Redirecting standard error
To redirect standard errors you specify the file descriptor ‘2’. Let’s first look at an
example, since they always explain better. Here grep searches for a pattern called
‘trident’ on a file called missiles.
$ grep “trident” missiles
grep: missiles: No such file or directory
grep reports no such file, and sends its errors to the terminal which is the default.
Let’s now send all errors to a file, in fact to the system dustbin, ‘/dev/null’.
$ grep “trident” missiles 2>/dev/null
Combining standard output and error
$ cat account_qtr.doc account_end.doc 1> accounts.out 2> accounts.err
Merging standard output and standard error
When merging output and errors, the shell evaluates the command from left to
right, which is really all you need to know when putting mergers together. Here’s an
example.
$ cleanup >cleanup.out 2>&1
In the above example the script cleanup directs all output (>) to a file called
cleanup.out, and all errors (2), are directed to (>) the same place as the output
(&1), which is cleanup.out.
$ grep “standard” * > grep.out 2>&1
In the above example all output from the grep command is put into the output file
grep.out. When you use here documents, you will probably need to capture all the
output to a file in case errors are encountered. To do this you need to use 2>&1 as
part of the command. Here’s how to do it:
$ cat>> filetest 2>&1 <<MAYDAY
> This is my home $HOME directory
> MAYDAY
Command execution order
Using &&
Here’s the general format of &&:
command1 && command2
The operation of these commands is fairly straightforward. The command on
the left of the && (command1) must be returned true (successfully executed) if the
command on the right (command2) is to be executed; or to put it another way, ‘If
this command works’ && ‘then execute this command’.
$ cp justice.doc justice.bak && echo “if you are seeing this then cp was
OK”
if you are seeing this then cp was OK
$ mv /apps/bin /apps/dev/bin && rm -r /apps/bin
In the above example, the directory /apps/bin is to be moved to /apps/dev/bin;
if this fails then the deletion of the directory /apps/bin will not happen.
In the following example, a file is to be sorted with the output going to a file
called quarter.sort; if this is successful, the file will be printed.
$ sort quarter_end.txt > quarter.sorted && lp quarter.sorted
Using ||
command1 || command2
The operation of the || is slightly different. If the command on the left of the ||
(command1) fails then execute the command on the right of the || (command2); or
to put it another way, ‘If this command fails’ || ‘then execute this command’.
Here’s another simple example that illustrates the use of the ||:
$ cp wopper.txt oops.txt || echo “if you are seeing this cp failed”
cp: wopper.txt: No such file or directory
if you are seeing this cp failed
The copy has failed so now the command on the right-hand side of the || is
executed.
A more practical example is this. I want to extract the first and fifth fields
from an accounts file and stick the output into a temp file. If the extraction does not
work, I want to be mailed.
$ awk ‘{print$1,$5}’ acc.qtr >qtr.tmp || echo “Sorry the payroll
extraction didn’t work” | mail dave
You don’t always have to use system commands; here I run the script comet on a file
called month_end.txt. If the script bombs out, the shell should terminate (exit).
$ comet month_end.txt || exit.
Using GREP
Searching more than one file
If I wanted to search for the string “sort” in all my .doc files in my directory I
would do this:
$ grep “sort” * .doc
Or I could search through all the files that have the word “sort it”.
$ grep “sort it” *
Line matches
$ grep -c “48” data.f
$ 4
Line numbers
$ grep -n “48” data.f
1:48 Dec 3BC1997 LPSX 68.00 LVX2A 138
2:483 Sept 5AP1996 USP 65.00 LVX2C 189
5:484 Nov 7PL1996 CAD 49.00 PLV2C 234
6:483 May 5PA1998 USP 37.00 KVM9D 644
Do not match
$ grep -v “48” data.f
47 Oct 3ZL1998 LPSX 43.00 KVM9D 512
219 Dec 2CC1999 CAD 23.00 PLV2C 68
216 Sept 3ZL1998 USP 86.00 KVM9E 234
Getting an exact match
$ grep ’48\>’ data.f
48 Dec 3BC1997 LPSX 68.00 LVX2A 138
Being case-sensitive
$ grep -i “sept” data.f
483 Sept 5AP1996 USP 65.00 LVX2C 189
216 sept 3ZL1998 USP 86.00 KVM9E 234
Pattern ranges
$ grep ’48[34]’ data.f
483 Sept 5AP1996 USP 65.00 LVX2C 189
484 nov 7PL1996 CAD 49.00 PLV2C 234
483 may 5PA1998 USP 37.00 KVM9D 644
Don’t match at the beginning of a line
$ grep ‘^[^48]’ data.f
219 dec 2CC1999 CAD 23.00 PLV2C 68
216 sept 3ZL1998 USP 86.00 KVM9E 234
Trapping upper and lower cases
$ grep ‘[Ss]ept’ data.f
483 Sept 5AP1996 USP 65.00 LVX2C 189
216 sept 3ZL1998 USP 86.00 KVM9E 234
Matching any characters
$ grep ‘K…D’ data.f
47 Oct 3ZL1998 LPSX 43.00 KVM9D 512
483 may 5PA1998 USP 37.00 KVM9D 644
$ grep ‘[A-Z][A-Z]..C’ data.f
483 Sept 5AP1996 USP 65.00 LVX2C 189
219 dec 2CC1999 CAD 23.00 PLV2C 68
484 nov 7PL1996 CAD 49.00 PLV2C 234
Date searching
$ grep ‘5..199[6,8]’ data.f
483 Sept 5AP1996 USP 65.00 LVX2C 189
483 may 5PA1998 USP 37.00 KVM9D 644
$ grep ‘[0-9]\ {3\}[8]’ data.f
47 Oct 3ZL1998 LPSX 43.00 KVM9D 512
483 may 5PA1998 USP 37.00 KVM9D 644
216 sept 3ZL1998 USP 86.00 KVM9E 234
Combining ranges
Staying with the use of the [ ] brackets to extract information, suppose we want to
get city codes where the first character could be any number, the second character
between 0 and 5 and the third between 0 and 6. If we use this pattern we’ll get the
following output:
$ grep ‘[0-9][0-5][0-6]’ data.f
48 Dec 3BC1997 LPSX 68.00 LVX2A 138
483 Sept 5AP1996 USP 65.00 LVX2C 189
47 Oct 3ZL1998 LPSX 43.00 KVM9D 512
219 dec 2CC1999 CAD 23.00 PLV2C 68
484 nov 7PL1996 CAD 49.00 PLV2C 234
483 may 5PA1998 USP 37.00 KVM9D 644
216 sept 3ZL1998 USP 86.00 KVM9E 234
Well, we certainly got a lot of information back. What we want is included but we
also got other records we didn’t want. However, looking at our pattern, all records
that have been returned are the correct ones according to our rules. We need to
specify that the pattern must start at the beginning of each line. We can use the ^ for
that.
$ grep ‘^[0-9][0-5][0-6]’ data.f
216 sept 3ZL1998 USP 86.00 KVM9E 234
Occurrences in a pattern
If we want to extract any row that has a number 4 repeated at least twice, we could
use this:
$ grep ‘4\{2,\}’ data.f
483 may 5PA1998 USP 37.00 KVM9D 644
$ grep ‘9\{3,\}’ data.f
219 dec 2CC1999 CAD 23.00 PLV2C 68
If you want to search for only so many occurrences then take the comma out.
This will search for only two occurrences of the number 9.
$ grep ‘9\{2\}’ data.f
88 The grep family
There may be a need to match between, say, two and six occurrences of a
number or maybe a letter. This will match between two and six occurrences of the
number 8 that ends with 3:
$ grep ‘6\{2,6}3’ myfile
83 – no match
888883 – match
8884 – no match
88883 – match
Blank lines
$ grep ‘^$’ myfile
Matching special characters
If you wish to search for characters that have special meanings, like one of the
following, $ . ‘ ” * [ ] ^ ( ) | \ + ?, then you must place a \ in front of them. Suppose
you want to search for all lines that contain a period (.), you would do this:
$ grep ‘\.’ myfile
Searching for ip addresses
Part of my job is looking after our DNS servers, which means maintaining a lot of
ip addresses that cover different networks. Our address ip file can contain over 200
addresses. Sometimes I want to look at just, say, the ‘nnn.nnn’ network addresses,
and forget about the rest that have only two digits in the second part, i.e. nnn.nn..
To extract all these nnn.nnn. addresses, use [0-9]\{3\}\.[0-0\[3\}\. This expression is
saying any number repeated three times followed by a period, any number repeated
three times followed by a period.
$ grep ‘[0-9]\{3\}\.[0-0\[3\}\.’ Ipfile
Pattern matching with wildcards
Let’s take a look at the use of wildcards in using grep. Suppose we have a file like
this:
$ pg testfile
looks
likes
looker
long
Here is what is displayed when using the following grep pattern:
$ grep ‘l.*s’ testfile
looks
likes
$ grep ‘l.*k.’ testfile
looks
likes
$ grep ‘ooo*’ testfile
looks
If you want to find a word only at the end of a line, try this:
$ grep ‘device$’ *
That will search all files for lines that have the word ‘device’ at the end of each line.
Class names 91
All this is saying is that you have typed a filename that does not exist. If we
use the -s switch in grep we can silence these error messages.
$ grep -s “louise” /etc/password
Using grep on a string
grep is not only reserved for files; you can also use grep on strings. All you need to
do is echo the string then pipe it through to grep.
$ STR=”Mary Joe Peter Pauline”
$ echo $STR | grep “Mary”
Mary Joe Peter Pauline
egrep
egrep stands for expression or extended grep depending on who you listen
Egrep accepts the full range of regular expressions. One of the nice features of egrep is that you can store your strings in a file and pass them into egrep. We do this with the -f switch. If we create a file called grep strings and then type 484 and 47 into it:
$ pg grepstrings
484
47
$ egrep -f grepstrings data.f
this would match all records with 484 or 47 in them. The -f switch really becomes
useful if you want to match a lot of patterns, and typing them in on the command
line becomes awkward.
If we want to search for store codes 32L or 2CC we can use the bar sign (|),
which means one or the other or both (all), separated by the bar sign.
$ egrep ‘(3ZL|2CC)’ data.f
47 Oct 3ZL1998 LPSX 43.00 KVM9D 512
219 dec 2CC1999 CAD 23.00 PLV2C 68
216 sept 3ZL1998 USP 86.00 KVM9E 234
You can use as many bars as you want. If we wanted to see if users louise,
matty or pauline were logged into the system we could use the who command and
pipe the output through to egrep.
$ who | egrep (louise|matty|pauline)
louise pty8
matty tty02
pauline pty2
You can also exclude certain strings using the caret sign (^). If I wanted to see who
was on the system, but I did not want to know if users matty and pauline were on I
could do this:
$ who | egrep -v ‘^(matty|pauline)’
If you want to search a directory listing for files that were called shutdown,
shutdowns, reboot or reboots, this is easily accomplished with egrep.
$ egrep ‘(shutdown | reboot) (s)?’ *
USING AWK
$ pg grade.txt
M.Tansley 05/99 48311 Green 8 40 44
J.Lulu 06/99 48317 green 9 24 26
P.Bunny 02/99 48 Yellow 12 35 28
J.Troll 07/99 4842 Brown-3 12 26 26
L.Tansley 05/99 4712 Brown-2 12 30 28
Printing all records
$ awk ‘{print $0}’ grade.txt
Printing individual records
$ awk ‘{print $1,$4}’ grade.txt
Printing report headers
$ awk ‘BEGIN {print “Name Belt\n——————————–“}
{print $1″\t”$4}’ grade.txt
Name Belt
—————————————
M.Tansley Green
J.Lulu green
P.Bunny Yellow
J.Troll Brown-3
L.Tansley Brown-3
Printing report trailers
$ awk ‘BEGIN {print “Name\n——-“} {print $1} END {“end-of-report”}’
grade.txt
Matching
$ awk {if($4~/Brown/) print $0}’ grade.txt
J.Troll 07/99 4842 Brown-3 12 26 26
L.Tansley 05/99 4712 Brown-2 12 30 28
$ awk ‘$0 ~ /Brown/’ grade.txt
J.Troll 07/99 4842 Brown-3 12 26 26
L.Tansley 05/99 4712 Brown-2 12 30 28
Exact match
$ awk ‘{if($3~/48/) print $0}’ grade.txt
M.Tansley 05/99 48311 Green 8 40 44
J.Lulu 06/99 48317 green 9 24 26
P.Bunny 02/99 48 Yellow 12 35 28
J.Troll 07/99 4842 Brown-3 12 26 26
Not matched
$ awk ‘$0 !~ /Brown/’ grade.txt
M.Tansley 05/99 48311 Green 8 40 44
J.Lulu 06/99 48317 green 9 24 26
P.Bunny 02/99 48 Yellow 12 35 28
We could have targeted just the belt grade field ‘field-4’ and done the test this way:
$ awk ‘{if($4!~/Brown/) print $0}’ grade.txt
M.Tansley 05/99 48311 Green 8 40 44
J.Lulu 06/99 48317 green 9 24 26
P.Bunny 02/99 48 Yellow 12 35 28
Less than
$ awk ‘{if ($6 < $7) print $0 “$1 Try better at the next comp”}’
grade.txt
M.Tansley Try better at the next comp
J.Lulu Try better at the next comp
Any characters
$ awk ‘$1 ~/^ . . . a/’ grade.txt
M.Tansley 05/99 48311 Green 8 40 44
L.Tansley 05/99 4712 Brown-2 12 30 28
Match either
$ awk ‘$0~/(Yellow|Brown)/’ grade.txt
P.Bunny 02/99 48 Yellow 12 35 28
J.Troll 07/99 4842 Brown-3 12 26 26
L.Tansley 05/99 4712 Brown-2 12 30 28
AND
$ awk ‘{if ($1==”P.Bunny” && $4==”Yellow”)print $0}’ grade.txt
P.Bunny 02/99 48 Yellow 12 35 28
OR
$ awk ‘{if ($4==”Yellow” || $4~/Brown/) print $0}’ grade.txt
P.Bunny 02/99 48 Yellow 12 35 28
J.Troll 07/99 4842 Brown-3 12 26 26
L.Tansley 05/99 4712 Brown-2 12 30 28
awk built-in variables
ARGC The number of command-line arguments
ARGV The array of command-line arguments
ENVIRON Holds the current system environment variables in the array
FILENAME The name of the current file awk is scanning
FNR The record number in the current file
FS Sets the input field separator; same as the command-line -F option
NF Number of fields in the current record
NR The number of records read so far
OFS Output field separator
ORS Output record separator
RS Controls the record separator
NF, NR and FILENAME
$ awk ‘{print NF,NR,$0}END{print FILENAME}’ grade.txt
7 1 M.Tansley 05/99 48311 Green 8 40 44
7 2 J.Lulu 06/99 48317 green 9 24 26
7 3 P.Bunny 02/99 48 Yellow 12 35 28
7 4 J.Troll 07/99 4842 Brown-3 12 26 26
7 5 L.Tansley 05/99 4712 Brown-2 12 30 28
grade.txt
$ awk ‘{if (NR >0 && $4~/Brown/)print $0}’ grade.txt
J.Troll 07/99 4842 Brown-3 12 26 26
L.Tansley 05/99 4712 Brown-2 12 30 28
Assigning input fields to field variable names
$ awk ‘{name=$1;belts=$4; if(belts ~/Yellow/)print name” is belt
“belts}’ grade.txt
P.Bunny is belt Yellow
Comparing fields with values
$ awk ‘{if($6 < 27)print$0}’ grade.txt
J.Lulu 06/99 48317 green 9 24 26
J.Troll 07/99 4842 Brown-3 12 26 26
$ awk ‘BEGIN {BASELINE=”27 “}{if($6 < BASELINE)print$0}’ grade.txt
J.Lulu 06/99 48317 green 9 24 26
J.Troll 07/99 4842 Brown-3 12 26 26
SED COMMAND
[root@debasis cls]# cat deb
debasis 1
debraj 1
debajyoti 1
debasis 1
[root@debasis cls]# sed ‘s/debasis/deb/g’ deb
deb 1
debraj 1
debajyoti 1
deb 1
[root@debasis cls]# sed ‘s/[^ ]*/M&/’ deb
Mdebasis 1
Mdebraj 1
Mdebajyoti 1
Mdebasis 1
[root@debasis cls]# sed ‘s/[^ ]*/&M/’ deb
debasisM 1
debrajM 1
debajyotiM 1
debasisM 1
[root@debasis cls]# sed ‘s/[^ ]*/&M/’ deb
debasisM 1
debrajM 1
debajyotiM 1
debasisM 1
[root@debasis cls]# sed ‘/debasis/ s/1/2/’ deb
debasis 2
debraj 1
debajyoti 1
debasis 2
[root@debasis cls]# sed ‘1 s/1/2/’ deb
debasis 2
debraj 1
debajyoti 1
debasis 1
[root@debasis cls]# sed ‘1,2 s/1/2/’ deb
debasis 2
debraj 2
debajyoti 1
debasis 1
[root@debasis cls]# sed -e ‘s/debasis/deb/’ -e ‘s/debraj/deba/’ deb
deb 1
deba 1
debajyoti 1
deb 1
[cls@debasis cls]$ sed ‘
> /debraj/ s/1/2/
> /debajyoti/ s/1/3/’ deb
debasis 1
debraj 2
debajyoti 3
debasis 1
[cls@debasis cls]$ sed ‘/debraj/ d’ deb
debasis 1
debajyoti 1
debasis 1
[cls@debasis cls]$ sed ‘1,3 d’ deb
[cls@debasis cls]$ sed ‘/^deb/ d’ deb
[cls@debasis cls]$ sed ‘/1$/ d’ deb
[cls@debasis cls]$ sed ‘/debasis/ s/1/2/;/debraj/ d’ deb
debasis 2
debajyoti 1
debasis 2
[cls@debasis cls]$ sed ‘$a\
> stop!!!\
> I am lost’ deb
debasis 1
debraj 1
debajyoti 1
debasis 1
stop!!!
I am lost
[cls@debasis cls]$ sed ‘3a\
stop!!!\
I am lost’ deb
debasis 1
debraj 1
debajyoti 1
stop!!!
I am lost
debasis 1
[cls@debasis cls]$ sed ‘3i\
stop!!!\
I am lost’ deb
debasis 1
debraj 1
stop!!!
I am lost
debajyoti 1
debasis 1
[cls@debasis cls]$ sed ‘/debasis/ c\
> no use’ deb
no use
debraj 1
debajyoti 1
no use
ENVIRONMENT AND SHELL
Local variables
$variable_name=value or ${variable_name = value}
Displaying a variable
$ GREAT_PICTURE=”die hard”
$ echo ${GREAT_PICTURE}
die hard
$ DOLLAR=99
$ echo ${DOLLAR}
99
$ LAST_FILE=ZLPSO.txt
$ echo ${LAST_FILE}
ZLPSO.txt
Clearing a variable
unset variable_name
$ PC=enterprise
$ echo ${PC}
enterprise
$ unset PC
$ echo ${PC}
Displaying all local shell variables
$ set
…
PWD=/root
SHELL=/bin/sh
SHLVL=1
TERM=vt100
UID=7
USER=dave
dollar=99
great_picture=die hard
last_file=ZLPSO.txt
Using variables to hold arguments for system commands
$ SOURCE=”/etc/passwd”
$ DEST=”/tmp/passwd.bak”
$ cp ${SOURCE} ${DEST}
Making a variable read-only
$ TAPE_DEV=”/dev/rmt/0n”
$ echo ${TAPE_DEV}
/dev/rmt/0n
$ readonly TAPE_DEV
$ TAPE_DEV=”/dev/rmt/1n”
sh: TAPE_DEV: read-only variable
SHELL SCRIPTING
File status tests
-d This is a directory
-f This is a regular file
-L This is a symbolic link
-r This file is readable
-s This file has a size greater than zero, not empty
-w This file is writeable
-u This file has the suid bit set
-x This is executable
We will use both test methods to test if the file scores.txt is writeable. We will use
the last status command to test it. Remember a zero status is OK, anything else is
an error.
$ ls -l scores.txt
-rw-r–r– 1 dave admin 0 May 15 11:29 scores.txt
$ [ -w scores.txt ]
$ echo $?
0
$ test -w scores.txt
$ echo $?
0
cutting columns(-c)
cut -c 1-20 deb [show columns from c1-c10]
cut -c1 /etc/passwd
get the first character from every line
cut -c1,5,10-20 /etc/passwd
get the first, fifth character and every character between 10 and 20
cut -d: -f2 /etc/passwd
get the second field
cut -d: -f3- /etc/passwd
get all fields from the third on
cut –f1,3- telneos > deb
cut –d’ ‘ –f2- file (delimiter as space)
cut –f1,3 telnos address>telnos.all(selecting particular columns from two files and pasting in one)
colrm 8 12 twinkle (deleting 8 and 12 column)
pasting columns
paste adc efg>>abcefg
paste –d: abc efg>>abcefg
paste telos – > telnos.new
using cut and paste to reorganize a file
cut –f1,3 telnos > temp
cut –f4- telnos>temp2
cut –f2 telos|paste temp-temp2>telnos.new
Using logical operators with tests
-a Logical AND, true, if both sides of the operator are true
-o Logical OR, true, if either sides of the operator can be true
! Logical NOT, true, if the condition is false
-rw-r–r– 1 root root 0 May 15 11:29 scores.txt
-rwxr-xr– 1 root root 0 May 15 11:49 results.txt
The following example tests whether both files are readable.
$ [ -w results.txt -a -w scores.txt ]
$ echo $?
0
Testing strings
= The two strings are equal
!= The two strings are not equal
-z This string is null
-n This string is not null
To test if the environment variable EDITOR is empty:
$ [ -z $EDITOR ]
$ echo $?
1
No, it isn’t. Is it set to vi?
$ [ $EDITOR = “vi” ]
$ echo $?
0
Testing numbers
-eq The two numbers are equal
-ne The two numbers are not equal
-gt The first number is greater than the second number
-lt The first number is less than the second number
-le The first number is less than or equal to the second number
-gt The first number is greater than or equal to the second number
$ NUMBER=130
$ [ “$NUMBER” -eq “130” ]
$ echo $?
0
Using expr
$ expr 900 + 600
1500
$ expr 30 / 3
10
$ expr 30 / 3 / 2
5
Incrementing counters
Expr does the incrementing of values when using loops. First the loop is initialized
to zero. Then one is added to the variable loop. The use of the quotes means
command substitution, which basically means take the output from the command
(expr) and put it in the variable loop.
$ LOOP=0
$ LOOP=‘expr $LOOP + 1‘
Pattern matching
$ VALUE=accounts.doc
$ expr $VALUE : October 8, ‘.*’
12
Control flow structures
$ pg iftest
#!/bin/sh
# iftest
# this is a comment line, all comment lines start with a #
if [ “10” -lt “12” ]
then
# yes 10 is less than 12
echo “Yes, 10 is less than 12”
fi
Testing values of variables
$ pg iftest2
#!/bin/sh
# if test2
echo -n “Enter your name :”
read NAME
# did the user just hit return ????
if [ “$NAME” = “” ] ; then
echo “You did not enter any information”
fi