Learn UNIX with a new approach

I need to go back to go forward. Back to basics. The foundation , those tiny details ,step by step and question everything because you know nothing. If there is one thing i am certain of , then that is i know nothing.

I have a Habit of wandering, so one might see a bit of philosophy  or psychology or science or fiction or opinion or quote here and then. For example today i have begun with a philosophical quote and continuing with the concepts of UNIX, which is very famous operating system   , still in use for lot of major servers everywhere due its utility.

It is very important that i quote this. In today’s world there is too much information overload. Lot of times The Journalists and Content creators use eye catchy line to draw attention, and when you read the content you will be like what a waste of time. I am neither a journalist nor a guy who is paid to write, i write because i like writing and sharing my ideas. Even though i am the only person reading this as of now. I like it.

So its gonna be more UNIX from this point . I wonder what went in the mind of Dennis Ritchie when he designed UNIX. There is hardly any video of this guy, who is hands down genius. Creator of C and UNIX. Backbone foundation of modern software.

d1

I need to remind myself this. Be sequential, try to be sequential and add more ideas incrementally on top of it.

Identity of UNIX is in its Philosophy.

These are quotes from the creators of UNIX.

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. Keep it simple, hence the name Uni.

Fancy algorithms are buggier than simple ones, and they’re much harder to implement. Use simple algorithms as well as simple data structures – Rob Pike

Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.

Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information

Use tools in preference to unskilled help to lighten a programming task – Doug Mcllroy

Controlling complexity is the essence of computer programming. Debugging dominates development time, and getting a working system out the door is usually less a result of brilliant design than it is of managing not to trip over your own feet too many times – Brian Kernighan

The original name was “UNICS” (UNiplexed Information and Computing Service)

What is Time sharing system ?

Time sharing is a technique which enables many people, located at various terminals, to use a particular computer system at the same time.  Most time sharing system use time-slice (round robin) scheduling of CPU. In this approach, Programs are executed with rotating priority that increases during waiting and drops after the service is granted. In Order to prevent a program from monopolizing the processor, a program executing longer than the system defined time-slice in interrupted by the operating system and placed at the end of the queue of waiting program.

Did you know that UNIX operating system is written in C?

In the narrowest sense UNIX is a time sharing Operating System. 

Echo copies the output directly onto the screen.

Return key or the Enter key signifies the end of the input.

Let us begin the journey . Yo hoo. for the 1000 th time.

a dialog between me and the UNIX

-bash-3.2$ date
 Tue Apr 12 06:17:30 BST 2016
-bash-3.2$ date +%d
 12
 -bash-3.2$ date +%m
 04
 -bash-3.2$ date +%y
 16
 -bash-3.2$ date +%d-%m-%y
 12-04-16
 -bash-3.2$ date +%t

-bash-3.2$ date +%T
 06:16:55
 -bash-3.2$ date +%r
 6:16:58 am BST
 -bash-3.2$ date +%H
 06
 -bash-3.2$ date +%H-%M
 06-17

-bash-3.2$ cal
 April 2016
 Su Mo Tu We Th Fr Sa
 1 2
 3 4 5 6 7 8 9
 10 11 12 13 14 15 16
 17 18 19 20 21 22 23
 24 25 26 27 28 29 30
-bash-3.2$ time

real 0m0.000s
 user 0m0.000s
 sys 0m0.000s
-bash-3.2$ who

to check who has logged into to the server
-bash-3.2$ ls -ltr -t
 total 108

cause the listing to be in time order

never observed this , total gives total amount of block size the files are using

Filenames

File names were limited to 14 bytes (equivalent to 14 characters) in early UNIX systems. However,
modern Unix-like systems support long file names, usually up to 255 bytes in length. File names can be as short as a single character.

 -bash-3.2$ wc test
 4 22 375 test

here the 4 is the number of lines and 22 is the number of words and 375 is the number of characters. Interesting point here is that the size of the file is same as the number of characters in it.

Great GREP and Scripting 

Get all the implementations of this powerful tool of UNIX and put it here, such that we can use its multiple uses as and when needed.

1)

Print the matched line, along with the 3 lines after it. -A 3 does the trick

-bash-3.2$ grep -A 3 -i _na input.sh
 echo `autorep -j IB_CT_CVA_1109_P1_TS_Beronimo_na_Watch` | awk '{print $19,$20}'
 echo ''
 echo 'Man vs wild *KLA MT*'
 echo `autorep -j IB_CT_CVA_1109_P1_TS_Tec_QBS_Feed_Watch` | awk '{print $19,$20}'

2)

Search for a given string in all files recursively

$ grep -r "ramesh" *

3)

Search for a given string in a file (case in-sensitive search).
$ grep -i "the" demo_file

4)

Simple Grep , which is to search for a pattern in a file
grep "string" FILE_PATTERN

5)

search for a word, and to avoid it to match the sub strings use -w

$ grep -i "is" demo_file
 THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.
 this line is the 1st lower case line in this file.
 This Line Has All Its First Character Of The Word With Upper Case.
 Two lines above this line is empty.
 And this is the last line.

 $ grep -iw "is" demo_file
 THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.
 this line is the 1st lower case line in this file.
 Two lines above this line is empty.
 And this is the last line.

6)

to count the number of matching occurrences
-bash-3.2$ grep -c -i autorep input.sh
 19

7)

Using If with the Grep. Autorep is a command. Anything other command will work too.

autorep -j job_*Cube* |awk ‘{ if ($4 == “OH” || $4 == “RU”) print $1}’ |mail -s “Status” manjit@abc.com

8)

Assign the positional parameter inside the script for use, this is basically the parameter that you pass to the script while execution.

job=$1

9)

Note the single equals for comparison and the IF , THEN structure 

if [ "$check_status" = "FA" ]; then
 echo "test"
 else
 echo "test"
 fi

10)

to sleep for 5 seconds
sleep 5

11)

In file to get the 21st field using AWK
awk '{print $21}'

12)

the structure of While statement

while [ $check != $test_su ]
 do
 echo " test "
 check=some condition
 done

13)

how to connect to the database using UNIX. Main points to note the use of Oracle client and the structure. 

#!/bin/sh

export ORACLE_HOME=/sbcimp/run/tp/oracle/client/v11.2.0.4-B1-32bit/client_1
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH

type=$1
date=$2
userid=$3
password=$4
echo $type
echo $date
echo $userid
echo $password

result=`$ORACLE_HOME/bin/sqlplus -s $userid/$password@db << END
set heading off
set pagesize 0
set tab off

select r.date,res.NAME FROM account_name.table_name res,account_name.table_name r,account_name.table_name cr WHERE r.id = res.id and r.date =$date res.status=’FAIL’

/
exit
END`

echo $type ‘Result is below’
echo $result

14)

to check the length of a string
 -z option in UNIX

15)

uniq command

-bash-3.2$ uniq testforuniq.txt

above command will remove the simultaneous duplicates only

test

-bash-3.2$ uniq -c testforuniq.txt << this will give the count of the lines again the simultaneous only

-bash-3.2$ uniq -d test << print only the duplicated lines

The following uniq command using option ‘w’ is compares first 8 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -c -w 8 testNew

the above failure of uniq command can be rectified with below command

-bash-3.2$ less testforuniq.txt | sort -u

test2.jpg

16)

back to grep.

You can use -v option to print inverts the match; that is, it matches only those lines that do not contain the given word. For example print all line that do not contain the word bar

$ grep -v bar /path/to/file

17)

egrep is the same as running grep -E.

fgrep is the same as running grep -F. In this mode, grep evaluates your PATTERN string as a “fixed string” — every character
in your string is treated literally. For example, if your string contains an asterisk (“*”), grep will try to match it with
an actual asterisk rather than interpreting this as a wildcard.

rgrep is the same as running grep -r. In this mode, grep will perform its search recursively. If it encounters a directory,
it will traverse into that directory and continue searching. (Symbolic links are ignored; if you want to search directories
that are symbolically linked, you should use the -R option instead).

18)

let’s say that you have an entire folder full of music files in a bunch of different formats.
You want to find all of the *.mp3 files from the artist JayZ, but you don’t want any of the remixed tracks.
Using a find command with a couple of grep pipes will do the trick:

# find . –name “*.mp3” | grep –i JayZ | grep –vi “remix”

19)

The –n option for grep is very useful when debugging files during compile errors. It displays the line number in the file of the given search string:

# grep –n “main” setup..py

20)

Passing the –w option to grep searches for the entire pattern that is in the string. For example, using:

# ifconfig | grep –w “RUNNING”

21)

display the lines which starts with “root” word in the file /etc/passwd.

root@Linux-world:~# grep ^root /etc/passwd

List all the lines of /etc/passwd that ends with “bash” word.

root@Linux-world:~# grep bash$ /etc/passwd

Exit Codes

On Unix and Linux systems, programs can pass a value to their parent process while terminating. This value is referred to as an exit code or exit status. On POSIX systems the standard convention is for the program to pass 0 for successful executions and 1 or higher for failed executions.

what happens If exit code is not mentioned ?

In Linux any script run from the command line has an exit code. With Bash scripts, if the exit code is not specified in the script itself the exit code used will be the exit code of the last command run.

sample

u1.jpg

When we execute this script (as a non-root user) the touch command will fail, ideally since the touch command failed we would want the exit code of the script to indicate failure with an appropriate exit code.

u2
As you can see after running the ./tmp.sh command the exit code was 0 which indicates success, even though the touch command failed. The sample script runs two commands touch and echo, since we did not specify an exit code the script exits with the exit code of the last run command. In this case, the last run command is the echo command, which did execute successfully.

Now using the same technique to handle the exit codes inside the script

u4.jpg
In the above revision of our sample script; if the exit code for touch is 0 the script will echo a successful message. If the exit code is anything other than 0 this indicates failure and the script will echo a failure message to stderr.

u5

While the above revision will provide an error message if the touch command fails, it still provides a 0 exit code indicating success.

Since the script failed, it would not be a good idea to pass a successful exit code to any other program executing this script. To add our own exit code to this script, we can simply use the exit command.

u6.jpg

With the exit command in this script, we will exit with a successful message and 0 exit code if the touch command is successful. If the touch command fails however, we will print a failure message to stderr and exit with a 1 value which indicates failure.

u7

u11

The above grouping of commands use what is called list constructs in bash. List constructs allow you to chain commands together with simple && for and and|| for or conditions. The above command will execute the ./tmp.sh script, and if the exit code is 0 the command echo "bam" will be executed. If the exit code of ./tmp.sh is 1 however, the commands within the parenthesis will be executed next. Within the parenthesis the commands are chained together using the && and || constructs again.

The list constructs use exit codes to understand whether a command has successfully executed or not. If scripts do not properly use exit codes, any user of those scripts who use more advanced commands such as list constructs will get unexpected results on failures.

The exit command in bash accepts integers from 0 - 255, in most cases 0 and 1 will suffice however there are other reserved exit codes that can be used for more specific errors.

P.S. More on UNIX in next blog

References

http://bencane.com/2014/09/02/understanding-exit-codes-and-how-to-use-them-in-bash-scripts/

http://www.ibm.com/developerworks/aix/library/au-vitips.html

http://www.cyberciti.biz/faq/linux-unix-formatting-dates-for-display/

advanced bash scripting guide

The Unix Programming Environment – Brian Kernighan

15 Practical Grep Command Examples In Linux / UNIX

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s