Unix Theory

Kernel

it is the hub of UNIX operating system. It handle memory to programs, File store and communications in response to system calls.

As an illustration of the way that the shell and the kernel work together, suppose a user types rm myfile (which has the effect of removing the file myfile). The shell searches the filestore for the file containing the program rm, and then requests the kernel, through system calls, to execute the program rm on myfile. When the process rm myfile has finished running, the shell then returns the UNIX prompt % to the user, indicating that it is waiting for further commands.

Shell

The shell acts as an interface between the user and the kernel. When a user logs in, the login program checks the username and password, and then starts another program called the shell. The shell is a command line interpreter (CLI). It interprets the commands the user types in and arranges for them to be carried out. The commands are themselves programs: when they terminate,
the shell gives the user another prompt (% on our systems).

 Everything in UNIX is either a file or a process.

A process is an executing program identified by a unique PID (process identifier).
A file is a collection of data. They are created by users using text editors, running compilers etc.
Examples of files:

-a document (report, essay etc.)
-the text of a program written in some high-level programming language
-instructions comprehensible directly to the machine and incomprehensible to a casual user, for example, a collection of binary digits (an executable or binary file);
-a directory, containing information about its contents, which may be a mixture of other directories (subdirectories) and ordinary files.
-All the files are grouped together in the directory structure. The file-system is arranged in a hierarchical structure, like an inverted tree. The top of the hierarchy is traditionally called root (written as a slash / )

u1

UNIX commands

to list all files including the hidden ones

$ls -a

Scp command

scp -i /home/user/hit/private_key server@testpap.ldn.bank.com:/test/dyn/data/check/PROD/test20160315_1.txt  /home/user/

above –I give the public key

then the server and username

then the path of the file on remote server

and the temp path where you want to store the file

the public key is generated using putty

SSH, or secure shell, is a secure protocol and the most common way of safely administering remote servers. Using a number of encryption technologies, SSH provides a mechanism for establishing a cryptographically secured connection between two parties, authenticating each side to the other, and passing commands and output back and forth.

to make a directory

$mkdir test

to go to home director, a

$cd

. is current directory

.. is parent directory

$cd . or $cd ..

pwd(print working directory)

~ home directory

cp copy and mv commands

rm and rmdir commands

clear command

cat concatenate

less and more . Less writes it one page at a time, press [space bar] to see another page, and press [q] to quit reading.

more has an inability to scroll up.

head and tail command

grep ( global regular expression parser)

% grep -i science science.txt
-ignore case
-v display those lines that do NOT match
-n precede each matching line with the line number
-c print only the total count of matched lines

word count

$wc – w file.txt – no of words

$wc -l file.txt – no of lines

appending a file

$cat >> file.txt – [ctrl D] to stop

standard input – 0, standard output – 1 & standard error – 2

sort

wildcard

$*test

$?test – match just one character

getting help

man command

whatis command – gives a one line description

apropos command – gives variations of a command

u2

to remove read write and execute permissions on the file biglist for the group and others, type
$chmod go-rwx biglist
To give read and write permissions on the file biglist to all,
$chmod a+rw biglist

$ps – used to find information about the processes

$Sleep 10

$Sleep 10 & – to run sleep in the background

$fg %jobid – to foreground a suspended processes

kill command

when the kill command is run it is actually sending a signal to the process. By default the kill command will send a SIGTERM signal to the specified process.The SIGTERM signal tells the process that it should perform it’s shutdown procedures to terminate the process cleanly by closing all log files, connections, etc.

however sometimes these processes can either take a long time or due to other issues not happen at all. Leaving the process in a state where it is not correctly running but also not terminated.
When a process is in a limbo state it is reasonable to send the process the SIGKILL signal, which can be invoked by running the kill command with the-9 flag.

Unlike SIGTERM the SIGKILL signal cannot be captured by the process and thus it cannot be ignored. The SIGKILL signal is handled outside of the process completely, and is used to stop the process immediately. The problem with using SIGKILL is that it does not allow an application to close its open files or database connections cleanly and over time could cause other issues; therefor it is generally better to reserve the SIGKILL signal as a last resort.

Kill pid – SIGTERM

Kill -9 pid – SIGKILL

if [ -n $value ]
then
echo “variable value has a value or $value”
fi
The -n operator is for checking if a variable has a string value or not.

if the argument is empty rather than if it is not empty, to do this you can use the -z operator.
if [ -z $1 ]
then
echo “sorry you didn’t give me a value”
fi

$ ps

Listing 6.1

PID TTY           TIME CMD

24696 ttys000    0:00.19 -bash

24697 ttys001    0:00.16 -bash

PID The process ID of the process.
TTY1 The controlling terminal for the process.
TIME The cumulative CPU time the process has used since the process started. The format for this time is [ dd-]hh:mm:ss.
CMD The name of the command that is running

$ ps u – The ps command has a flag, u, that will display common useful information, such as CPU and memory usage.

Listing 6.2

USER        PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND

markbates 57896   0.0  0.0  4529729   2720 s005  S+   10:27AM   0:00.38

forego start -p 3000

markbates 57892   0.0  0.0  2465360   7636 s005  S+   10:27AM   0:00.04

$ ps -e – other processes running on a modern machine

$ ps -U root – processes that are being run by the user ID provided

$ ps -O etime,%cpu,%mem –
$ ps -m -O %mem -u root – sort processes by their memory usage.
$ ps -r -O %cpu -u root – sort processes by their CPU usage.

fork() is how you create new processes in Unix. When you call fork, you’re creating a copy of your own process that has its own address space.
This allows multiple tasks to run independently of one another as though they each had the full memory of the machine to themselves.

Here are some example usages of fork:

-Your shell uses fork to run the programs you invoke from the command line.
-Web servers like Apache use fork to create multiple server processes, each of which handles requests in its own address space. If one dies or leaks memory,
others are unaffected, so it functions as a mechanism for fault tolerance.
-Google Chrome uses fork to handle each page within a separate process. This will prevent client-side code on one page from bringing your whole browser down.
-fork is used to spawn processes in some parallel programs (like those written using MPI). Note this is different from using threads, which don’t have their
own address space and exist within a process.
-Scripting languages use fork indirectly to start child processes. For example, every time you use a command sub process Popen in Python, you fork a child
process and read its output. This enables programs to work together.

The shell spawns a child process using exec and waits for it to complete, then continues with its own execution. Note that you don’t have to use fork this way.
You can always spawn off lots of child processes, as a parallel program might do, and each might run a program concurrently. Basically, any time you’re creating
new processes in a Unix system, you’re using fork(). For the Windows equivalent, take a look at CreateProcess.

When a user logs out of a system, the kernel needs to terminate all the processes the user had running (otherwise, users would leave a slew of old processes sitting around waiting for input that can never arrive). To simplify this task, processes are organized into sets of sessions.
The session’s ID is the same as the pid of the process that created the session through the setsid() system call. That process is known as the session
leader for that session group. All of that process’s descendants are then members of that session unless they specifically remove themselves from it.
The setsid()function does not take any arguments and returns the new session ID.

Every session is tied to a terminal from which processes in the session get their input and to which they send their output. That terminal may be the machine’s local console, a terminal connected over a serial line, or a pseudo terminal that maps to an X window or across a network  The terminal to which a session is related is called the controlling terminal (or controlling tty) of the session. A terminal can be the controlling
terminal for only one session at a time.

to check quota of usage

$quota -v

$df

$du -s [here s is for summary, use -h for readable format]

$gzip to zip

$gunzip to unzip

$zcat to read zipped files

$file [ to know the type of file]

$diff, sdiff [ difference between files]

$find . -name abc -print

$find . -size  +1M [ more than 1 MB]

$history

!! (recall last command)
% !-3 (recall third most recent command)
% !5 (recall 5th command in list)
% !grep (recall last command starting with grep)
You can increase the size of the history buffer by typing
% set history=100

The simplest way to compile a package is:
1. cd to the directory containing the package’s source code.
2. Type ./configure to configure the package for your system.
3. Type make to compile the package.
4. Optionally, type make check to run any self-tests that come with the package.
5. Type make install to install the programs and any data files and documentation.
6. Optionally, type make clean to remove the program binaries and object files from the source code directory

How to replace a certain character in Unix
sed -i ‘s/old-word/new-word/g’ *.txt

A file system is a logical collection of files on a partition or disk. A partition is a container for information and can span an entire hard drive if desired.
Your hard drive can have various partitions which usually contains only one file system, such as one file system housing the / file system or another containing the /home file system.

One file system per partition allows for the logical maintenance and management of differing file systems.
Everything in Unix is considered to be a file, including physical devices such as DVD-ROM’s, USB devices, floppy drives, and so forth.

Unix uses a hierarchical file system structure, much like an upside-down tree, with root (/) at the base of the file system and all other directories spreading from there.
A UNIX file system is a collection of files and directories that has the following properties

-It has a root directory (/) that contains other files and directories.
-Each file or directory is uniquely identified by its name, the directory in which it resides, and a unique identifier, typically called an inode.
-By convention, the root directory has an inode number of 2 and the lost+found directory has an inode number of 3. Inode numbers 0 and 1 are not used.
-File inode numbers can be seen by specifying the -i option to ls command.
-It is self contained. There are no dependencies between one filesystem and any other.
The directories have specific purposes and generally hold the same types of information for easily locating files. Following are the directories that exist on
the major versions of Unix.

/ This is the root directory which should contain only the directories needed at the top level of the file structure.
/bin This is where the executable files are located. They are available to all user.
/dev These are device drivers.
/etc Supervisor directory commands, configuration files, disk configuration files, valid user lists, groups, ethernet, hosts, where to send critical messages.
/lib Contains shared library files and sometimes other kernel-related files.
/boot Contains files for booting the system.
/home Contains the home directory for users and other accounts.
/mnt Used to mount other temporary file systems, such as cdrom and floppy for the CD-ROM drive and floppy diskette drive, respectively
/proc Contains all processes marked as a file by process number or other information that is dynamic to the system.
/tmp Holds temporary files used between system boots
/usr Used for miscellaneous purposes, or can be used by many users. Includes administrative commands, shared files, library files, and others
/var Typically contains variable-length files such as log and print files and any other type of file that may contain a variable amount of data

$which file [shows the location of the file if it is in your path]

-bash-3.2$ which alias.sh

/usr/bin/which: no alias.sh in (/opt/CA/UnicenterAutoSysJM/autosys/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin)

The du (disk usage) command enables you to specify directories to show disk space usage on a particular directory.

The command df -k (disk free) displays the disk space usage in kilobytes

A file system must be mounted in order to be usable by the system. To see what is currently mounted (available for use) on your system, use this command −
$ mount
/dev/vzfs on / type reiserfs (rw,usrquota,grpquota)
proc on /proc type proc (rw,nodiratime)
devpts on /dev/pts type devpts (rw)

For example, if you want to mount a CD-ROM to the directory /mnt/cdrom, for example, you can type −
$ mount -t iso9660 /dev/cdrom /mnt/cdrom
This assumes that your CD-ROM device is called /dev/cdrom and that you want to mount it to /mnt/cdrom. Refer to the mount man page for more specific
information or type mount -h at the command line for help information.
After mounting, you can use the cd command to navigate the newly available file system through the mountpoint you just made.

All files accessible in Unix, or a Unix-style system such as Linux, are arranged in one big tree: the file hierarchy, rooted at /.
These files can be spread out over several devices. The mount command attaches a filesystem, located on some device or other, to the file tree.
Conversely, the umount command will detach it again.
The standard form of the mount command is:

mount -t type device dir
This tells the kernel to attach the filesystem found on device (which is of type type) at the directory dir. The previous contents (if any), owner,
and mode of dir become invisible, and as long as this filesystem remains mounted, the pathname dir refers to the root of the filesystem on device.
If only directory or device is given, for example:

mount /dir
then mount looks for a corresponding mountpoint (and then, if not found, for a corresponding device) entry in the /etc/fstab file, and attempts to mount it.

Example: if I insert a USB stick in Windows I get a new drive, e.g., F:; if in Linux I mount it on directory /media/usb, then the top directory on the USB stick
(what I would see by opening the F: drive in Windows) will be visible in Linux as directory /media/usb. In this case, the /media/usb directory is called a “mount point”.
Now, drives/partitions/etc. are traditionally called “(block) devices” in the UNIX world, so you always speak of mounting a device on a directory. By abuse of language,
you can just say “mount this device” or “unmount that directory”.

Basically, and to put it easily, the operating system needs to know how to access the files on that device.
mount is not only “giving you access to the files”, it’s telling the OS the filesystem the drive has, if it’s read only or read/write access, etc.
/dev/cdrom is a low-level device, the operating system functions wouldn’t know how to access them… imagine you put a weirdly formatted cdrom in it
(even an audio cd), how would ls tell which files (if any) are there on the cd-rom without “mounting” it first?
Note that this happens automatically in many OSes (even Linux on some distributions and graphic interfaces), but that doesn’t mean other OSes are not “mounting” the drives.

 

An inode number stores all the information about a regular file, directory, or other file system object, except its data and name. To find an inode,
either use the ls or stat command.

-bash-3.2$ ls -li alias.sh
1118336 -rw-r—– 1 user u36061 793 Jul 24 02:10 alias.sh
-bash-3.2$ ls -d */
ls -l | grep ^d

gives only directory

-bash-3.2$ find /home/vxu/ -type d
gives sub directory

-bash-3.2$ find /home/lmp/ -user encuat

$ a=`echo 1+1 | bc -l`
$ echo $a
How to find length of string in UNIX
Echo ${#variable}

Cron Job

[Format]

M H Dom Mon Dow Command

Field Description Allowed Value
MIN Minute field 0 to 59
HOUR Hour field 0 to 23
DOM Day of Month 1-31
MON Month field 1-12
DOW Day Of Week 0-6
CMD Command Any command to be executed.

examples
1)30 08 10 06 * /home/ramesh/full-backup

30 – 30th Minute
08 – 08 AM
10 – 10th Day
06 – 6th Month (June)
* – Every day of the week

2)00 11,16 * * * /home/ramesh/bin/incremental-backup

00 – 0th Minute (Top of the hour)
11,16 – 11 AM and 4 PM
* – Every day
* – Every month
* – Every day of the week

3)00 09-18 * * * /home/ramesh/bin/check-db-status

00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
* – Every day of the week

4)00 09-18 * * 1-5 /home/ramesh/bin/check-db-status

00 – 0th Minute (Top of the hour)
09-18 – 9 am, 10 am,11 am, 12 am, 1 pm, 2 pm, 3 pm, 4 pm, 5 pm, 6 pm
* – Every day
* – Every month
1-5 -Mon, Tue, Wed, Thu and Fri (Every Weekday)

5)To view your crontab entries type crontab -l from your unix account as shown.Catch here is that you need to log in as root

6)To View Other Linux User’s Crontabs entries
crontab -u sathiya -l

7)crontab -e

8)To edit crontab entries of other Linux users, login to root and use -u
crontab -u sathiya -e

9)Schedule a Job for Every Minute Using Cron
* * * * * CMD

10)Schedule a Background Cron Job For Every 10 Minutes
*/10 * * * * /home/ramesh/check-disk-space

11)There are special cases in which instead of the above 5 fields you can use @ followed by a keyword — such as reboot, midnight, yearly, hourly.

k1.jpg
12)Schedule a Job For First Minute of Every Year using @yearly
@yearly /home/ramesh/red-hat/bin/annual-maintenance

13)Schedule a Cron Job Beginning of Every Month using @monthly
@monthly /home/ramesh/suse/bin/tape-backup

14)Schedule a Background Job Every Day using @daily
@daily /home/ramesh/arch-linux/bin/cleanup-logs “day started”

15)How to Execute a Linux Command After Every Reboot using @reboot?
@reboot CMD

16)How to Execute a Linux Cron Jobs Every Second Using Crontab.
You cannot schedule a every-second cronjob. Because in cron the minimum unit you can specify is minute.
In a typical scenario, there is no reason for most of us to run any job every second in the system.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s