Total Hit Counter

Monday, December 3, 2012

Script to monitor error in log file



Script to monitor error in log file


#!/bin/bash
#####
# The Script will check the pattern in mentioned file
# Author: Shankar Patel
################################################################################
function help {
echo "The plugin will check the pattern in mentioned file
Usage:
$0 -p <Pattern> -f <file>
Options:
p - Patern to be matched
f - file to monitor given pattern.
h - help.
examples:
$0 -p 'file' -f /var/log/messages
OK. 'file' not found in /var/log/messages | error_found=0"
exit 3;
}

while getopts "p:f:h" OPT; do
case $OPT in
"p") PATTERN=$OPTARG;;
"f") FILE2TAIL=$OPTARG;;
"h") help ;;
esac
done

if [ "$PATTERN" = "" ] || [ "$FILE2TAIL" = "" ] ; then help ; fi

FPATH=$(dirname $FILE2TAIL)
FFILE=$(basename $FILE2TAIL)
INODE_NUM=$(ls -i ${FPATH}/${FFILE} | awk '{ print $1 }')
PATTERN_CODE="$(echo -n "$PATTERN"|tr -d \* | tr -d ' ' | tr -d \. )"
FSIZE_FILE=${FPATH}/.${FFILE}_${PATTERN_CODE}_${INODE_NUM}.size
SIZE_CURRENT=$(ls -l ${FPATH}/${FFILE} | awk '{ print $5 }')
exit_status=0

last_exit()
{# save current size
ls -l ${FPATH}/${FFILE} | awk '{ print $5 }' > ${FSIZE_FILE}
/bin/rm -f /tmp/ts$$ /tmp/tsF$$
exit $exit_status
}
if [ ! -f ${FSIZE_FILE} ]; then
# first time, save line count
echo "OK. saving initial size of ${FPATH}/${FFILE} | error_found=0"
last_exit
fi

# last size is available
SIZE_LAST=$(cat ${FSIZE_FILE})
SIZE_DELTA=$(($SIZE_CURRENT - $SIZE_LAST))
if [ $SIZE_DELTA -gt 0 ]
then
# new bytes to scan
tail "-${SIZE_DELTA}c" ${FPATH}/${FFILE} > /tmp/ts$$
egrep "$PATTERN" /tmp/ts$$ > /tmp/tsF$$
GREP_STATUS=$?
if [ $GREP_STATUS -eq 0 ]
then
echo "CRITICAL. '$PATTERN' found in '${FFILE}' file | error_found=1"
exit_status=2
last_exit
else
# found no matches
echo "OK. '$PATTERN' not found in '${FFILE}' file | error_found=0"
exit_status=0
last_exit
fi

else
# found no matches
echo "OK. '$PATTERN' not found in '${FFILE}' file | error_found=0"
exit_status=0
last_exit
fi

Thursday, November 1, 2012

How to check memory used by process




Shell Script to check Memory usage by Process  in linux


#!/bin/bash
#####
# To check memory used by particular process
#Created by Shankar Patel
#Create date : 1-11-2012
########################################################################
if [ "$1" = "" ] ; then  echo -e "Please Provide process name as a argument . \nExample: $0 java" ; exit 1;fi 
pidsof_procs=`pidof $1`
for i in $pidsof_procs
do
        mem[$i]="`cat /proc/$i/status | awk '/VmPeak/ {print $2}'`"
        #cat /proc/$i/status | awk '/VmPeak/ {print $2}'
done
for i in $pidsof_procs
do
        echo -e "$i is using $((${mem[i]}/1024)) mb"
done
exit 0



Friday, October 26, 2012

Time of command execution


How to get time of execution of any command in Unix/Linux

Simply execute the following command on shell prompt and you will get date and time execution of any command.

$ export HISTTIMEFORMAT='%F %T '

To make it permanent edit the

To display timezone:
$ export HISTTIMEFORMAT='%d-%b-%Y %r %Z'
$ history
351 08-Sep-2009 09:53:47 PM PDT ls -lrt
352 08-Sep-2009 09:53:49 PM PDT cd ..

To display AM / PM: You can use %r option
To display in dd-MON-YY time format: You can format it as you want %d-%b-%Y %r


Tuesday, October 23, 2012

Find examples with exec command


Search files with find and delete them with exec, this is probably one of the most common actions with exec, and you should not use exec for this, read later, here are some examples of common uses:

1) List all files starts with “abc”:
 find / -name "abc*" -exec /bin/ls {} \;

2) Search all files start with “abc” and delete them:
 find / -name "abc*" -exec /bin/rm {} \;
3) Search all files with size > of 10 MB and delete them:
 find / -size +10M -exec /bin/rm {} \;
Sometimes some programs goes wild and create thousands of small files into one directoy, in this case you cannot use a simple rm * because the shell would not be able to manages the expansion of the character * with all these file names, but you can use find to delete all files in a directory one by one.
 find . -exec /bin/rm {} \;
You should NOT use these examples, In newer verison you will find the option -delete which is safer then “-exec /bin/rm {} ;”. For example:
find / -name "*.old" -delete
In older Unix system you could not have the -delete option, and so you have no choice but to use the -exec action.
4) To change permissions on files recursively, leave directories alone.
find ./ -type f -exec chmod 755 {} \;
5) With the option -type f you select only the files and after that is easy to do a chmod on them. Recursively change the ownership of all the files from old user to new user
find / -user test_old  -type f  -exec chown  test_new {} \;

6) Recursively change the permissions of all, and only, the directory
find . -type d -exec chmod 655 {} \;
In this example I’ve used again the option -type with d parameter to identify only the directories.

Monday, October 22, 2012

Shell Script Interview questions and answers


Basic shell scripting questions


  1. How do you find out what’s your shell? - echo $SHELL
  2. What’s the command to find out today’s date? - date
  3. What’s the command to find out users on the system? - who
  4. How do you find out the current directory you’re in? - pwd
  5. How do you remove a file? - rm
  6. How do you remove a - rm -rf
  7. How do you find out your own username? - whoami
  8. How do you send a mail message to somebody? - mail somebody@techinterviews.com -s ‘Your subject’ -c ‘cc@techinterviews.com‘

  9. How do you count words, lines and characters in a file? - wc
  10. How do you search for a string inside a given file? - grep string filename
  11. How do you search for a string inside a directory? - grep string *
  12. How do you search for a string in a directory with the subdirectories recursed? - grep -r string *
  13. What are PIDs? - They are process IDs given to processes. A PID can vary from 0 to 65535.
  14. How do you list currently running process? - ps
  15. How do you stop a process? - kill pid
  16. How do you find out about all running processes? - ps -ag
  17. How do you stop all the processes, except the shell window? - kill 0
  18. How do you fire a process in the background? - ./process-name &
  19. How do you refer to the arguments passed to a shell script? - $1, $2 and so on. $0 is your script name.
  20. What’s the conditional statement in shell scripting? - if {condition} then … fi
  21. How do you do number comparison in shell scripts? - -eq, -ne, -lt, -le, -gt, -ge
  22. How do you test for file properties in shell scripts? - -s filename tells you if the file is not empty, -f filename tells you whether the argument is a file, and not a directory, -d filename tests if the argument is a directory, and not a file, -w filename tests for writeability, -r filename tests for readability, -x filename tests for executability
  23. How do you do Boolean logic operators in shell scripting? - ! tests for logical not, -a tests for logical and, and -o tests for logical or.
  24. How do you find out the number of arguments passed to the shell script? - $#
  25. What’s a way to do multilevel if-else’s in shell scripting? - if {condition} then {statement} elif {condition} {statement} fi
  26. How do you write a for loop in shell? - for {variable name} in {list} do {statement} done
  27. How do you write a while loop in shell? - while {condition} do {statement} done
  28. How does a case statement look in shell scripts? - case {variable} in {possible-value-1}) {statement};; {possible-value-2}) {statement};; esac
  29. How do you read keyboard input in shell scripts? - read {variable-name}
  30. How do you define a function in a shell script? - function-name() { #some code here return }
  31. How does getopts command work? - The parameters to your script can be passed as -n 15 -x 20. Inside the script, you can iterate through the getopts array as while getopts n:x option, and the variable $option contains the value of the entered option.

Shell scripts






Check multiple websites' status.
[ Note: First copy the check_http plugin in /usr/lib/nagios/plugins/ path ]

#!/bin/bash

#####
#Script will check all given websites and give status by the help of check_http plugin
#Created by Shankar Patel
#Create date :10-10-2012
#######################################################################

F_P=""
msg_ok=""
exit_status=0
#also you can create a file with the list of websites and 
#SITES_LIST="file_name_of_list_websites"
#for WEB in $(cat $SITES_LIST)
for WEB in {staging.sabsebolo.com,staging.sabsetalk.com,google.com,yyyyyyyyyyyyyyyyahoo.com}
do
/usr/lib/nagios/plugins/check_http $WEB > /dev/null
result_check=$?
if [ "$result_check" -ne 0 ] ; then
       F_P="$F_P $WEB"
       exit_status=2
else
       msg_ok="$msg_ok $WEB"
       exit_status=0
fi
done
if [ "$F_P" = "" ]
then
      echo "OK. All Websites are OK."
else
      echo "CRITICAL. $F_P websites showing down. and $msg_ok are up."
      exit $exit_status;
fi




Script Daily Backup To remote location

#!/bin/bash
SSHKEY=/backup/id_rsa
TOBACKUP=$1
HOSTNAME=$(hostname)
TODAY=`date +%Y%m%d`
check_time=0
tar_done=1
tar_done=1
#create tar file
tak_bkp()
{
 if [ "$check_bkp" -lt 4 ]
 then
        scp -o StrictHostKeyChecking=no -i $SSHKEY /backup/${HOSTNAME}_${TODAY}.tar.gz incoming-backup@heimdall:/mnt/datastore1/incoming-backup/ > /backup/backup_done
        [ "$?" -eq 0 ] && rm /backup/${HOSTNAME}_${TODAY}.tar.gz && exit 0 || check_bkp=$(($check_bkp+1)) && tak_bkp ;
 else
        echo -e "Can not create BKP file! \n ackup not done properly" > /backup/backup_done
        exit 2;
 fi

}
create_tar()
{
 if [ "$check_tar" -lt 4 ]
 then
   tar -cpzf /backup/${HOSTNAME}_${TODAY}.tar.gz --exclude="${HOSTNAME}_${TODAY}.tar.gz" --one-file-system  $TOBACKUP
   [ "$?" -eq 0 ] && tak_bkp
   check_time=$(($check_time+1))
 else
        echo -e "Can not create tar file! \nbackup not done properly" > /backup/backup_done
        exit 2;
 fi
}

create_tar


Script to check backup done or not 


#!/bin/bash
#####
#Script will check backup copied or not on remote location.
#Created by Shankar Patel
#Create date :10-10-2012
#######################################################################
TODAY=`date +%Y%m%d --date="1 day ago"`
bkp_exist=`ssh -o StrictHostKeyChecking=no -i /backup/id_rsa incoming-backup@heimdall "ls /mnt/datastore1/incoming-backup/${HOSTNAME}_${TODAY}.tar.gz &>/dev/null && echo -e 'OK' || echo -e 'CRITICAL'" 2>/dev/null`
[ "$bkp_exist" == "OK" ] && echo "OK. Yesterday's backup done."  && exit 0
[ "$bkp_exist" == "CRITICAL" ] &&  echo "CRITICAL. Yesterday's backup done." && exit 2




Nagios plugin to check Openfiles
check_openfiles.sh 

if it is greater then 85% then it will show warning
if it is greater then 90% then it will show warning
otherwise it will show ok. 



#!/bin/bash
###########
# This script will check no open files and if open files will be greater then 85% of max-open-files then it will show warning and open files will be greater then 90% then it will show critical.
#
# It does not require any additional perameter at execution time
# it automatically takes perameter from system files
########### Shankar Patel 22/08/1012

open_files=`cat /proc/sys/fs/file-nr | awk '{print $1}'`
max_open_files=`cat /proc/sys/fs/file-nr | awk '{print $3}'`

warn_of=$(($max_open_files*15/100))
crit_of=$(($max_open_files*10/100))

if [ "$open_files" -lt "$warn_of" ]
then
echo "OK. Number of open files = $open_files | openfiles= $open_files"
exit 0;
elif [ "$open_files" -gt "$warn_of" ] && [ "$open_files" -lt "$crit_of" ]
then
echo "WARNING. Number of open files = $open_files | openfiles= $open_files"
exit 1;
else
echo "CRITICAL. Number of open files = $open_files | openfiles= $open_files"
exit 2;
fi


Script to monitor log file error.


#!/bin/bash

FILE2TAIL=$1
PATTERN=$2
CATCH_UP=0
FPATH=$(dirname $1)
FFILE=$(basename $1)
INODE_NUM=$(ls -i ${FPATH}/${FFILE} | awk '{ print $1 }')
PATTERN_CODE="$(echo -n "$PATTERN"|tr -d \* | tr -d ' ' | tr -d \. )"

FSIZE_FILE=${FPATH}/.${FFILE}_${PATTERN_CODE}_${INODE_NUM}.size

SIZE_CURRENT=$(ls -l ${FPATH}/${FFILE} | awk '{ print $5 }')
exit_status=0

last_exit()
{
# save current size
ls -l ${FPATH}/${FFILE} | awk '{ print $5 }' > ${FSIZE_FILE}
exit $exit_status
}
if [ ! -f ${FSIZE_FILE} ]; then
# first time, save line count
echo "saving initial size of ${FPATH}/${FFILE}"
last_exit
fi
# last size is available
SIZE_LAST=$(cat ${FSIZE_FILE})

SIZE_DELTA=$(($SIZE_CURRENT - $SIZE_LAST))

if [ $SIZE_DELTA -gt 0 ]
then
# new bytes to scan
tail "-${SIZE_DELTA}c" ${FPATH}/${FFILE} > /tmp/ts$$
egrep "$PATTERN" /tmp/ts$$ > /tmp/tsF$$
GREP_STATUS=$?
if [ $GREP_STATUS -eq 0 ]
then
# found a match
SUBJECT="Found $PATTERN in $FILE2TAIL"
BODY="$SUBJECT\n$(cat /tmp/tsF$$)"
echo "$SUBJECT"
#### exec
#cho -e $BODY # | $EXEC_ON_MATCH -s "$SUBJECT" $EMAILADDR
echo "CRITICAL. 'Error writing data to the connection' found in ${FFILE} | error_found=1"
last_exit
fi

if [ $GREP_STATUS -ne 0 ]
then
# found no matches
SUBJECT="Missing $PATTERN in $FILE2TAIL on $HOST_NAME"
BODY="$SUBJECT\n"
echo "OK. 'Error writing data to the connection' not found in ${FFILE} | error_found=0"
last_exit
#### exec
fi

/bin/rm -f /tmp/ts$$ /tmp/tsF$$
fi


find out notusable data in linux


How to keep disk-space to free from not usable data in LINUX/Unix:


Agedu: Correlate disk usage with last-access times to identify large and disused data
                 agedu scans a directory tree and produces reports about how much disk space is used in each directory and subdirectory, and also how that usage of disk space corresponds to files with last-access times a long time ago.
                 In other words, agedu is a tool you might use to help you free up disk space. It lets you see which directories are taking up the most space, as du does; but unlike du, it also distinguishes between large collections of data which are still in use and ones which have not been accessed in months or years - for instance, large archives downloaded,unpacked, used once, and never cleaned up. Where du helps you find what's using your disk space, agedu helps you find what's wasting your disk space.

How to install:
==> for Debian / Ubuntu Linux user type the following apt-get command to install agedu:
$sudo  apt-get install agedu

==> for RHEL / CentOS / Fedora users  turn on EPEL repo and type the following
#yum install agedu

Example how to use it:

Step1: The following command will conllect information about disk used by files in /home/shankar directory.

agedu -s /home/shankar/

Step2: Then execute the following command on command prompt to view the disk usage in web browser:
agedu -w
you will see the following output :
Using Linux /proc/net magic authentication
URL: http://127.0.0.1:42823/

Use this url in web browser and you will see the file and folder level usage graphically.
Note: Don't stop this command.

For more information use man agedu



Thursday, October 18, 2012

ss command


How to monitor current TCP and UDP connections


ss :- Used to investigate tcp/udp sockets

USAGE EXAMPLES
       ss -t -a
              Display all TCP sockets.

       ss -u -a
              Display all UDP sockets.

       ss -o state established '( dport = :ssh or sport = :ssh )'
              Display all established ssh connections.

       ss -x src /tmp/.X11-unix/*
              Find all local processes connected to X server.

       ss -o state fin-wait-1 '( sport = :http or sport = :https )' dst 193.233.7/24 
              List all the tcp sockets in state FIN-WAIT-1 for our apache to network 193.233.7/24 and look at their timers.



MAN page :

DESCRIPTION
                      ss is used to dump socket statistics. It allows showing information similar to netstat.  It can display more TCP and state informations than other tools.

OPTIONS
       These programs follow the usual GNU command line syntax, with long options starting with two dashes (`-').  A  summary  of  options  is included below.

       
-h, --help
              Show summary of options.

       
-V, --version
              Output version information.

       
-n, --numeric
              Do now try to resolve service names.

       
-r, --resolve
              Try to resolve numeric address/ports.

       
-a, --all
              Display all sockets.

       
-l, --listening
              Display listening sockets.

       
-o, --options
              Show timer information.

       
-e, --extended
              Show detailed socket information

       
-m, --memory
              Show socket memory usage.

       
       -p, --processes
              Show process using socket.

       
-i, --info
              Show internal TCP information.

       
-s, --summary
              Print  summary  statistics.  This  option  does not parse socket lists obtaining summary from various sources. It is useful when amount of sockets is so huge that parsing /proc/net/tcp is painful.

       -
4, --ipv4
              Display only IP version 4 sockets (alias for -f inet).

       
-6, --ipv6
              Display only IP version 6 sockets (alias for -f inet6).

       
-0, --packet
              Display PACKET sockets.

       
-t, --tcp
              Display only TCP sockets.

       
-u, --udp
              Display only UDP sockets.

       
-d, --dccp
              Display only DCCP sockets.

       
-w, --raw
              Display only RAW sockets.

       
-x, --unix
              Display only Unix domain sockets.

       
-f FAMILY, --family=FAMILY
              Display sockets of type FAMILY.  Currently the following families are supported: unix, inet, inet6, link, netlink.

       
-A QUERY, --query=QUERY
              List of socket tables to dump, separated by commas. The following identifiers are understood: all, inet, tcp,  udp,  raw,  unix, packet, netlink, unix_dgram, unix_stream, packet_raw, packet_dgram.

       
-D FILE
              Do  not  display  anything,  just  dump raw information about TCP sockets to FILE after applying filters. If FILE is - stdout is Manual page ss(8) line 47
        -f FAMILY, --family=FAMILY
              Display sockets of type FAMILY.  Currently the following families are supported: unix, inet, inet6, link, netlink.

       -A QUERY, --query=QUERY
              List of socket tables to dump, separated by commas. The following identifiers are understood: all, inet, tcp,  udp,  raw,  unix,
              packet, netlink, unix_dgram, unix_stream, packet_raw, packet_dgram.

       -D FILE
              Do  not  display  anything,  just  dump raw information about TCP sockets to FILE after applying filters. If FILE is - stdout is
              used.

       -F FILE, --filter=FILE
              Read filter information from FILE.  Each line of FILE is interpreted like single command line option. If  FILE  is  -  stdin  is
              used.

       FILTER := [ state TCP-STATE ] [ EXPRESSION ]
              Please take a look at the official documentation (Debian package iproute-doc) for details regarding filters.



PS command

ps ax -o user,vsz,rss,pcpu,command --columns 10000 | sed -e 1d -e 's/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4) /'