Digitally Enlightened

Shell scripting best practices

Thu, 28 May 2020 05:14:37 +0000

Majority of time shell script is used for doing some “hack” job of sewing together various utilities and their output. While shell scripting is easy and straight-forward most of the time, it does pop surprises time to time. End result is not just bugs but silent bugs. It would save lot of headache and debugging hours to treat shell scripting as proper programming language and use coding discipline as demanded by other high level languages. For very complex jobs, it might be worth considering python, as after all shell scripting comes with very limited vernacular. Following are some pointers for robust shell programming. Note that there are variations between various shells and POSIX standard. You might want to try out and change usage accordingly.

Environment

Using env ensures portability vs using direct path to shell.

#!/usr/bin/env bash

Declare variables

It is a good practice to declare variables upfront in shell script. Global variables should be upper case and local should be declared early in function.

GLOBAL_VAR=
foo() {
  local var1 var2=0
  var1=${GLOBAL_VAR:-100} # if GLOBAL_VAR is empty use 100 as default
}

Notice, no use of keyword ‘function’. Also, surround variables in {} to avoid shell incorrectly interpreting variable name. For special built-in variables like $@, always use quotes.

Prefer `[[` over `[`

While [ makes your script portable (as this is POSIX compliant condition test), all shell support [[ now which is more robust and extends functionality. With [[ conditional construct, quotes are not needed for variables, supports extended pattern matching, multiple conditions can be naturally combined (instead using -o or -a etc), and various other things.

Exit if externally sourced file doesn’t exist

Instead of sourcing an external file and moving on, use circuit breaker to exit immediately. Otherwise, your script might continue and either print lot of error messages or even silently corrupt data, depending on what was used from external file.

. common_file || exit 1

Check return code of all external commands or utils

If an external command can fail, then check it’s return code. Such commands can also be wrapped in a general function.

var=$( cmd )
[[ $? -ne 0 ]] && exit 2

Use whitespace with $( )

Makes it easier to read.

PROGNAME=$( basename $0 )

Use $() instead of ``

For command substitution, use $(). The commands can be further nested with it.

last_line=$( dmesg | tail -n1 )
all_lines=$( echo $( cat file ) )

When changing directories, use directory stack

If supported by your shell, it’s better to use pushd and popd for saving and returning current working directory instead explicitly using cd.

( pushd /tmp; ls )

Use ((var++))

Instead of let var=var+1. Same for substraction and other arithmatics.

Use regex expression

For complex regular expressions, use a separate variable for regex pattern.

re="^[0-9]{2}-[0-9]{4}-[0-9]{2} *$"
if [[ ! $serial_num =~ $re ]]; then
fi

Use signal traps for cleanup

Traps can be used to clean temporary resources (lockfiles, temp files), or even rollback to previous state (say if setup of files were being modified and one failed).

trap cleanup EXIT INT TERM
cleanup() {
  echo "Cleaning up resources ..."
  rm -f $lockfile
  printf \\e[2J\\e[H\\e[m    # cls
  trap - EXIT INT TERM       # clear traps and exit
  exit 0
}

Use `set` builtins for robust script

These options could be either set globally, for a function, or whithin a block, depending context. Some of the most common and useful ones are listed below.

errexit (-e) makes your script exit when failure occurs. You can allow script to continue on a failing command by adding || true.

nounset (-u) exit when your script tries to use undeclared variables.

pipefail will fail your script if any of command in pipe chain fail. Return code is set to one from failing command. Default is return status of last item in pipe chain.

Set these options explicitly instead of #!/usr/bin/env bash -e.

set -euo pipefail

Reading a file

file="$(<"file")"         # to read whole file at once. Make sure size is manageable before reading it.
IFS=$'\n' read -d "" -ra file < "file" # read line by line
mapfile -t file < "file"  # read line by line. mapfile is available in bash 4.

Also, using while read pattern can be useful in specific situations.

awk '{print $2}' file.txt | while read num
do
  if [ "$num" = "0" ]; then
    echo "zero it is!"
  fi
done

Avoid temporary files for ephemeral use

Instead use <(cmd) which transforms output into something which can be used a file.

Using lockfiles without race conditions

First checking if lockfile exists and then creating one if not, obviously has race condition. Following can be used instead.

if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null;
then
   trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
   do_critical_section
   rm -f "$lockfile"
   trap - INT TERM EXIT
else
   echo "failed to acquire lock, held by $(cat $lockfile)."
fi

On Linux, one can use flock instead.

Use colors but with caution

Using colors, particularly when dumping lot of information, is very useful. However, some terminals might not be able to render colors and could screw up the output.

Debugging options

set -n to dry run the script. Useful to do syntax check.

set -v to print every command run

set -x to trace every command and expanded use

Instead of setting these variables, it might be useful control debugging behavior via global DEBUG variable. echo can be wrapped around with another print function, which changes verbosity based on DEBUG variable.

debug() {
  ((DEBUG)) && echo ">>> $*";
}

Debug with function names

While printing debug info, FUNCNAME[x] can be used to print current or prior functions in call stack. FUNCNAME[@] is array of all functions in call chain.

"${FUNCNAME[0]}"
"${FUNCNAME[@]}"

Use here-docs instead of individual prints

cat << EOF
usage: $PROGNAME <arg> <arg>
version: 0.1
EOF

Use brace expansion

echo a{d,c,b}e    # will print - ade ace abe
echo repos/ntrivedi/code/{0,1}{1..9}.cpp  # cartesian product of two braces

Use parameter expansion

${parameter:-word}          # if `parameter` unset or null, use `word` as substitute
${parameter:=word}          # if `parameter` unset or null, `word` is assigned to parameter
${parameter:?word}
${parameter:+word}
${parameter:offset:length}  # substring expansion

Some one-liners and clever ways of using shell script

Collected from internet over a period of time

Remove strangely named files

touch \-test
rm -- -test # dangerous
rm ./-test # remove using ./
find . -inum $inode -exec rm -i {} \;  # remmove by inode. Only way for file named '01/01/2001'

As RPC

ssh user@remote "
     $(declare -p var1 var2 var3)
     $(declare -f func1 func2 remotemain)
     remotemain
   "
ssh user@remote "dump_logs | gzip -c" | gunzip -c | read_logs

find the longest string matching “*/” from first in $0
```
${0##*/}
```
find the longest string matching “b” from last in $string
```
${str%%b}
```

Find email addresses in a file

grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file.txt

Print 5th to 10th line in file

sed -n '5,10p' file                    # using sed
awk 'NR>=5{print} NR==10{exit}' file   # using awk

Extract lines between two patterns
```
sed -n '/pattern1/,/pattern2/p' file
```

Remove dups but preserve the order

awk '!visited[$0]++' your_file > deduplicated_file

Copy from bash to clipboard using OSC command

echo -e "\033]52;c;$(base64 <<< "copy something here")\a"

Raise notification from terminal using OSC command
```
echo -e "\033]9;This is a notification\a"
```

References

Books read in 2019

Sun, 05 Jan 2020 05:28:07 +0000

All the books read in 2019.

“Never split the difference” - Chris Voss, Tahl Raz
“The Cartoon Introduction to Economics: Volume One: Microeconomics” - Yoram Bauman, Grady Klein
“How to solve it” - G. Polya
“Seven Brief Lessons on Physics” - Carlo Rovelli
“On writing: A Memoir of the Craft” - Stephen King
“Immortal Talks- Book 1” - Shunya
“A mind for numbers: How to Excel at Math and Science (Even If You Flunked Algebra)” - Barbara Oakley
“Why I am an atheist” - Bhagat Singh
“Thinking in Systems: A primer” - Donella H. Meadows
“The Supermen: The Story of Seymour Cray and the Technical Wizards Behind the Supercomputer” - Charles J. Murray
“Weird things customers say in bookshops” - Jen Campbell
“The Field Guide to Dumb Birds of North America” - Matt Kracht
“Scary stories to tell in the Dark” - Alvin Schwatz, Stephen Gammell
“More Scary stories to tell in the Dark” - Alvin Schwatz, Stephen Gammell
“Fearless Salary Negotiation” - Josh Doody

Memory aligned malloc

Sat, 01 Jun 2019 16:22:07 +0000

A very common interview question is to allocate memory aligned to given block size, such as memalign_malloc(size_t bytes, size_t align). Despite being common, I have found that not many people really understand the implementation, including the interviewer. In practice, one would either totally avoid such aligned allocation (relying on machine natural alignment using malloc) or use native aligned_alloc(), or memalign(), or posix_memalign() family.

When allocating memory through malloc (or its cousins), it returns a block of memory which is multiple of 8 or 16 for 32 or 64-bit system, resp. However, in some special cases such alignment might not be enough such as in an embedded system, while implementing your own memory manager (e.g. GC), for allocating specifically on page boundary, while interfacing with certaing I/O hardware, or for CPU cache alignment, and other such cases. Hence, there exists memalign and similar routines.

To implement memalign_malloc, our first intution might be to allocate requested size block plus extra memory to adjust for misaligned address from malloc. However, if we return this new aligned address then while using memalign_free we might leave a hole between malloc returned address and aligned address, and possibly corrupt memory too. We need a place to save the original address as that is what should be used by free which would be eventually called by memalign_free. Most memory allocators keep some metadata associated with each block of allocated memory which might contain block size, actual address, access/proection info, etc. For memalign_malloc, we would only store the original address as returned by malloc at some location. Our complementary memalign_free() would know where to pick this address from and eventually call free using this address.

How much extra space should we allocate? It could be that the address returned by malloc needs adjustment by upto (align - 1). So, we need to allocate (align - 1) extra bytes just for this adjustment. Also, we need to save the original address as returned by malloc, which needs space equal to the size of a pointer for underlying machine. Total memory we allocate is -

size_t extra = (align - 1) + sizeof(void *);
void *mem = malloc(bytes + extra);

The trick for aligning an address is by masking the lower bits of address which correspond to alignment size such as address & ~(align - 1). So, if we were trying to align to 16 bytes, ~(16 - 1) or a mask of 0xF is required, since inverting only those bits and ‘and’ing would mean those many last bits would be 0 in address, i.e. new address will be multiple of 16. Note, that due to this calculation the alignment size can only be for power of 2. The new aligned address is going to be somewhere between mem and mem + (align-1). But we also need to save address mem and best location would be to store it right before new aligned address. So, best would be to find aligned address from mem + sizeof(void *) to mem + extra. Our new address will become (mem + extra) & ~(align-1). But that’s not enough - mem is pointer so we can’t do normal arithmatic with it. To circumvent that, we can typecast it to uintptr_t.

void *new_address = ((uintptr_t)mem + extra) & ~(align-1));

Now only remaining piece is how to save mem? We already know it must be saved right before new_address for easy acccess as well as to avoid being trampled by data. A neat trick is to treat this block of memory as an array to pointers, and then if arr[0] points to new_address, arr[-1] is where we can store mem. When treating block as an array, when we access previous or later element of array, stride will be exactly the size of element (in out case it would be void *), so now we dont need to typecast, add, subtract, etc. With that new address can also be seen as array of pointers.

void **ptr_array = (void **)(((uintptr_t)mem + extra) & ~(align-1));

Notice use of (void **). In C, array is nothing but contiguous block of memory. Its only when this block is accessed, one can access it as array (and move as much as size of element) or regular bytes. So array of interegers can be represented as int arr[] or int *arr. Hence, an array of (void *) can be void *arr[] or void **arr and still one can use arr[0] format. That means in above code snippet, we can use ptr_array[-1] and it’ll be essentially (ptr_array - sizeof(void *)). That’s where we can save mem.

ptr_array[-1] = mem;

To free this memory block, all we need to do is free((void **)addr[-1]). With that our final code looks like following -

void *memalign_malloc(size_t bytes, size_t align)
{
    size_t extra = (align - 1) + sizeof(void *);
    void *mem = malloc(bytes + extra);
    void **ptr_array = (void **)(((uintptr_t)mem + extra) & ~(align-1));
    ptr_array[-1] = mem;
    return ptr_array;
}

void memalign_free(void *addr)
{
    if (addr) {
        free(((void **)addr)[-1]);
    }
}

Life as summarized in Lisp

Sat, 01 Jun 2019 07:15:49 +0000

You have a right to perform your prescribed duty, but you are not entitled to the fruits of action. - Bhagavad Gita

(define life
    (lambda (karma)
        (cond
            ((null? karma) 'moksha)
            (else (life (cdr karma))))))

(define karma '(ma phaleshu kadachana))

(life karma)

Welcome to digital enlightment

Sat, 01 Jun 2019 06:55:49 +0000

A new blog post using jekyll and github pages. The theme is taken from Agus Makmun’s layout and scantily modified.

Another theme I liked was Athena but could not make it work. Also tried using Octopress, but will wait until version 3.0 is released.