1. when which does not work
  2. The which command executable cannot find shell alies and shell builtins (such as cd). To see those, use:

    type -a some-alias-or-shell-bultin

  3. how to find a user's login shell
  4. Most of the time this is stored in the /etc/passwd file so the following would work:

    grep username /etc/passwd

    However, when some kind of directory service is used the information may not reside there. So, the following always works:

    getent passwd username

  5. on the various forms of boolean tests in Bash using (), [] and [[]]
  6. This is explained wonderfully in this Unix stackexchange post.

    TL;DR:

    if command
    execute command and proceed based on exit code
    if (command)
    same as above but execute command in subshell
    if ((arithmetic condition))
    this is no longer a command but an arithmetic condition (NB: non-zero result is interpreted as true)
    if [ condition ]
    some condition like -e
    if [[ condition ]]
    same as above but with some additional goodies

    Below are some examples to build up intution:

    $ ((1+1))
    $ echo $?
    0
    $ ((1-1))
    $ echo $?
    1
    $ if ls > /dev/null ; then echo 'a'; else echo 'b'; fi
    a
    Some other examples:
    $ touch foo ; if [ -e foo ] ; then echo 'a'; else echo 'b' ; fi
    a
    $ rm -f foo ; if [ -e foo ] ; then echo 'a'; else echo 'b' ; fi
    b
    $ if ((1+2>2)); then echo 'a'; else echo 'b'; fi
    a
    
    The below prints a because the assignment succeeds and so the condition for the if statement is evaluated to true:
    $ a=0; if [ a=1 ]; then echo 'a'; else echo 'b'; fi
    a
    … the correct way is to use arithmetic evaluation:
    $ a=0; if (( $a==1 )); then echo 'a'; else echo 'b'; fi
    b
    The below prints nothing as the assignment was done in a subshell:
    $ ( b=1 ); echo $b
    
    

  7. make set print only the shell variables, not the functions
  8. (source)
    (set -o posix; set)
  9. env vs. printenv vs. set
  10. env is POSIX, printenv isn't. Other than that, in my system env and printenv yield identical output.

    set however is a different beast and provides more comprehensive output as it includes both shell-local variables (since it's a shell builtin it can see them) and environment variables which are passed on to every executed program. env however is a program and so can only see environment variables, not shell-local variables (source)

    BTW, the fact that set is a built-in can be established by the fact that which set produces no output.

  11. backticks (`) vs. $()
  12. The grave (`) was the original syntax and should not be used any more unless you want your scripts to run in old Bourne shells. Otherwise simply use:
  13. how to execute a command when one ore more files change
  14. Sources:
    There's two ways:
  15. how to write a script that executes sudo with the password
    1. Create a file:
    2. emacs ~/.easy.install
    3. inside the file invoke sudo with -S:
    4. echo <password> | sudo -S do-your-shit
    5. make the file executable
    6. sudo chmod +x ~/.easy.install
    7. Change the permissions of the file so only root can read and edit it:
    8. sudo chmod 700 ~/.easy.install
  16. how to process the output of find when there are spaces in the filenames
  17. source
    Replace the usual for iteration with a while iteration that is line-based as opposed to word-based:
    find . -iname Thundercats\ -\ 10\*  | while read f; do  sudo cp "$f" /media/mperdikeas/thundercats-season-1/; done
          
  18. how to put an scp that requires a password in a script file
  19. Use sshpass as shown below:
    sshpass -f $PASS_FILE scp rawdar.zip user@$DEST:~/rawdar.war
          
    If, after the copy, you also need to ssh to the destination and you further need sudo privilleges, sshpass can't supply both passwords, so you need to put the action that requires sudo in a script and execute it. E.g. as shown below:
    sshpass -f $PASS_FILE ssh user@$DEST <<'ENDSSH'
    ./publish-rawdar.sh
    ENDSSH
          
    ... where the script at the ssh destination is:
    user@user-virtual-machine:~$ cat publish-rawdar.sh
    #!/usr/bin/env bash
    readonly ECHO=/bin/echo
    $ECHO  'top-secret-password' | sudo -S cp rawdar.war /var/lib/tomcat7/webapps
    user@user-virtual-machine:~$
          
    Relevant excrepts from functional file:
    #!/usr/bin/env bash
    readonly PASS_FILE=~/esac.neuropublic.pwd
    readonly DEST=172.31.148.22
    rm -f rawdar.zip && zip -r rawdar.zip rawdar.html rawdar.org.files/
    sshpass -f $PASS_FILE scp rawdar.zip user@$DEST:~/rawdar.war
    sshpass -f $PASS_FILE ssh user@$DEST <<'ENDSSH'
    ./publish-rawdar.sh
    ENDSSH
          
  20. extract and print the 35th and 36th byte from a set of files
  21. for f in $(find . -iname \*.edds); do printf "%s ==> "  $f && od -j 34 -N 2 -A x -t x1z -v $f | cut -c7-30 | head -1 ; done
  22. bash join lines
  23. tr "\\n" " "
          

    source.
    E.g. to prepare a line for emacs to open all Java files containing "fromValue", use:
    findinjava fromValue | sort | awk '{print $1}' | rev | cut -c 2- | rev | tr "\\n" " "
          
  24. remove the last n characters from a bash line (that makes no use of awk or sed
  25. echo foobar | rev | cut -c (n+1)- | rev
          

    source
  26. named pipes
  27. Pipes are a special type of file. You interface with these as if they were files (i.e. with Unix redirection, not pipes), but they behave like pipes:
    $ mkfifo npipe
    $ ls npipe 
    prw-r--r-- 1 mperdikeas users 0 Mar 11 14:12 npipe|
    $ echo "boo" > npipe  &
    [1] 22271
    $ cat < npipe 
    boo
    [1]+  Done                    echo "boo" > npipe
          

    More can be found here.
  28. replace a string in a fileset
  29. The below incantation replaces all instances of XMLUtils.getString with XMLUtils.getStringNew in all Java files in the current directory (recursively):
    find . -type f -iname \*.java -print0 | xargs -0 sed -i 's/XMLUtils.getString/XMLUtils.getStringNew/g'
          
  30. change (recursively) all symlinks in a directory that point to a certain target Apoint to another target B
  31. I used the following script to replace all symlinks to a file `ant-contrib-1.0b3.jar` to another file `ant-contrib.jar` (that happened to reside in the same directory as the original target - I am not sure whether that was essential to the solution below). The script was called from a location where the path repo-wide-libs/ant-contrib-1.0b3.jar existed and was valid.
    #!/usr/bin/env bash
    for file in $(find . -type l); do
        if [[ -h "$file" ]]; then
            echo "file $file is a symlink"
            if [[ "$(readlink -f "$file")" == "$(realpath repo-wide-libs/ant-contrib-1.0b3.jar)" ]]; then
                ln -sf "$(dirname "$(readlink "$file")")"/ant-contrib.jar "$(dirname "$file")" && rm -f "$file"
            fi
        fi
    done
          
    source
  32. replace a string in a set of files
  33. find . -type f -exec sed -i 's/foo/bar/' {} +
    source To correctly escape the string passed to `sed` use my `sedescape` function (lives in .bashrc).
  34. attach multiple times to a single screen session
  35. screen -x
  36. the ultimate screen re-attach command
  37. source
    screen -dRR
          


    The above attaches to a screen session. If the session is attached elsewhere, detaches that other display. If no session exists, creates one. If multiple sessions exist, uses the first one.)
  38. total number of lines in files found
  39. source
    find . -iname *.java -print0 | wc -l --files0-from=-
          
  40. find files and execute command in local directory
  41. use the -execdir switch. E.g.:
    find . -type d -execdir touch {}/foo \;
    NB: On why {}/foo is needed, see: this SO post.
  42. total size of directory
  43. du -ch .
  44. how to scp directory with spaces in its filename
  45. scp -rp 'mperdikeas@192.168.2.2:~/"mperd-digital camera images"' /home/mperdikeas/
  46. binary diffing on Ubuntu
  47. Use my ~/tools/bincomp tool or the vbindiff or dhex programs.
  48. how to make a replacement on every line with awk
  49. awk '{gsub("/","."); print}'
  50. properly quoting arguments to underlying invocations
  51. Inside a bash script, always use "$1" to pass an argument to an underlying program instead of just $1. See: ~/playground/java/why-I-should-always-quote-arguments for more.
  52. how to use diff to generate a patch in a Makefile
  53. this is really a bash topic (see below). Relative SO discussion here.
    The Makefile run will fail when a command that's executed as part of a recipe returns a non-zero exit code, unless the command is preceded with "-" (in which case it ignores the exit code).
    This is exactly what's happenning in the ~/playground/patch/Makefile Makefile, also shown below:
    patch:
    	-diff -u b a > b2a.patch
    	patch -o b.corrected b < b2a.patch
    	diff a b.corrected
    
    clean:
    	rm -f b2a.patch b.corrected
            
    However, this idiom will ask mask real problems (such as using -dif instead of -diff. Therefore the correct way to solve this conundrum (i.e. to not don't ignore the exit value, but also to not fail if it's non-zero) is shown in /home/mperdikeas/playground/ant-patch/02/Makefile file, also shown below:
    patch: FORCE
    	diff -u src/a/random/path/b b.corrected > b2a.patch; [ $$? -eq 1 ]
    	ant
    	diff src/a/random/path/a src/a/random/path/b.corrected
    
    clean:
    	rm -f b2a.patch src/a/random/path/b.corrected
    
    FORCE:
            
    The ability to use the following syntax: ; [ $? -eq 1 ] is plain bash (the double '$' in the Makefile is to account for Make's escaping conventions), as is shown in the following bash examples:
    $ echo $[3+2]
    5
    $ [ $[3+2] -eq 5 ]
    $ echo $?
    0
    $ [ $[3+2] -eq 6 ]
    $ echo $?
    1
            
  54. using named pipes in Bash
  55. original article
    diff <(ls) <(ls)
    Named pipes can be created with mkfifo:
    mkfifo foo
    find ~ > foo &
    tail -f foo
            
    The difference of the above to creating a file is that the size of a named pipe is 0. A named pipe lives on the filesystem and survives a reboot but never holds any data, only metadata.
  56. how to see the alias of a command in the shell
  57. $ type emacs
    emacs is aliased to `emacs -nw'
            
    (or the more low tech: alias | grep emacs)
  58. how to recursively diff two directories and exclude certain files (e.g. .svn files in case of a subversion repository)
  59. diff -r --exclude=.svn dir-a dir-b
  60. count the number of "new" keywords in a set of files (per file - not total):
  61. grep -c new `find . -name '*.java'`
  62. count total number of lines in all Java files
  63. wc -l `find . -name '*.java'`

    the following also works (and strangely gives a different result, and requires a CTRL+C at the end):
    find . -name '*.java' | xargs wc -l
  64. record shell commands and their input/output

  65. to record:
    $script session.log

    to exit a recording session: exit
  66. seq program
  67. for i in `seq 1 10`; do echo "number is: " $i ; done
  68. redirect standard error to standard out and both to file
  69. gksudo software-center > trace 2>&1
    2>&1 means "put the output of channel 2 (stderr) to where channel 1 (stdout) currently goes" which is why the redirection to file has to occur before the stderr redirection to stdout.
  70. add a line at the beginning of every text file in a group
  71. Use sed with for iteration on find results.
    E.g. the following adds package headers in a group of Java files:
    for f in $(find src/ -type f -iname \*.java) ; do sed -i '1ipackage _int.esa.esavo.harvesting;' $f ; done
            
  72. find files and sort them by modification time
  73. find -type f -print0 | xargs -r0 stat -c %y\ %n | sort 
    source
  74. recursively find the latest modified file in a directory
  75. Original article in stackoverflow.
    find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "
            
  76. ask diff to ignore lines starting with Java single-line comments
  77. diff --ignore-matching-lines='^//' Foo.java
  78. How to obtain the sha256 of a password
  79. Say the password is "foo":
    echo -n foo | sha256sum
            
    The -n is necessary as otherwise echo will also sent the new line character to be SHA256-encoded as part of the password. In cases of salting the usually arrangement is a simple concatenation of the salt followed by the password (in that order so the cracker doesn't have an attack vector). So if password is "foo" and salt is "salt":
    echo -n saltfoo | sha256sum
            
  80. How to copy a file and connect to a remote server to do stuff with it
  81. #!/bin/bash
    scp VatRefund.ear root@node-testapp1:/var/tmp/ || exit 1
    cat <<OEF | ssh node-testapp1 || exit 1
    cd /opt/jboss/jboss-eap-6.0/bin
    ./jboss-cli.sh  --connect controller=123.30.228.54:9999
    undeploy --server-groups=other-server-group VatRefund.ear
    deploy --server-groups=other-server-group /var/tmp/VatRefund.ear
    OEF
    exit 0
            
  82. Programmer's grep
  83. Use the ack utility. It is by default only operating on a white list of files. With the -u option it operates on all files except those on the black list (.svn etc).
  84. find files and get the 10 most recently modified
  85. find . -iname *.script -type f -printf '%T@ %p\n' | sort -k 1nr | sed 's/^[^ ]* //' | head -n 10
            
  86. to send an email directly using the SMTP protocol:
  87. nc mail.semantix.gr 25
    This method ensures that nothing's stored anywhere.
  88. to easily navigate folder hierarchies in Bash:
  89. sudo apt-get install autojump
  90. to record some task in Bash
  91. Use script followed by CTRL+D to stop recording (the output is written in file 'typescript')
  92. how to ensure that no # appears in ini files except as the first character
  93. this cost me several hours with a shiro.ini cookie path I had set where the comment (or so I thought) was interpreted as part of the path
    find . -iname \*.ini -exec grep '^[^#].*#' /dev/null '{}' ';' 
            
  94. chain logical expressions in bash
  95. find . -iname \*.csv -exec sh -c "grep foo {} && echo {}" \;
  96. find files and print their absolute path name
  97. find  $(pwd) -iname report2.jrxml
  98. Thing to notice when trying to find the types of files
  99. The below code might not working depending on the environment settings, e.g. if the grep command outputs special characters (like bold):
    find  . -iname \*.java | grep facade | while read f ; do file $f ; done
            
    In such, cases do a:
    find  . -iname \*.java | /bin/grep facade | while read f ; do file $f ; done
            
  100. 'materialize' soft-links - easy way
  101. cp -pr -L --copy-contents
  102. 'materialize' soft-links - trying to be clever way
  103. We do it in two steps:
    1. create the directories
    2. cd srcPath && find . -type d -exec mkdir -p /destPath/'{}' ';'
    3. copy the contents - use the cattrickery to always get the data
    4. cd srcPath && find . ! -type d -exec sh -c 'cat "{}" > /destPath/"{}"' \;
  104. trace the system calls by an executable
  105. strace -f ./NeuroCode.exe Project.xml  2>&1 | less
    Once in the output of less, you can use a / to search for NeuroCode.exe and find the system calls it makes.
  106. more elegant 'ps aux | grep terminal | grep -v grep'
  107. The following incantation:
    ps aux | grep '[t]erminal'
    will return all lines containing terminal which the line you invoked does not.
  108. Generate import statements out of the classes of a jar
  109. jar tvf Orders-ejb.jar  | grep entities| cut -c38- | grep -v /$ | sed 's,^,import ,' | sed 's,/,.,g'
            
    (in the above example at the sed command the comma character is used as the separator - this is handier than the usual slash character when one has to replaces slashes)
  110. Total number of lines in all found files
  111. find . -iname \*.java -print0 | wc -l --files0-from -
  112. Find all different file types in a folder hierarchy
  113. find . -exec file {} \; | awk -F/ '{print $NF}' | awk -F: '{print $NF}' | sort | uniq
            
  114. Find all XML files with a specific string inside ("login-module") and print the name of the file concerned
  115. find . -iname \*.xml -exec grep "login-module" '{}' /dev/null \;
            
    The following simpler version also seems to work:
    find . -iname *.xml -exec grep "login-module" {} /dev/null \;
            
  116. Return just the names of the jar files in a find
  117. The cut utility cannot be used to return only the last field, but awk can do the trick:
    find jboss-as-7.1.1.Final/ -iname *.jar | awk -F/ '{print $NF}'
            
  118. Search for a class in a set of jar files
  119. To search all jars in the current folder and below for class named a.b.c.D do a:
    find . -iname *.jar | while read JARF; do jar tvf $JARF | grep a/b/c/D.class && echo $JARF ; done
            
    if there is too much output, try:
    find /home/mperdikeas/git-jboss/ -iname *.jar | while read f ; do echo $f && (jar tvf $f | grep a.b.c.D) ; done | grep -B1 ^\
            
    ("." instead of "/" works practically as well as it stands for any character).
  120. How to count the number of lines with exactly five spaces
  121. five spaces means 6 fields in AWK, so (for file 3.ply):
    awk 'NF==6;{}' 3.ply | wc -l
            
  122. How to create files of arbitrary size
  123. mperdikeas@mperdikeas ~
    $ dd if=/dev/zero of=a_file_of_3MBs bs=1M count=3
    3+0 records in
    3+0 records out
    3145728 bytes (3.1 MB) copied, 0.00516917 s, 609 MB/s
    
    mperdikeas@mperdikeas ~
    $ dd if=/dev/zero of=a_file_of_1453bytes bs=1 count=1453
    1453+0 records in
    1453+0 records out
    1453 bytes (1.5 kB) copied, 0.0184214 s, 78.9 kB/s
            
  124. How to copy trees
  125. How to copy a whole tree to another location in the same network. example copy folder "wa-integration" and its tree to the /c/BW location.
    tar cf - wa-integration/ | (cd /c/BW/ ; tar xvf - )
            
  126. How to count files
  127. How to recursively count all "normal" files in a directory
    find directory_name -type f | wc -l
            
  128. How to grep in a large number of files
  129. for i in `find ./sample/`; do grep -H "sample string" $i; done
            
  130. How to use curl to test GET, POST, etc.
  131. GET:
    curl -HAccept:text/plain http://example.com/base
    PUT:
    curl -XPUT -HContent-type:text/plain --data "stuff:morestuff" http://example.com/base?param=val
    DELETE:
    curl -XDELETE http://example.com/base/user/123
    POST:
    curl -d "param1=value1¶m2=value2" http://example.com/base/

    Another curl tip: To send a POST request whose data is not in application/x-www-urlencoded format, use the --data-binary option instead of --data.

  132. How to use cut to only retain find's output from the 73rd column onwards
  133. find folderToSearch/ | cut -c 73-
  134. Magic
  135. $ cat nice.sh
    tr -cs A-Za-z '\n' |
    	tr A-Z a-z |
    	sort |
    	uniq -c |
    	sort -rn |
    	sed ${1}q
              
    What it does:

    Given the above script the following text:

    "Tests a distributed system made up of two LEON boards connected with the spacewire, and each one of them having a local FPGA component. The code is clear; the bug reports on Mantis"

    becomes (when piped with: echo "(the above text)" | nice.sh 5) :

          3 the
          2 of
          2 a
          1 with
          1 up
                  
  136. How not to lose tty context
  137. A utility that allows you to save all tty context and re-connect and recover everything in case of a connection lost (e.g. ssh) is screen. Invoked with "screen". Reconnect with "screen -r". To detach from the screen type "ctrl-a d". The screen deamon is visible with "ps auex".
  138. The scp utility to download / upload files from remote server (no FTP)
  139. To download recursively a whole folder from remote directory:
    scp -r mperdikeas@elrond:~/TAPKIT/Conversions/Genesis/sources-auto /cygdrive/c/fromElrond
    To upload single file to remote directory:
    scp buildSpecification.py mperdikeas@elrond:~/TAPKIT/Conversions/Genesis/python
    scp can be setup to not prompt for password using ssh-keygen and appending the public key to the remote host(s)'s ~/.ssh/authorized_keys file (for the respective user).

  140. diff alternative in Linux
  141. Use the vimdiff utility (navigation to next / previous difference with [c and ]c
  142. How to browse the web from a different IP address:
  143. One can use the SOCKS protocol. The idea would be to open an SSH tunnel to a remote machine and then configure the browser to use the local machine as SOCKS host. The browser would then encapsulate all HTTP traffic into a SOCKS packet and send it through the tunnel to the remote SSH server which will also decapsulate the SOCKS protocol. So, the resolution of the domain name at the HTTP GET will be done at the remote server. Instructions for Firefox:
    1. open cygwin shell in local machine and at the prompt enter: ssh -D 9987 root@tapeditor.com
    2. it will typically output something about an RSA fingerprint at which point you type yes
    3. you are prompted for password (enter it)
    4. you are now in the remote host and the tunnel is operational.
    5. fire Mozilla Firefox
    6. find the Firefox's proxy config page (tools->options->Network->Settings) and setup a Manual proxy configuration with SOCKS Host the local machine (enter "localhost", port 9987 (the one used in step [1] - it can be any port) and SOCKS version v5.
    7. if you now visit www.whatismyipaddress.com you see the (usually public static IP address) of the other end of the tunnel.
    8. you can now browse the web using that public IP address (so long as the tunnel opened in step [4] remains open).

  144. emacs shortcuts in bash
  145. Standard emacs shortcuts such as C-l, C-p and C-r come very handy in bash. C-l clears the screen whereas C-r allows easy repetition of previous commands (much better than history or C-p)

  146. how to time a job in bash and redirect all output (incl. error) to /dev/null
  147. date ; time find / > /dev/null 2>&1; date
                                                
  148. completely purge a package
  149. dpkg -P mysql-server
  150. to view processes
  151. ps auxf
    ('f' stands for 'full')
  152. print physical directory (ignoring symlinks)
  153. pwd -P