1. when which does not work

    2. The which command executable cannot find shell alies and shell builtins (such as cd). To see those, use: 

    type -a some-alias-or-shell-bultin

  2. how to find a user's login shell

    3. Most of the time this is stored in the /etc/passwd file so the following would work: 

    grep username /etc/passwd

    However, when some kind of directory service is used the information may not reside there. So, the following always works: 

    getent passwd username

  3. on the various forms of boolean tests in Bash using (), [] and [[]]

    4. This is explained wonderfully in this Unix stackexchange post.

    TL;DR:

    if command
    execute command and proceed based on exit code
    if (command)
    same as above but execute command in subshell
    if ((arithmetic condition))
    this is no longer a command but an arithmetic condition (NB: non-zero result is interpreted as true)
    if [ condition ]
    some condition like -e
    if [[ condition ]]
    same as above but with some additional goodies

    Below are some examples to build up intution: 

    $ ((1+1))
$ echo $?
0
$ ((1-1))
$ echo $?
1
$ if ls > /dev/null ; then echo 'a'; else echo 'b'; fi
a
    Some other examples: 
    $ touch foo ; if [ -e foo ] ; then echo 'a'; else echo 'b' ; fi
a
$ rm -f foo ; if [ -e foo ] ; then echo 'a'; else echo 'b' ; fi
b
$ if ((1+2>2)); then echo 'a'; else echo 'b'; fi
a
    The below prints a because the assignment succeeds and so the condition for the if statement is evaluated to true: 
    $ a=0; if [ a=1 ]; then echo 'a'; else echo 'b'; fi
a
    … the correct way is to use arithmetic evaluation: 
    $ a=0; if (( $a==1 )); then echo 'a'; else echo 'b'; fi
b
    The below prints nothing as the assignment was done in a subshell: 
    $ ( b=1 ); echo $b

  4. make set print only the shell variables, not the functions
    5. (source) 
    (set -o posix; set)
  5. env vs. printenv vs. set

    6. env is POSIX, printenv isn't. Other than that, in my system env and printenv yield identical output.

    set however is a different beast and provides more comprehensive output as it includes both shell-local variables (since it's a shell builtin it can see them) and environment variables which are passed on to every executed program. env however is a program and so can only see environment variables, not shell-local variables (source)

    BTW, the fact that set is a built-in can be established by the fact that which set produces no output.

  6. backticks (`) vs. $()
    7. The grave (`) was the original syntax and should not be used any more unless you want your scripts to run in old Bourne shells. Otherwise simply use:
  7. how to execute a command when one ore more files change
    8. Sources:
    There's two ways:
  8. how to write a script that executes sudo with the password
    1. Create a file:
      2. emacs ~/.easy.install
    2. inside the file invoke sudo with -S:
      3. echo <password> | sudo -S do-your-shit
    3. make the file executable
      4. sudo chmod +x ~/.easy.install
    4. Change the permissions of the file so only root can read and edit it:
      5. sudo chmod 700 ~/.easy.install
  9. how to process the output of find when there are spaces in the filenames
    10. source
    Replace the usual for iteration with a while iteration that is line-based as opposed to word-based: 
    find . -iname Thundercats\ -\ 10\*  | while read f; do  sudo cp "$f" /media/mperdikeas/thundercats-season-1/; done
  10. how to put an scp that requires a password in a script file
    11. Use sshpass as shown below: 
    sshpass -f $PASS_FILE scp rawdar.zip user@$DEST:~/rawdar.war
    If, after the copy, you also need to ssh to the destination and you further need sudo privilleges, sshpass can't supply both passwords, so you need to put the action that requires sudo in a script and execute it. E.g. as shown below: 
    sshpass -f $PASS_FILE ssh user@$DEST <<'ENDSSH'
./publish-rawdar.sh
ENDSSH
    ... where the script at the ssh destination is: 
    user@user-virtual-machine:~$ cat publish-rawdar.sh
#!/usr/bin/env bash
readonly ECHO=/bin/echo
$ECHO  'top-secret-password' | sudo -S cp rawdar.war /var/lib/tomcat7/webapps
user@user-virtual-machine:~$
    Relevant excrepts from functional file: 
    #!/usr/bin/env bash
readonly PASS_FILE=~/esac.neuropublic.pwd
readonly DEST=172.31.148.22
rm -f rawdar.zip && zip -r rawdar.zip rawdar.html rawdar.org.files/
sshpass -f $PASS_FILE scp rawdar.zip user@$DEST:~/rawdar.war
sshpass -f $PASS_FILE ssh user@$DEST <<'ENDSSH'
./publish-rawdar.sh
ENDSSH
  11. extract and print the 35th and 36th byte from a set of files
    12. for f in $(find . -iname \*.edds); do printf "%s ==> "  $f && od -j 34 -N 2 -A x -t x1z -v $f | cut -c7-30 | head -1 ; done
  12. bash join lines
    13. tr "\\n" " "

    source.
    E.g. to prepare a line for emacs to open all Java files containing "fromValue", use: 
    findinjava fromValue | sort | awk '{print $1}' | rev | cut -c 2- | rev | tr "\\n" " "
  13. remove the last n characters from a bash line (that makes no use of awk or sed
    14. echo foobar | rev | cut -c (n+1)- | rev

    source
  14. named pipes
    15. Pipes are a special type of file. You interface with these as if they were files (i.e. with Unix redirection, not pipes), but they behave like pipes: 
    $ mkfifo npipe
$ ls npipe 
prw-r--r-- 1 mperdikeas users 0 Mar 11 14:12 npipe|
$ echo "boo" > npipe  &
[1] 22271
$ cat < npipe 
boo
[1]+  Done                    echo "boo" > npipe

    More can be found here.
  15. replace a string in a fileset
    16. The below incantation replaces all instances of XMLUtils.getString with XMLUtils.getStringNew in all Java files in the current directory (recursively): 
    find . -type f -iname \*.java -print0 | xargs -0 sed -i 's/XMLUtils.getString/XMLUtils.getStringNew/g'
  16. change (recursively) all symlinks in a directory that point to a certain target Apoint to another target B
    17. I used the following script to replace all symlinks to a file `ant-contrib-1.0b3.jar` to another file `ant-contrib.jar` (that happened to reside in the same directory as the original target - I am not sure whether that was essential to the solution below). The script was called from a location where the path repo-wide-libs/ant-contrib-1.0b3.jar existed and was valid. 
    #!/usr/bin/env bash
for file in $(find . -type l); do
    if [[ -h "$file" ]]; then
        echo "file $file is a symlink"
        if [[ "$(readlink -f "$file")" == "$(realpath repo-wide-libs/ant-contrib-1.0b3.jar)" ]]; then
            ln -sf "$(dirname "$(readlink "$file")")"/ant-contrib.jar "$(dirname "$file")" && rm -f "$file"
        fi
    fi
done
    source
  17. replace a string in a set of files
    18. find . -type f -exec sed -i 's/foo/bar/' {} +
    source To correctly escape the string passed to `sed` use my `sedescape` function (lives in .bashrc).
  18. attach multiple times to a single screen session
    19. screen -x
  19. the ultimate screen re-attach command
    20. source 
    screen -dRR


    The above attaches to a screen session. If the session is attached elsewhere, detaches that other display. If no session exists, creates one. If multiple sessions exist, uses the first one.)
  20. total number of lines in files found
    21. source 
    find . -iname *.java -print0 | wc -l --files0-from=-
  21. find files and execute command in local directory
    22. use the -execdir switch. E.g.: 
    find . -type d -execdir touch {}/foo \;
    NB: On why {}/foo is needed, see: this SO post.
  22. total size of directory
    23. du -ch .
  23. how to scp directory with spaces in its filename
    24. scp -rp 'mperdikeas@192.168.2.2:~/"mperd-digital camera images"' /home/mperdikeas/
  24. binary diffing on Ubuntu
    25. Use my ~/tools/bincomp tool or the vbindiff or dhex programs.
  25. how to make a replacement on every line with awk
    26. awk '{gsub("/","."); print}'
  26. properly quoting arguments to underlying invocations
    27. Inside a bash script, always use "$1" to pass an argument to an underlying program instead of just $1. See: ~/playground/java/why-I-should-always-quote-arguments for more.
  27. how to use diff to generate a patch in a Makefile
    28. this is really a bash topic (see below). Relative SO discussion here.
    The Makefile run will fail when a command that's executed as part of a recipe returns a non-zero exit code, unless the command is preceded with "-" (in which case it ignores the exit code).
    This is exactly what's happenning in the ~/playground/patch/Makefile Makefile, also shown below: 
    patch:
	-diff -u b a > b2a.patch
	patch -o b.corrected b < b2a.patch
	diff a b.corrected

clean:
	rm -f b2a.patch b.corrected
    However, this idiom will ask mask real problems (such as using -dif instead of -diff. Therefore the correct way to solve this conundrum (i.e. to not don't ignore the exit value, but also to not fail if it's non-zero) is shown in /home/mperdikeas/playground/ant-patch/02/Makefile file, also shown below: 
    patch: FORCE
	diff -u src/a/random/path/b b.corrected > b2a.patch; [ $$? -eq 1 ]
	ant
	diff src/a/random/path/a src/a/random/path/b.corrected

clean:
	rm -f b2a.patch src/a/random/path/b.corrected

FORCE:
    The ability to use the following syntax: ; [ $? -eq 1 ] is plain bash (the double '$' in the Makefile is to account for Make's escaping conventions), as is shown in the following bash examples: 
    $ echo $[3+2]
5
$ [ $[3+2] -eq 5 ]
$ echo $?
0
$ [ $[3+2] -eq 6 ]
$ echo $?
1
  28. using named pipes in Bash
    29. original article 
    diff <(ls) <(ls)
    Named pipes can be created with mkfifo: 
    mkfifo foo
find ~ > foo &
tail -f foo
    The difference of the above to creating a file is that the size of a named pipe is 0. A named pipe lives on the filesystem and survives a reboot but never holds any data, only metadata.
  29. how to see the alias of a command in the shell
    30. $ type emacs
emacs is aliased to `emacs -nw'
    (or the more low tech: alias | grep emacs)
  30. how to recursively diff two directories and exclude certain files (e.g. .svn files in case of a subversion repository)
    31. diff -r --exclude=.svn dir-a dir-b
  31. count the number of "new" keywords in a set of files (per file - not total):
    32. grep -c new `find . -name '*.java'`
  32. count total number of lines in all Java files
    33. wc -l `find . -name '*.java'`

    the following also works (and strangely gives a different result, and requires a CTRL+C at the end): 
    find . -name '*.java' | xargs wc -l
  33. record shell commands and their input/output

    34. to record: 
    $script session.log

    to exit a recording session: exit
  34. seq program
    35. for i in `seq 1 10`; do echo "number is: " $i ; done
  35. redirect standard error to standard out and both to file
    36. gksudo software-center > trace 2>&1
    2>&1 means "put the output of channel 2 (stderr) to where channel 1 (stdout) currently goes" which is why the redirection to file has to occur before the stderr redirection to stdout.
  36. add a line at the beginning of every text file in a group
    37. Use sed with for iteration on find results.
    E.g. the following adds package headers in a group of Java files: 
    for f in $(find src/ -type f -iname \*.java) ; do sed -i '1ipackage _int.esa.esavo.harvesting;' $f ; done
  37. find files and sort them by modification time
    38. find -type f -print0 | xargs -r0 stat -c %y\ %n | sort
    source
  38. recursively find the latest modified file in a directory
    39. Original article in stackoverflow. 
    find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "
  39. ask diff to ignore lines starting with Java single-line comments
    40. diff --ignore-matching-lines='^//' Foo.java
  40. How to obtain the sha256 of a password
    41. Say the password is "foo": 
    echo -n foo | sha256sum
    The -n is necessary as otherwise echo will also sent the new line character to be SHA256-encoded as part of the password. In cases of salting the usually arrangement is a simple concatenation of the salt followed by the password (in that order so the cracker doesn't have an attack vector). So if password is "foo" and salt is "salt": 
    echo -n saltfoo | sha256sum
  41. How to copy a file and connect to a remote server to do stuff with it
    42. #!/bin/bash
scp VatRefund.ear root@node-testapp1:/var/tmp/ || exit 1
cat <<OEF | ssh node-testapp1 || exit 1
cd /opt/jboss/jboss-eap-6.0/bin
./jboss-cli.sh  --connect controller=123.30.228.54:9999
undeploy --server-groups=other-server-group VatRefund.ear
deploy --server-groups=other-server-group /var/tmp/VatRefund.ear
OEF
exit 0
  42. Programmer's grep
    43. Use the ack utility. It is by default only operating on a white list of files. With the -u option it operates on all files except those on the black list (.svn etc).
  43. find files and get the 10 most recently modified
    44. find . -iname *.script -type f -printf '%T@ %p\n' | sort -k 1nr | sed 's/^[^ ]* //' | head -n 10
  44. to send an email directly using the SMTP protocol:
    45. nc mail.semantix.gr 25
    This method ensures that nothing's stored anywhere.
  45. to easily navigate folder hierarchies in Bash:
    46. sudo apt-get install autojump
  46. to record some task in Bash
    47. Use script followed by CTRL+D to stop recording (the output is written in file 'typescript')
  47. how to ensure that no # appears in ini files except as the first character
    48. this cost me several hours with a shiro.ini cookie path I had set where the comment (or so I thought) was interpreted as part of the path 
    find . -iname \*.ini -exec grep '^[^#].*#' /dev/null '{}' ';'
  48. chain logical expressions in bash
    49. find . -iname \*.csv -exec sh -c "grep foo {} && echo {}" \;
  49. find files and print their absolute path name
    50. find  $(pwd) -iname report2.jrxml
  50. Thing to notice when trying to find the types of files
    51. The below code might not working depending on the environment settings, e.g. if the grep command outputs special characters (like bold): 
    find  . -iname \*.java | grep facade | while read f ; do file $f ; done
    In such, cases do a: 
    find  . -iname \*.java | /bin/grep facade | while read f ; do file $f ; done
  51. 'materialize' soft-links - easy way
    52. cp -pr -L --copy-contents
  52. 'materialize' soft-links - trying to be clever way
    53. We do it in two steps:
    1. create the directories
      2. cd srcPath && find . -type d -exec mkdir -p /destPath/'{}' ';'
    2. copy the contents - use the cattrickery to always get the data
      3. cd srcPath && find . ! -type d -exec sh -c 'cat "{}" > /destPath/"{}"' \;
  53. trace the system calls by an executable
    54. strace -f ./NeuroCode.exe Project.xml  2>&1 | less
    Once in the output of less, you can use a / to search for NeuroCode.exe and find the system calls it makes.
  54. more elegant 'ps aux | grep terminal | grep -v grep'
    55. The following incantation: 
    ps aux | grep '[t]erminal'
    will return all lines containing terminal which the line you invoked does not.
  55. Generate import statements out of the classes of a jar
    56. jar tvf Orders-ejb.jar  | grep entities| cut -c38- | grep -v /$ | sed 's,^,import ,' | sed 's,/,.,g'
    (in the above example at the sed command the comma character is used as the separator - this is handier than the usual slash character when one has to replaces slashes)
  56. Total number of lines in all found files
    57. find . -iname \*.java -print0 | wc -l --files0-from -
  57. Find all different file types in a folder hierarchy
    58. find . -exec file {} \; | awk -F/ '{print $NF}' | awk -F: '{print $NF}' | sort | uniq
  58. Find all XML files with a specific string inside ("login-module") and print the name of the file concerned
    59. find . -iname \*.xml -exec grep "login-module" '{}' /dev/null \;
    The following simpler version also seems to work: 
    find . -iname *.xml -exec grep "login-module" {} /dev/null \;
  59. Return just the names of the jar files in a find
    60. The cut utility cannot be used to return only the last field, but awk can do the trick: 
    find jboss-as-7.1.1.Final/ -iname *.jar | awk -F/ '{print $NF}'
  60. Search for a class in a set of jar files
    61. To search all jars in the current folder and below for class named a.b.c.D do a: 
    find . -iname *.jar | while read JARF; do jar tvf $JARF | grep a/b/c/D.class && echo $JARF ; done
    if there is too much output, try: 
    find /home/mperdikeas/git-jboss/ -iname *.jar | while read f ; do echo $f && (jar tvf $f | grep a.b.c.D) ; done | grep -B1 ^\
    ("." instead of "/" works practically as well as it stands for any character).
  61. How to count the number of lines with exactly five spaces
    62. five spaces means 6 fields in AWK, so (for file 3.ply): 
    awk 'NF==6;{}' 3.ply | wc -l
  62. How to create files of arbitrary size
    63. mperdikeas@mperdikeas ~
$ dd if=/dev/zero of=a_file_of_3MBs bs=1M count=3
3+0 records in
3+0 records out
3145728 bytes (3.1 MB) copied, 0.00516917 s, 609 MB/s

mperdikeas@mperdikeas ~
$ dd if=/dev/zero of=a_file_of_1453bytes bs=1 count=1453
1453+0 records in
1453+0 records out
1453 bytes (1.5 kB) copied, 0.0184214 s, 78.9 kB/s
  63. How to copy trees
    64. How to copy a whole tree to another location in the same network. example copy folder "wa-integration" and its tree to the /c/BW location. 
    tar cf - wa-integration/ | (cd /c/BW/ ; tar xvf - )
  64. How to count files
    65. How to recursively count all "normal" files in a directory 
    find directory_name -type f | wc -l
  65. How to grep in a large number of files
    66. for i in `find ./sample/`; do grep -H "sample string" $i; done
  66. How to use curl to test GET, POST, etc.
    67. GET: 
    curl -HAccept:text/plain http://example.com/base
    PUT: 
    curl -XPUT -HContent-type:text/plain --data "stuff:morestuff" http://example.com/base?param=val
    DELETE: 
    curl -XDELETE http://example.com/base/user/123
    POST: 
    curl -d "param1=value1¶m2=value2" http://example.com/base/

    Another curl tip: To send a POST request whose data is not in application/x-www-urlencoded format, use the --data-binary option instead of --data.

  67. How to use cut to only retain find's output from the 73rd column onwards
    68. find folderToSearch/ | cut -c 73-
  68. Magic
    69. $ cat nice.sh
tr -cs A-Za-z '\n' |
	tr A-Z a-z |
	sort |
	uniq -c |
	sort -rn |
	sed ${1}q
    What it does:

    Given the above script the following text:

    "Tests a distributed system made up of two LEON boards connected with the spacewire, and each one of them having a local FPGA component. The code is clear; the bug reports on Mantis"

    becomes (when piped with: echo "(the above text)" | nice.sh 5) : 

          3 the
      2 of
      2 a
      1 with
      1 up
  69. How not to lose tty context
    70. A utility that allows you to save all tty context and re-connect and recover everything in case of a connection lost (e.g. ssh) is screen. Invoked with "screen". Reconnect with "screen -r". To detach from the screen type "ctrl-a d". The screen deamon is visible with "ps auex".
  70. The scp utility to download / upload files from remote server (no FTP)
    71. To download recursively a whole folder from remote directory: 
    scp -r mperdikeas@elrond:~/TAPKIT/Conversions/Genesis/sources-auto /cygdrive/c/fromElrond
    To upload single file to remote directory: 
    scp buildSpecification.py mperdikeas@elrond:~/TAPKIT/Conversions/Genesis/python
    scp can be setup to not prompt for password using ssh-keygen and appending the public key to the remote host(s)'s ~/.ssh/authorized_keys file (for the respective user).

  71. diff alternative in Linux
    72. Use the vimdiff utility (navigation to next / previous difference with [c and ]c
  72. How to browse the web from a different IP address:
    73. One can use the SOCKS protocol. The idea would be to open an SSH tunnel to a remote machine and then configure the browser to use the local machine as SOCKS host. The browser would then encapsulate all HTTP traffic into a SOCKS packet and send it through the tunnel to the remote SSH server which will also decapsulate the SOCKS protocol. So, the resolution of the domain name at the HTTP GET will be done at the remote server. Instructions for Firefox:
    1. open cygwin shell in local machine and at the prompt enter: ssh -D 9987 root@tapeditor.com
    2. it will typically output something about an RSA fingerprint at which point you type yes
    3. you are prompted for password (enter it)
    4. you are now in the remote host and the tunnel is operational.
    5. fire Mozilla Firefox
    6. find the Firefox's proxy config page (tools->options->Network->Settings) and setup a Manual proxy configuration with SOCKS Host the local machine (enter "localhost", port 9987 (the one used in step [1] - it can be any port) and SOCKS version v5.
    7. if you now visit www.whatismyipaddress.com you see the (usually public static IP address) of the other end of the tunnel.
    8. you can now browse the web using that public IP address (so long as the tunnel opened in step [4] remains open).

  73. emacs shortcuts in bash
    74. Standard emacs shortcuts such as C-l, C-p and C-r come very handy in bash. C-l clears the screen whereas C-r allows easy repetition of previous commands (much better than history or C-p)

  74. how to time a job in bash and redirect all output (incl. error) to /dev/null
    75. date ; time find / > /dev/null 2>&1; date
  75. completely purge a package
    76. dpkg -P mysql-server
  76. to view processes
    77. ps auxf
    ('f' stands for 'full')
  77. print physical directory (ignoring symlinks)
    78. pwd -P