Bash scripting

Multi-line comments

: '
This is a
very neat comment
in bash
'

Output common lines between two files

grep -Fxf file1 file2 # using grep
sort file1 file2 | uniq -d # using sort

Replace white space with underscore in filename

for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done

Processing a large number of files

To avoid the error Argument list too long:

ls | xargs -n 32 -P 8 cat >> ../saved_output

The xargs utility will take the output of ls, break it into chunks of 32 (32 arguments) and spawn up to 8 concurrent cat processes. cat will be respawned as long as their is data coming from ls.

Randomly select N files

ls | shuf -n 50

Replace string in file

OSX

sed -i '' 's/resources\/input\//\/Users\/othrif\/projects\/aoede\/resources\/input\/versioned\/v1\//g' *txt

Linux

sed 's/resources\/input\//\/Users\/othrif\/projects\/aoede\/resources\/input\/versioned\/v1\//g' *txt

Diff between two commands

diff <(ls file1) <(ls file2)

Find size of all files including hidden files

du -shc .[^.]* *