BASH Tricks

How many lines in my text file?

Totally useful when you have some kind of training file with many rows and columns:

FILENAME=myfile.csv
nl ${FILENAME} | awk '{ print $1 }'

Make a list of directory names

We often organize subject data so that each subject gets their own directory. Freesurfer uses a subjects file when batch processing. Rather than manually type out each folder name into a text file, it can be generated in one line of code:

ls -1 -d */ | sed "s,/$,," >  subjects

This lists in 1 column all the directories (-1 -d) and uses sed to snip off the trailing forward slashes in the directory names

Make a series of numbered directories

FreeSurfer BOLD data goes in a series of directories, numbered 001, 002, ... , 0nn. A one-liner of code to create these directories in the command line:

for i in $(seq -f "%03g" 1 6); do mkdir ${i}; done
#this will create directories 001 to 006. Obviously, if you need more directories, change the second value from 6 to something else

Protip: If you want to also make the runs file that some of our scripts use at the same time, the above snippet can be modified:

 for i in $(seq -f "%03g" 1 6); do mkdir ${i}; echo ${i} >> runs; done

Restart Window Manager

This has happened a couple times before: you step away from the computer for awhile (maybe even overnight) and when you come back, you find it is locked up and completely unresponsive. The nuclear option is to reboot the whole machine:

sudo shutdown -r now #Sad for anyone running autorecon or a neural network

Unfortunately, that will stop anything that might be running in the background. A less severe solution might be to just restart the window manager. To do this you will need to ssh into the locked-up computer from a different computer, and then restart the lightdm process. This will require superuser privileges.

ssh hostname

Then after you have connected to the frozen computer:

sudo restart lightdm

Any processes that were dependent on the window manager will be terminated (e.g., so if you had been in the middle of editing labels in tksurfer, you will find that tksurfer has been shutdown and you will need to start over), however anything that was running in the background (e.g., autorecon) should be unaffected.

Renaming Multiple Files

Rename Using rename

A perl command, called rename might be available on your *nix system:

rename [OPTIONS] perlexpr files

Among useful options are the -n flag, which just reports what all the file renames would be, but doesn't actually execute them. A handy application of rename is to hide files and/or directories. Files with names beginning with a dot are hidden by default and don't show up in directory listings. This can be a handy way of excluding chunks of data from your scripts.

Use-Case: Hiding Session 2 Data

In our Multisensory Imagery experiment, we collect 6 runs at time points 1 and 2. If we wish to be able to analyze all the data, these would be stored together as runs 001 to 012. Suppose we wish to temporarily hide the second time point data:

rename -n 's/01/\._01/' `find ./ -type d -name "01*"`

This would find all the directories ("-type -d") named 01*, then it would show you how it would rename them. If everything looked right, you would execute the same command again, but omit the -n flag so that the renaming actually takes place. Note that this example only gets the 010, 011 and 012 directories. You would do something similar for directories 00[6-9].

Use-Case: Unhiding Directories

This one is easier, since all the hidden directories start with "._" using the approach described above:

rename 's/\._//' `find ./ -type d -name "._0*"`

In case you're curious about the syntax of the perl expression, you might want to read up a bit about regular expressions, but in this case, 's/;">\._//' indicates we are doing a substitution that will replace every instance of ._ with an empty string (//). The extra back-slash in front of the period is an escape character, which is needed because otherwise the dot (period) will be interpreted as a special character.

Rename Using mv=

If you don't have access to the rename command (Mac OSX), you can fake it:

PREFIX=LO
for file in `find . -name "*.txt"`; do mv ${file##*/} ${PREFIX}_${file##*/}; done

Source: [1]

sed Tricks

Replacing Text in Multiple Files

sed -i 's/oldtext/newtext/g' *.ext

Remove punctuation and convert to lowercase

$FILENAME=file.txt
sed 's/[[:punct:]]//g' $FILENAME | sed $'s/\t//g' | tr '[:upper:]' '[:lower:]'  > lowercase.$FILENAME

Archiving Specific Files in a Directory Tree

The tar has an --include switch which will archive only matching file patterns, however it appears that this filtering breaks when trying to archive files in subdirectories. Fortunately, the person who posed the question on StackExchange already had a workaround that works fine (it's just ugly):

 find ./ -name "*.wav.txt" -print0 | tar -cvzf ~/adhd.tgz --null -T -

No idea what the -T does, nor what the trailing - does, but there you have it. This works. Just replace your file pattern with whatever it is you're filtering out, and of course specify an appropriate tgz archive name.

mysql on the terminal

So I learned tonight how to export query results to a text file from the shell interface. Note that MySQL server is running with the --secure-file-priv option enabled, so you can't just willy-nilly write files wherever you want. However /var/lib/mysql-files/ is fair game, so for example:

select * from conceptstats inner join concepts on conceptstats.concid=concepts.concid where pid=183 and norm=1 into outfile '/var/lib/mysql-files/0183.txt'