BASH Tricks

How many lines in my text file?

Totally useful when you have some kind of training file with many rows and columns:

FILENAME=myfile.csv
nl ${FILENAME} | awk '{ print $1 }'

Make a list of directory names

We often organize subject data so that each subject gets their own directory. Freesurfer uses a subjects file when batch processing. Rather than manually type out each folder name into a text file, it can be generated in one line of code:

ls -1 -d */ | sed "s,/$,," >  subjects

This lists in 1 column all the directories (-1 -d) and uses sed to snip off the trailing forward slashes in the directory names

Make a series of numbered directories

FreeSurfer BOLD data goes in a series of directories, numbered 001, 002, ... , 0nn. A one-liner of code to create these directories in the command line:

for i in $(seq -f "%03g" 1 6); do mkdir ${i}; done
#this will create directories 001 to 006. Obviously, if you need more directories, change the second value from 6 to something else

Protip: If you want to also make the runs file that some of our scripts use at the same time, the above snippet can be modified:

 for i in $(seq -f "%03g" 1 6); do mkdir ${i}; echo ${i} >> runs; done

Restart Window Manager

This has happened a couple times before: you step away from the computer for awhile (maybe even overnight) and when you come back, you find it is locked up and completely unresponsive. The nuclear option is to reboot the whole machine:

sudo shutdown -r now #Sad for anyone running autorecon or a neural network

Unfortunately, that will stop anything that might be running in the background. A less severe solution might be to just restart the window manager. To do this you will need to ssh into the locked-up computer from a different computer, and then restart the lightdm process. This will require superuser privileges.

ssh hostname

Then after you have connected to the frozen computer:

sudo restart lightdm

Any processes that were dependent on the window manager will be terminated (e.g., so if you had been in the middle of editing labels in tksurfer, you will find that tksurfer has been shutdown and you will need to start over), however anything that was running in the background (e.g., autorecon) should be unaffected.

Renaming Multiple Files

If you don't have access to the rename command (Mac OSX), you can fake it:

PREFIX=LO
for file in `find . -name "*.txt"`; do mv ${file##*/} ${PREFIX}_${file##*/}; done

Source: [1]

sed Tricks

Replacing Text in Multiple Files

sed -i 's/oldtext/newtext/g' *.ext

Remove punctuation and convert to lowercase

$FILENAME=file.txt
sed 's/punct://g' $FILENAME | sed $'s/\t//g' | tr '[:upper:]' '[:lower:]'  > lowercase.$FILENAME

Archiving Specific Files in a Directory Tree

The tar has an --include switch which will archive only matching file patterns, however it appears that this filtering breaks when trying to archive files in subdirectories. Fortunately, the person who posed the question on StackExchange already had a workaround that works fine (it's just ugly):

 find ./ -name "*.wav.txt" -print0 | tar -cvzf ~/adhd.tgz --null -T -

No idea what the -T does, nor what the trailing - does, but there you have it. This works. Just replace your file pattern with whatever it is you're filtering out, and of course specify an appropriate tgz archive name.

mysql on the terminal

So I learned tonight how to export query results to a text file from the shell interface. Note that MySQL server is running with the --secure-file-priv option enabled, so you can't just willy-nilly write files wherever you want. However /var/lib/mysql-files/ is fair game, so for example:

select * from conceptstats inner join concepts on conceptstats.concid=concepts.concid where pid=183 and norm=1 into outfile '/var/lib/mysql-files/0183.txt'