Dammit Jim! » Bash/UNIX

Syntax highlighting in terminal

ScottS-M — Fri, 16 Oct 2009 04:53:09 +0000

I was just quickly flipping through code on a terminal and got to thinking that it would be pretty handy to be able to syntax highlight when using head or grep without having to open an editor. Luckily, I remembered Pygmentize was pretty handy when syntax highlighting in LaTeX so I wondered if they had an equivalent output for terminal. It turns out they do (in standard or 256 color no less) so in a couple minutes I had a really short script for highlighting code at the terminal saved it to bin/ccat and was ready to go. Pygmentize really is impressive. Here's the script if anyone else is looking to do the same:

BASH:

#!/bin/bash#!/bin/bash
if [ ! -t 0 ];then
file=/dev/stdin
elif [ -f $1 ];then
file=$1
else
echo "Usage: $0 code.c"
echo "or e.g. head code.c|$0"
exit 1
fi
pygmentize -f terminal -g $file

It just looks for stdin and if it doesn't find it it looks for a file name or takes direct input. Obviously it requires Pygmentize (which is really easy to install if you already have Python).

Tab Indented Standard Input Redirect in Bash

ScottS-M — Mon, 13 Jul 2009 02:52:43 +0000

I managed to forget how to redirect standard input (when you want to feed a bunch of lines to a program) in a bash script while still indenting and had to go digging around for it. So I figured I'd make a note here so I don't forget again and for anyone else in the same boat. It's just <<- instead of <<. For example if you want to keep indentation within a loop:

BASH:

for i in 1 2 3 4;do
cat<<-EOF
This is loop $i
More advanced stuff could go here
EOF
done

You can use whatever you want to indicate the end of the input instead of EOF if it floats your boat (as long as you use the same thing both times) but unfortunately <<- doesn't work with spaces for indentation (although I'm a tab man myself).

Counting Q20 Bases in a .qual File

ScottS-M — Wed, 18 Mar 2009 16:43:17 +0000

I sometimes get asked to count the number of bases with qualities greater than or equal to 20 in a quality file. I'm not entirely sure this is all that good a metric with 454 sequencing but that's another story. It always takes me a minute or two to come up with the right Unix commands to do it so I'm going to post it here so I remember (and maybe save someone else a couple minutes).

cat *qual|grep '^[^>]'|sed 's/ /\n/g'|grep -c [234][0-9]

This is very quick and dirty (just removing lines starting with ">", replacing spaces with newlines and counting the resulting lines with quals 20-40) but it seems to work ok for me. Also yes I know it's stupid to cat to a grep but I often replace the cat with head for testing. And I'm sure you could do it in a single awk or sed step but it gets done in a minute or two for several hundred million bases so I haven't really been motivated to change it.

Using quotation marks effectively in Unix

ScottS-M — Sun, 26 Nov 2006 19:00:39 +0000

This is pretty basic knowledge but I've helped a few people out recently that had been using Unix/Linux for a while and didn't know and it sure helped me out when I figured it out. If you had asked me how many quotation marks were on a keyboard before I started doing Bash stuff I would have said two. But I, and it seems most non-programmers, often forget the little ` on the same key as the tilde ~ (to the left of the numbers on standard keyboards). So there are actually three types of quotation marks and each one means something different to Unix:

Single quote/forward quote ': Just for clarity this is the key next to the Enter key on most keyboards. A pair of single quotes tells Unix that the contents are a string and that it should not mess around inside. For example: x=17;echo 'This is $x'; will return This is $x. Unix did not replace the variable $x with 17.
Double quote ": Double quotes again tell Unix that the contents are a string. Double quotes and the above single quotes useful for keeping Unix from messing up spaces. For example: grep "To be or not to be" hamlet.txt will search hamlet.txt for the famous line. In contrast, grep To be or not to be hamlet.txt searches for To in the files be, or, not, to, be, hamlet.txt (not what was intended). Double quotes are different from single quotes in that they allow Unix to replace variables inside them. For example: x=17;echo "This is $x"; will return This is 17.
Backtick/back quote `: Just for clarity this is the key shared with ~ on most keyboards. These are really different than the other two. They tell Unix to run whatever command is inside and paste in the output. These are equivalent to using $(). For example: echo "You are in `pwd`" will return You are in /usr/bin/ or whatever directory you are in. These can come in really handy. One common use is looping through files. Here's a slightly more complex example that uses both double and back quotes to add the filename to every row in all text files in a directory: for i in `ls *.txt`;do sed "s/^/$i /" $i >$i.new;done

Knowing how these are used can really help with using Unix. The difference between 'commands' and "commands" is pretty subtle but can make a big difference in the results. I learned this the hard way but hopefully now you won't have to.

Reading Standard Input in a Bash Shell Script

ScottS-M — Sat, 08 Jul 2006 12:34:01 +0000

I guess this is obvious to most people but it took me quite a while to dig it out on the internet. I wanted my bash script to be able to read from standard input like cat sourcefile.txt|bash myscript.bash. It turns out the standard input can be read from /dev/stdin. So if I wanted to sort the standard input, myscript.bash would look like: sort /dev/stdin

I think this is probably basic UNIX knowledge but I'm not a programmer.