One of the cool things you can do with the high throughput DNA analysis of pyrosequencing, is to collect a sample from the environment, isolate the DNA from everything in it and sequence it. Then you can match the DNA up with known sequences and see what sort of microbes you had. Dinsdale and a bunch of coauthors collected the data from a bunch of such studies. They managed to find 45 bacterial samples and 42 viral samples from 9 broad environmental classifications. You can see all the different samples the authors pooled together (circles microbial and squares viral).

The interesting thing about this study was that instead of looking at the taxonomy of the critters as usual, they looked at the function of the genes. By simply looking at what the genes do, the researchers hoped to get a feel for what activities were going on in that environment without necessarily having to identify the species of the bacteria and viruses. To do this, they fed their 14.5 million sequences (pyrosequencing sure can generate data) into the SEED database, a big collection of genes which have been assigned to functions (for example membrane transport or sulphur metabolism) by experts. They were able to match 1 million of the bacterial and 500,000 of the viral sequences to previously identified gene functions.

It might seem odd that they would look at viral DNA since viruses are rather simple and have only a few basic genes. But the researchers were actually looking at bacterial genetic sequences being carried inside viruses. This of course brings up the question of what bacterial DNA is doing inside viruses. It turns out there are a lot of bacteriophage viruses that like to infect bacteria and sometimes these viruses capture some of the DNA of their bacterial hosts and carry it to their next host. Looking at the bacterial DNA present in a viral population gives an interesting look at what types of genes are being passed around between individual bacteria (and even between bacterial species).

So here are the high level classifications of the function of the genes they found for each environment.

It’s pretty cool that the viruses were carrying around so much of a variety of bacterial DNA. The authors suggest that motility genes coding for things like flagella and cilia (which could help the bacterial host spread the virus further) were enriched in the viral samples but it seems a bit hard to say that for certain without a bit more analysis.

A useful way to look at huge masses of data, like their 1.5 million matches, is to try and reduce all the different counts in the functional categories into a couple of condensed variables. This can be seen in the next couple plots. They could use a little explaining. Bacterial sequences are on top and viral sequences on the bottom. Lines show how the various functional categories have been condensed into the x and y variables. For example, samples that contained lots of genes for making cell walls will tend to be at the top of the plot in the bacterial samples and tend not to have many genes for respiration.

It’s pretty cool to see how the various environments clustered with other samples from the same environment. For example, all the yellow diamond fish farm samples ended up on the right side of the bacteria graphs even though they were sampled independently. It appears that functions seem to correlate with environmental conditions. For example, the fish food at the fish farms contained a lot of sulfur supplements and the bacteria from those samples were rich in sulfur metabolism genes and the bacteria from corals contained many different respiration genes to deal with the highly variable oxygen concentrations found there. Dinsdale and her coauthors go so far as to suggest that gene function may provide a better indicator of environment than the taxonomy of the bacteria present.

The paper did have a little trouble in the math in one part but the authors already have a correction in for it so it’s really not worth worrying about. Overall, it was a pretty interesting story and a good example of stuff to do with a sequencing machine (also it must have taken a good bit of work to collect all that data together from all those authors).

References

Elizabeth A. Dinsdale, Robert A. Edwards, Dana Hall, Florent Angly, Mya Breitbart, Jennifer M. Brulc, Mike Furlan, Christelle Desnues, Matthew Haynes, Linlin Li, Lauren McDaniel, Mary Ann Moran, Karen E. Nelson, Christina Nilsson, Robert Olson, John Paul, Beltran Rodriguez Brito, Yijun Ruan, Brandon K. Swan, Rick Stevens, David L. Valentine, Rebecca Vega Thurber, Linda Wegley, Bryan A. White, Forest Rohwer (2008). Functional metagenomic profiling of nine biomes Nature, 452 (7187), 629-632 DOI: 10.1038/nature06810

There was some discussion in one of my SAS posts about where to find SAS help and communities. It seemed like a pretty useful topic so I thought I’d expand it a bit and make a post out of it. First, let me say I’m not the most knowledgeable since I’m more of a find-wall-bang-head type of programmer but I did my best to dig up some possible answers. If anyone has any other suggestions, feel free to leave them in the comments.

• To start with, there’s always the official online documentation although this tends to be more for polishing something you already know how to do than starting cold.
• Speaking of official, there’s also the official SAS forums. I didn’t know about these until I started looking around for this post so I can’t say much about them but the topics they have available seem rather specific and I can’t figure out where one would go to post a basic question.
• Edit:There’s also the SAS Knowledge Base that has a lot of good papers and notes detailing SAS features complete with sample code and explanations. It’s really useful if you’re a learn by example type. (Thanks to Alison for pointing this one out).
• Kelly Levoyer of SAS points out SAScommunity.org which seems like it is a little sparse but does have a surprisingly long list of SAS-related blogs.
• The SAS company also appears to have jumped on the blogging band wagon although really only SAS Dummy looks helpful for learning SAS at the moment.
• The only place that seem to be available for asking general question is the SAS-L email list (which I just found out is the same as the comp.soft-sys.sas Usenet group). There’s a nice paper on SAS-L etiquette (mostly do your homework first) (found via the sascommunity site).

Offline, there are also SAS user groups. I often get emails from our local one but I’ve never actually gone. The SAS company also has trainers that travel and teach quick classes. Our university stats department brought in one to teach a couple short two-day classes about statistical functions and macros. The classes were pretty good although I’m not sure how much it cost or how frequent they are. It might be worth checking on if you’re near a university.

Finally, you can also read my poor attempts at explaining SAS macro variables and SAS macros. Also, if you have any specific questions you can try asking in the comments here and if it’s not too time consuming I’ll try to lend a hand.

So what is a macro? Macros are a part of SAS that look through your code before the normal part of SAS sees it and writes out your code for you based on a special syntax. If you've ever found yourself copying and pasting code then you've probably been in a situation well suited for macros. They're also great if you need to perform different functions under different conditions. Once I learned macros, SAS seemed a lot more like a usable (although weird) programming language and tasks seemed to get a lot easier (except actually picking the statistical techniques to use).

Probably the easiest way to see what macros do is an example. So say we once again have a data set of tree heights

I already talked about how to find and use the mean and standard deviation for the whole data set. Now what if we wanted to standardize each species by its own seperate mean and deviation? We could cut and paste but once we get a few more species or want to change something later this really becomes a hassle. So this is where macros come in.

The first thing to do is to calculate the mean and standard deviations for each species. We can use proc means again to do this. Since we won't be using the output I'll add the noprint option and since we only want the means for the individual species and not the whole dataset I'll add the nway option. The class name; statement tells SAS to find the statistics seperately for each species and the output line tells SAS to save the mean and deviation in a dataset called meansd.

Now we just need to get the values from the meansd dataset into macro variables. We'll use the _NULL_ dataset and call symput again to create macro variables. This time we need to create seperate macro variables for each species. Luckily SAS automatically numbers each observation in a dataset in a column called _N_. Since each line of the dataset corresponds to a tree species, we can easily use this identifier to create the macro variables by using call symput('mean'||left(_N_), meanheight);. The left() and ()trim functions (numeric variables have extra spaces to the left and string variables have spaces to the right) removes any unnecessary spaces and the || concatenates (connects) the text "mean" with the line number to give give mean1, mean2, etc.. I'll do the same thing for standard deviation and tree name. Once the macro variables are created, there is still one problem remaining. We don't know how many species there were or how many macro variables were created. Luckily SAS will make another column that indicates the last line of the dataset when it sees end=newcolumnname following a set statement. Then we just need to check if SAS is on the last line and if so save the line number (_N_) to know the number of species of trees.

If you ever want to check what macro variables you have in your program, you can use %PUT _USER_; to print them all to the log file. Or if you want to see every macro variable available (SAS has quite a few automatic ones like operating system and date) use %PUT _ALL_;. Inserting %PUT _USER_; here produces:

Now we've set a lot of macro variables but we still haven't created a real macro. In SAS, macros are started with %MACRO macroname; and finished with %MEND; (short for M[acro]END). %'s are used to indicate commands that the SAS macro facility will read and remove before normal SAS sees the code. Anything not with a % will be printed out by the macro facility. Macros don't spit out their code for SAS until they're are called using %macroname.

So I'll call my macro treestandardizer but you can call it whatever you want. I'm going to use a pretty simple and specific macro but if you were going to use this often and for different datasets you would want to program it better. The first thing to do is create the final dataset and set it to the trees dataset. Since we need to loop through each species of tree, we'll need a %DO loop. Everything between %DO and %END will be repeated while i increments from 1 to the number of tree species. If you want to combine text and a macro variable to reference another macro variable, you use the double ampersand && in SAS. For example, we want to get the mean for species 1 by looking in the macro variable &mean1 so we use &&mean&i. I think the macro processing part of SAS ends up running through the code twice, the first time finding the && and replacing it with & and the &i and replacing it with 1 to leave &mean1 and the second time finding &mean1 and pasting in the appropriate value (65). So we'll have the do loop write out a series of if statements to check what the name of the tree is and use the appropriate mean and deviation. Note that when using a string macro variable like &nameX, you need to surround it with double quotes (the macro processor doesn't look inside single quotes) so SAS doesn't think it is a variable name.

The previous code prepared the macro but nothing actually happens until we call it using %treestandardizer. Unlike almost everything else in SAS this line doesn't have to end in a semicolon (although it's pretty unlikely to hurt if you forget and add one). So to call the macro:

If you want to see what happens when you call a macro, you can have SAS print the code generated by the macro to the log file with the option option mprint; (make sure to set it before actually calling the macro). In this case, it gives:

So it worked and we now have the standardized heights in the stheight column of the final dataset. This particular example could be done a few different ways (the easiest and probably better way being to merge the meancv dataset with the trees) but I hope it gives a decent introduction to SAS macros. If you have any specific questions or something wasn't clear, feel free to ask in a comment.

Here is the SAS source code if you don't feel like copying and pasting.

After making the WP_MonsterID WordPress plugin to create a random monster avatar from an assortment of parts for each commenter (based on other people's code), fruityoaty asked This looks nifty, but how many monster images are available for assigning?

I'd been meaning to calculate this anyway so I did the math and posted it in the comments:

The current part totals are: 17 eyes, 8 hairs, 12 mouths, 15 bodies, 10 legs. That is 244,800 possible combinations. In addition, the body color can range between 20-235 for red, green, and blue. If we count that as 20 distinguishable values for red, green, and blue that adds 8000 possible colors and brings the unique monster count to 2 billion. The only problem is that the algorithm is only using the first 6 digits of the md5 hash of the email which only provides 16 million possible combinations. So I guess the answer is 16 million monsters currently and in the next release I’ll use a few more digits of the hash and increase it to a billion or so. Edit: I did change this so in version 0.3 and later there should be a couple billion possibilities.

Calculating this got me wondering how many unique users it would take before there was likely to be a duplicate monster. For two users it was easy (1 out of 2 billion) but as the number of users increased things got messy since each new monster could match any of the prior monsters. Luckily I remembered enough of my stats class to google for something on calculating the chance of people in a group sharing a birthday.

If you've never heard of this problem, stop and take a quick guess for how many people you think it would take for the odds to be better than 50% for two people sharing a birthday. Or as my statistics professor put it, There are twenty-five people in the room will you bet me that no one shares the same birthday?

...

...

Guessed?

Now I know just enough statistics to know betting against a statistics professor is a bad idea but I have to say, at the time, I thought it would have been a fairly good bet. It turns out that I, like most non-statistics professors, underestimated the chance of any two people in a group sharing the same birthday. Actually, if there are 23 people in a room there is a greater than 50% chance that at least two will share a birthday. If there are 47 people in a room, there is a 95% chance that at least one pair share a birthday. This greatly increasing probability occurs because like the monsters each person added to the room can match any of the previous people (the 5th person can match person 1,2,3,4; the 25th person can match person 1,2,3,4,5,6,...,24;...).

All this was interesting in understanding the problem but didn't really get any closer to finding the probability. Luckily Wikipedia provides an approximation for determining the number of people at a given probability of overlap:

Substituting in 2 billion for 365, results in a probablity of overlap that looks like:

Even with 2 billion monsters there is still a 50% chance of overlap with only 52,000 monsters and a 1 out of 10 chance of overlap with only 20,000 monsters. Most unintuitive to me is that there's a 99% chance of overlap with only 135,000 monsters. The chances of an overlap really does pile up as the number of already present monsters grow. In the plus side, most normal sized blogs should be safe from monster overlap with only a .1% chance of overlap even with 2000 commenters.

So what does all this mean? Well besides not getting suckered in any birthday betting, it's a good reminder to be careful about assuming uniqueness among a group just because the chance of a match is rare. For example, if in some application each user was assigned a random key of 4 digits (10000 possible combinations). There would be a greater than 50% chance of overlap after only 1% (117 users) of the keys were assigned.

If any one feels like messing around with the calculations themselves here's the function in R to calculate the miminimum number of assignments to reach a certain probability of overlap from a total number of possible combinations. I'm sure it would be trivial to convert to any other language. Note that's natural log not log10. number_assignments=function(total_number,probability_overlap){sqrt(2*total_number*log(1/(1-probability_overlap)))}