## Monte Carlo methods, nonparametric tests and you

03/18/2012

As part of various research projects, I have occasionally developed methods for testing hypotheses about ecological and evolutionary phenomena.  A point of confusion occasionally arises for some people using these tests when they come to the point of having to compare their empirical observations to a null distribution: it’s not something they’ve done so explicitly before, and they’re not quite sure how to do it.  In this post I’m going to try to explain in the simplest possible terms how hypothesis testing, and in particular nonparametric tests based on Monte Carlo methods, work.

Let’s say we’ve got some observation based on real data.  In our case, we’ll say it’s a measurement of niche overlap between ENMs built from real occurrence points for a pair of species (figure partially  adapted (okay, stolen) from a figure by Rich Glor).  We have ENMs for two species, and going grid cell by grid cell, we sum up the differences between those ENMs to calculate a summary statistic measuring overlap, in this case D.
Due to some evolutionary or ecological question we’re trying to answer, we’d like to know whether this overlap is what we’d expect under some null hypothesis.  For the sake of example, we’ll talk about the “niche identity” test of Warren et al. 2008.  In this case, we are asking whether the occurrence points from two species are effectively drawn from the same distribution of environmental variables.  If that is the case, then whatever overlap we see between our real species should be statistically indistinguishable from the overlap we would see under that null hypothesis.  But how do we test that idea quantitatively?

In the case of good old parametric statistics, we would do that by comparing our empirical measurement to a parametric estimate of the overlap expected between two species (i.e., we would say "if the null hypothesis is true, we would expect an overlap of 0.5 with a standard deviation of .05", or something like that).  That would be fine if we could accurately make a parametric estimate of the expected distribution of overlaps under that null hypothesis, i.e., we need to be able to specify a mean and variance for expected overlap under that null hypothesis.  How do we do that?  Well, unfortunately, in our case we can’t.  For one thing we simply can’t state that null in a manner that makes it possible for us to put numbers on those expectations.  For another, standard parametric statistics mostly require the assumption that the distribution of expected measurements under the null hypothesis meets some criteria, the most frequent being that the distribution is normal.  In many cases we don’t know whether or not that’s true, but in the case of ENM overlaps we know it’s probably not true most of the time.  Overlap metrics are bound between 0 and 1, and if the null hypothesis generates expectations that are near one of those extremes, the distribution of expected overlaps is highly unlikely to be even approximately normal.  There can also be (and this is based on experience), multiple peaks in those null distributions, and a whole lot of skew and kurtosis as well.  So a specification of our null based on a normal distribution would be a poor description of our actual expectations under the null hypothesis, and as a result any statistical test based on parametric stats would be untrustworthy.  I have occasionally been asked whether it’s okay to do t-tests or other parametric tests on niche overlap statistics, and, for the reasons I’ve just listed, I feel that the answer has to be a resounding “no”.

So what’s the alternative?  Luckily, it’s actually quite easy.  It’s just a little less familiar to most people than parametric stats are, and requires us to think very precisely about the ideas we’re trying to test.  In our case, what we need to do is to find some way to estimate the distribution of overlaps expected between a pair of species using this landscape and these sample sizes if they were effectively drawn from the same distribution of environments.  What would that imply?  Well, if each of these sets of points were drawn from the same distribution, we should be able to generate overlap values similar to our empirical measurement by repeating that process.  So that’s exactly what we do!

We take all of the points for these two species and we throw them in a big pool.  Then we randomly pull out points for two species from that pool, keeping the sample sizes consistent with our empirical data.  Then we build ENMs for those sets of points and measure overlaps between them.  That gives us a single estimate of expected overlaps under the null hypothesis.  So now we’ve got our empirical estimate (red) and one realization of the null hypothesis (blue)

All right, so it looks like based on that one draw from the null distribution, our empirical overlap is a lot lower than you’d expect.  But how much confidence can we have in this conclusion can we have based on one single draw from the null distribution?  Not very much.  Let’s do it a bunch more times and make a histogram:
All right, now we see that, in 100 draws from that null distribution, we never once drew an overlap value that was as low as the actual value that we get from our empirical data.  This is pretty strong evidence that, whatever process generated our empirical data, it doesn’t look much like the process that generated that null distribution, and based on this evidence we can statistically reject that null hypothesis.  But how do we put a number on that?  Easy!  All we need to do is figure out what the percentile in that distribution is that corresponds to our empirical measurement.  In this case our empirical value is lower than the lowest number in our null distribution.  That being the case, we can’t specify exactly what the probability of getting our empirical result is, only that it’s lower than the lowest value we obtained, so it’s p < (whatever that number is).  Since we did 100 iterations of that null hypothesis, the resolution of our null distribution is 1/100 = .01.  Given our resolution, that means p is between 0 and .01 or, as we normally phrase it, p < .01.  If we’d done 500 simulation runs and our empirical value was still lower than our lowest value, it would be p < 1/500, or p < .0002.  If we’d done 500 runs and found that our empirical value was between the lowest value and the second lowest value, we would know that .0002 < p < .0004, although typically we just report these things as p < .0004.  Basically the placement of our empirical value in the distribution of expected values from our null hypothesis is an estimate of the probability of getting that value if that hypothesis were true.  This is exactly how hypothesis testing works in parametric statistics, the only difference being that in our case we generated the null distribution from simulations rather than specifying it mathematically.

So there you go!  We now have a nonparametric test of our hypothesis.  All we had to do was (1) figure out precisely what our null hypothesis was, (2) devise a way to generate the expected statistics if that hypothesis were true, (3) generate a bunch of replicate realizations of that null hypothesis to get an expected distribution under that null, and (4) compare our empirical observations to that distribution.  Although this approach is certainly less easy than simply plugging your data into Excel and doing a t-test or whatnot, there are many strengths to the Monte Carlo approach. For instance, we can use this approach to test pretty much any hypothesis that we can simulate – as long as we can produce summary statistics from a simulation that are comparable to our empirical data, we can test the probability of observing our empirical data under the set of assumptions that went into that simulated data.  It also means we don’t have to make assumptions about the distributions that we’re trying to test – by generating those distributions directly and comparing our empirical results to those distributions, we manage to step around many of the assumptions that can be problematic for parametric statistics.

The chief difficulty in applying this method is in steps 2 and 3 above – we have to be able to explicitly state our null hypothesis, and we have to be able to generate the distribution of expected measurements under that null.  Honestly, though, I think this is actually one of the greatest strengths of Monte Carlo methods: while this process may be more intensive than sticking our data into some plug-and-chug stats package, it requires us to think very carefully about what precisely our null hypothesis means, and what it means to reject it.  It requires more work, but more importantly it requires a more thorough understanding of our own data and hypotheses.

## Author

Dan Warren is a postdoctoral researcher in the Parmesan lab at UT Austin.

03/03/2013 12:42

06/12/2013 00:35

Monte Carlos methods is a good piece of information for me. I am very much excited to read more about it from this blog and greatly indebted to you for such a vibrant post. I will recall you as one of the best bloggers I have met in my life and wish you good luck.

07/20/2013 03:17

This is exactly how hypothesis testing works in parametric statistics, the only difference being that in our case we generated the null distribution from simulations rather than specifying it mathematically.

07/23/2013 03:24

The simplest possible terms how hypothesis testing, and in particular nonparametric tests based on Monte Carlo methods, work.

07/29/2013 00:40

Thank you for another essential article. Where else could anyone get that kind of information in such a complete way of writing? I have a presentation incoming week, and I am on the lookout for such information.

07/29/2013 01:36

a vibrant post. I will recall you as one of the best bloggers I have met in my life and wish you good luck.

07/30/2013 10:06

Thank you for another essential article. Where else could anyone get that kind of information in such a complete way of writing? I have a presentation incoming week, and I am on the lookout for such information. This is exactly how hypothesis testing works in parametric statistics,

07/30/2013 22:45

A point of confusion occasionally arises for some people using these tests when they come to the point of having to compare their empirical observations to a null distribution: it’s not something they’ve done so explicitly before, and they’re not quite sure how to do it. In this post I’m going to try to explain in the simplest possible terms how hypothesis testing, and in particular nonparametric tests based on Monte Carlo methods, work.

07/31/2013 02:54

A point of confusion occasionally arises for some people using these tests when they come to the point of having to compare their empirical observations to a null distribution: it’s not something they’ve done so explicitly before, and they’re not quite sure how to do it.

08/01/2013 06:01

This is fetching sturdy testify that, whatever writ produced our experimental material, it doesn’t physiognomy ample prefer the procedure that produced that void dispensation, moreover based on this symptom we can statistically forsake that nothing hypothesis.

08/07/2013 10:57

Hi there! I could have sworn I've been to this site before but after browsing through some of the post I realized it's new to me. Anyhow, I'm definitely glad I found it and I'll be book-marking and checking back frequently!

08/10/2013 04:56

I was also searching related stuff from long time.You have solved my problem.Thanks for sharing this great stuff with us.

08/19/2013 12:34

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.

08/21/2013 07:37

In such cases, we have been wanting to know perhaps the incidence items by 2 types usually are efficiently driven in the similar submission regarding the environmental specifics.

08/28/2013 00:45

I have occasionally developed methods for testing hypotheses about ecological and evolutionary phenomena.

09/04/2013 09:41

We own ENMs for couple species, further going network compartment by network prison, we compendium up the variations amid those ENMs to reckon a summary statistic measuring intersect, in this case D.

09/05/2013 00:12

09/07/2013 03:52

In our case, we’ll say it’s a measurement of niche overlap between ENMs built from real occurrence points for a pair of species (figure partially adapted (okay, stolen) from a figure by Rich Glor). We have ENMs for two species, and going grid cell by grid cell, we sum up the differences between those ENMs to calculate a summary statistic measuring overlap, in this case D.

09/07/2013 04:40

Thank you for posting the great content…I was looking for something like this…, hopefully you will keep posting such blogs…

09/07/2013 13:49

If that is the case, then whatever overlap we see between our real species should be statistically indistinguishable from the overlap we would see under that null hypothesis.

09/14/2013 06:57

In this dispatch I’m going to harass to explicate in the simplest viable names how hypothesis testing, moreover in detail nonparametric proves based on Monte Carlo disciplines, process.

09/18/2013 01:15

Overlap metrics are bound between 0 and 1, and if the null hypothesis generates expectations that are near one of those extremes, the distribution of expected overlaps is highly unlikely to be even approximately normal.

09/18/2013 03:57

Nice post and good design web, its very clean people be enjoying to read and search in your article..

09/18/2013 12:55

I am an overseas student in Australia and my instructor says my proficiency in English language is not up to the mark. I need immediate but reasonably priced essay writer for my class assignments. What are the charges and how long does it take to complete one essay?

09/19/2013 06:58

Excellent is the only word I can give u for this wonderful blog, keep it up. I will come back again to read some more interesting things on this topic.

09/19/2013 14:54

Overlap metrics are bound between 0 and 1, and if the null hypothesis generates expectations that are near one of those extremes, the distribution of expected overlaps is highly unlikely to be even approximately normal.

09/20/2013 20:47

09/22/2013 07:12

Total utility condition is mere beneficial yet there are few topics beside the stipend computers. The locale essay editing online attempts a marry of honorarium selections such as Visa moreover Mastercard.

09/23/2013 06:19

This site is excellent and so is how the subject matter was explained. I also like some of the comments too.Waiting for next post.

10/30/2013 21:00

Perfect! These articles written too great, they rich contents and data are accurately.they help me.I expect to see your new share

09/24/2013 08:24

Thank you for sharing excellent information. Your web-site is very cool. I’m impressed by the details that you¡¦ve on this web site. It reveals how nicely you understand this subject.

09/25/2013 23:41

I admit with your Blog and I will be back to examine it more in the future so please keep up your work.

09/26/2013 04:52

We hold ENMs for twin species, moreover going network dungeon by gridiron chamber, we aggregate up the disparitys amid those ENMs to plan a synopsis statistic measuring overhang, in this case D.

10/01/2013 01:33

I have occasionally developed methods for testing hypotheses about ecological and evolutionary phenomena.

09/26/2013 22:51

I am glad to find your impressive way of writing the post. Now it become easy for me to understand and implement the concept.

10/02/2013 05:50

Now and then I’ll stumble across a post like this and I’ll recall that there really are still interesting pages on the web. ^_^. Thanks.

10/02/2013 05:51

I like the cut of your job :) or at least your thought process but sorry to say, I honestly think you would have fully sold me on the idea had you been able to back up your premis with a substantial bit more solid facts.

10/03/2013 03:17

Bright-futures-scholarship is very good and useful information. Why not write a book about this topic.For today’s economic fluctuation and lack of opportunities, it really will be a very hot topic. Thanks anyway. All the best.

10/04/2013 04:44

That's very informative.. Thanks for sharing this. There are only few people who cares for the environment.

10/06/2013 23:43

thanks ..i have bookmarked it already

10/07/2013 11:30

Found this blog from Weebly's index, nice!

10/09/2013 04:20

A spot of mess hardly looms for sundry specimens using these experiments when they come to the point of having to contemplate their test words to a nugatory exemption: it’s not something they’ve weary so explicitly anterior, yet they’re not exceedingly true how to do it.

10/09/2013 22:39

For the sake of example, we’ll talk about the “niche identity” test of Warren et al. 2008. In this case, we are asking whether the occurrence points from two species are effectively drawn from the same distribution of environmental variables.

10/11/2013 04:37

Blog is very informative and wonderful,

10/18/2013 22:55

10/22/2013 02:40

hi good looking your site nice posting thanks for sharing information i like this blog.

10/29/2013 05:31

Hello Love your article. Special thanks to the author. Hope to write some more.

11/12/2013 01:17

I also found your posts very interesting. In fact after reading, I had to go show it to my friend and he enjoyed it as well!Thanks so much for this!

11/28/2013 04:56

12/02/2013 21:17

I wish more authors of this type of content would take the time you did to research and write so well. I am very impressed with your vision and insight.