Difference between revisions of "General Statistics"

From Bridges Lab Protocols
Jump to: navigation, search
(Updated to match github version)
(Updated general statistics wiki page)
Line 1: Line 1:
[[ Category: General ]]
+
<div class="cell">
[[ Category: Data ]]
+
[[ Category: Statistics ]]
+
  
= General Statistical Methods =
+
<syntaxhighlight lang="r">library(knitr)
 +
#figures makde will go to directory called figures, will make them as both png and pdf files
 +
opts_chunk$set(fig.path='figures/',dev=c('png','pdf'))
 +
options(scipen = 2, digits = 3)
 +
# set echo and message to TRUE if you want to display code blocks and code output respectively
  
There are several important concepts that we will adhere to in our group. These involve design considerations, execution considerations and analysis concerns. The standard for our field is null hypothesis significance testing, which means that we are generally comparing our data to a null hypothesis, generating an '''effect size''' and a '''p-value'''. As a general rule, we report both of these both within our Rmd scripts, and in our publications.
+
knitr::knit_hooks$set(inline = function(x) {
 +
  knitr:::format_sci(x, 'md')
 +
})
  
We generally use an <math>\alpha</math> of <math>p<0.05</math> to determine significance, which means that (if true) we are rejecting the null hypothesis.
 
  
== Experimental Design ==
+
superpose.eb <- function (x, y, ebl, ebu = ebl, length = 0.08, ...)
 +
  arrows(x, y + ebu, x, y - ebl, angle = 90, code = 3,
 +
  length = length, ...)
  
Where possible, prior to performing an experiment or study perform a power analysis. This is mainly to determine the appropriate sample sizes. To do this, you need to know a few of things:
+
 
 +
se <- function(x) sd(x, na.rm=T)/sqrt(length(x))
  
* Either the sample size or the difference. The difference is provided in standard deviations. This means that you need to know the standard deviation of your measurement in question. It is a good idea to keep a log of these for your data, so that you can approximate what this is. If you hope to detect a correlation you will need to know the expected correlation coefficient.
+
#load these packages, nearly always needed
* The desired false positive rate (normally 0.05). This is the rate at which you find a difference where there is none. This is also known as the type I error rate.
+
library(tidyr)
* The desired power (normally 0.8). This indicates that 80% of the time you will detect the effect if there is one. This is also known as 1 minus the false negative rate or 1 minus the Type II error rate.
+
library(dplyr)</syntaxhighlight>
 +
<div class="cell-output cell-output-stderr">
  
We use the R package '''pwr''' to do a power analysis (Champely, 2020). Here is an example:
+
<pre>
 +
Attaching package: 'dplyr'</pre>
  
=== Pairwise Comparasons ===
+
</div>
 +
<div class="cell-output cell-output-stderr">
  
<pre class="r">require(pwr)
+
<pre>The following objects are masked from 'package:stats':
false.negative.rate &lt;- 0.05
+
statistical.power &lt;- 0.8
+
sd &lt;- 3.5 #this is calculated from known measurements
+
difference &lt;- 3  #you hope to detect a difference
+
pwr.t.test(d = difference, sig.level = false.negative.rate, power=statistical.power)</pre>
+
<pre>##
+
##      Two-sample t test power calculation
+
##
+
##              n = 3.07
+
##              d = 3
+
##      sig.level = 0.05
+
##          power = 0.8
+
##    alternative = two.sided
+
##
+
## NOTE: n is number in *each* group</pre>
+
This tells us that in order to see a difference of at least 3, with at standard devation of 3.5 we need at least '''3''' observations in each group.
+
  
=== Correlations ===
+
    filter, lag</pre>
  
The following is an example for detecting a correlation.
+
</div>
 +
<div class="cell-output cell-output-stderr">
  
<pre class="r">require(pwr)
+
<pre>The following objects are masked from 'package:base':
false.negative.rate &lt;- 0.05
+
statistical.power &lt;- 0.8
+
correlation.coefficient &lt;- 0.6 #note that this is the r, to get the R2 value you will have to square this result.
+
pwr.r.test(r = correlation.coefficient, sig.level = false.negative.rate, power=statistical.power)</pre>
+
<pre>##
+
##      approximate correlation power calculation (arctangh transformation)
+
##
+
##              n = 18.6
+
##              r = 0.6
+
##      sig.level = 0.05
+
##          power = 0.8
+
##    alternative = two.sided</pre>
+
This tells us that in order to detect a correlation coefficient of at least 0.6 (or an R^2 of 0.36) you need more than '''18''' observations.
+
  
 +
    intersect, setdiff, setequal, union</pre>
 +
 +
</div>
 +
<syntaxhighlight lang="r"># sets maize and blue color scheme
 +
color.scheme <- c('#00274c', '#ffcb05')</syntaxhighlight>
 +
 +
</div>
 +
<span id="general-statistical-methods"></span>
 +
= General Statistical Methods =
 +
 +
There are several important concepts that we will adhere to in our group. These involve design considerations, execution considerations and analysis concerns. The standard for our field is null hypothesis significance testing, which means that we are generally comparing our data to a null hypothesis, generating an '''effect size''' and a '''p-value'''. As a general rule, we report both of these both within our Rmd/qmd scripts, and in our publications.
 +
 +
We generally use an <math display="inline">\alpha</math> of <math display="inline">p<0.05</math> to determine significance, which means that (if true) we are rejecting the null hypothesis. This is known as null hypothesis significance testing.
 +
 +
An alternative approach is to use a Bayesian approach, described in more detail in [https://bridgeslab.github.io/Lab-Documents/Experimental%20Policies/bayesian-analyses.html this document]
 +
 +
<span id="pairwise-testing"></span>
 
= Pairwise Testing =
 
= Pairwise Testing =
  
If you have two groups (and two groups only) that you want to know if they are different, you will normally want to do a pairwise test. This is '''not''' the case if you have paired data (before and after for example). The most common of these is something called a Student's ''t''-test, but this test has two key assumptions:
+
If you have two groups (and two groups only) that you want to know if they are different, you will normally want to do a pairwise test. This is '''not''' the case if you have paired data (before and after for example). The most common of these is something called a Student’s ''t''-test, but this test has two key assumptions:
  
 
* The data are normally distributed
 
* The data are normally distributed
 
* The two groups have equal variance
 
* The two groups have equal variance
  
 +
<span id="testing-the-assumptions"></span>
 
== Testing the Assumptions ==
 
== Testing the Assumptions ==
  
 
Best practice is to first test for normality, and if that test passes, to then test for equal variance
 
Best practice is to first test for normality, and if that test passes, to then test for equal variance
  
 +
<span id="testing-normality"></span>
 
=== Testing Normality ===
 
=== Testing Normality ===
  
 
To test normality, we use a Shapiro-Wilk test (details on [https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test Wikipedia] on each of your two groups). Below is an example where there are two groups:
 
To test normality, we use a Shapiro-Wilk test (details on [https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test Wikipedia] on each of your two groups). Below is an example where there are two groups:
  
<pre class="r">#create seed for reproducibility
+
<div class="cell">
 +
 
 +
<syntaxhighlight lang="r">#create seed for reproducibility
 
set.seed(1265)
 
set.seed(1265)
test.data &lt;- tibble(Treatment=c(rep(&quot;Experiment&quot;,6), rep(&quot;Control&quot;,6)),
+
test.data <- tibble(Treatment=c(rep("Experiment",6), rep("Control",6)),
 
           Result = rnorm(n=12, mean=10, sd=3))
 
           Result = rnorm(n=12, mean=10, sd=3))
#test.data$Treatment &lt;- as.factor(test.data$Treatment)
+
#test.data$Treatment <- as.factor(test.data$Treatment)
kable(test.data, caption=&quot;The test data used in the following examples&quot;)</pre>
+
kable(test.data, caption="The test data used in the following examples")</syntaxhighlight>
{|
+
<div class="cell-output-display">
 +
 
 +
{| class="wikitable"
 
|+ The test data used in the following examples
 
|+ The test data used in the following examples
! Treatment
 
!align="right"| Result
 
 
|-
 
|-
| Experiment
+
! style="text-align: left;"| Treatment
|align="right"| 11.26
+
! style="text-align: right;"| Result
 
|-
 
|-
| Experiment
+
| style="text-align: left;"| Experiment
|align="right"| 8.33
+
| style="text-align: right;"| 11.26
 
|-
 
|-
| Experiment
+
| style="text-align: left;"| Experiment
|align="right"| 9.94
+
| style="text-align: right;"| 8.33
 
|-
 
|-
| Experiment
+
| style="text-align: left;"| Experiment
|align="right"| 11.83
+
| style="text-align: right;"| 9.94
 
|-
 
|-
| Experiment
+
| style="text-align: left;"| Experiment
|align="right"| 6.56
+
| style="text-align: right;"| 11.83
 
|-
 
|-
| Experiment
+
| style="text-align: left;"| Experiment
|align="right"| 11.41
+
| style="text-align: right;"| 6.56
 
|-
 
|-
| Control
+
| style="text-align: left;"| Experiment
|align="right"| 8.89
+
| style="text-align: right;"| 11.41
 
|-
 
|-
| Control
+
| style="text-align: left;"| Control
|align="right"| 11.59
+
| style="text-align: right;"| 8.89
 
|-
 
|-
| Control
+
| style="text-align: left;"| Control
|align="right"| 9.39
+
| style="text-align: right;"| 11.59
 
|-
 
|-
| Control
+
| style="text-align: left;"| Control
|align="right"| 8.74
+
| style="text-align: right;"| 9.39
 
|-
 
|-
| Control
+
| style="text-align: left;"| Control
|align="right"| 6.31
+
| style="text-align: right;"| 8.74
 
|-
 
|-
| Control
+
| style="text-align: left;"| Control
|align="right"| 7.82
+
| style="text-align: right;"| 6.31
 +
|-
 +
| style="text-align: left;"| Control
 +
| style="text-align: right;"| 7.82
 
|}
 
|}
  
 +
 +
</div>
 +
 +
</div>
 
Each of the two groups, in this case '''Test''' and '''Control''' must have Shapiro-Wilk tests done separately. Some sample code for this is below (requires dplyr to be loaded):
 
Each of the two groups, in this case '''Test''' and '''Control''' must have Shapiro-Wilk tests done separately. Some sample code for this is below (requires dplyr to be loaded):
  
<pre class="r">#filter only for the control data
+
<div class="cell">
control.data &lt;- filter(test.data, Treatment==&quot;Control&quot;)
+
 
 +
<syntaxhighlight lang="r">#filter only for the control data
 +
control.data <- filter(test.data, Treatment=="Control")
 
#The broom package makes the results of the test appear in a table, with the tidy command
 
#The broom package makes the results of the test appear in a table, with the tidy command
 
library(broom)
 
library(broom)
  
 
#run the Shapiro-Wilk test on the values
 
#run the Shapiro-Wilk test on the values
shapiro.test(control.data$Result) %&gt;% tidy %&gt;% kable</pre>
+
shapiro.test(control.data$Result) %>% tidy %>% kable(caption="Shapiro-Wilk test for normality of control data")</syntaxhighlight>
{|
+
<div class="cell-output-display">
!align="right"| statistic
+
 
!align="right"| p.value
+
{| class="wikitable"
! method
+
|+ Shapiro-Wilk test for normality of control data
 
|-
 
|-
|align="right"| 0.968
+
! style="text-align: right;"| statistic
|align="right"| 0.88
+
! style="text-align: right;"| p.value
| Shapiro-Wilk normality test
+
! style="text-align: left;"| method
 +
|-
 +
| style="text-align: right;"| 0.968
 +
| style="text-align: right;"| 0.88
 +
| style="text-align: left;"| Shapiro-Wilk normality test
 
|}
 
|}
  
<pre class="r">experiment.data &lt;- filter(test.data, Treatment==&quot;Experiment&quot;)
+
 
shapiro.test(test.data$Result) %&gt;% tidy %&gt;% kable</pre>
+
</div>
{|
+
<syntaxhighlight lang="r">experiment.data <- filter(test.data, Treatment=="Experiment")
!align="right"| statistic
+
shapiro.test(test.data$Result) %>% tidy %>% kable(caption="Shapiro-Wilk test for normality of the test data")</syntaxhighlight>
!align="right"| p.value
+
<div class="cell-output-display">
! method
+
 
 +
{| class="wikitable"
 +
|+ Shapiro-Wilk test for normality of the test data
 
|-
 
|-
|align="right"| 0.93
+
! style="text-align: right;"| statistic
|align="right"| 0.377
+
! style="text-align: right;"| p.value
| Shapiro-Wilk normality test
+
! style="text-align: left;"| method
 +
|-
 +
| style="text-align: right;"| 0.93
 +
| style="text-align: right;"| 0.377
 +
| style="text-align: left;"| Shapiro-Wilk normality test
 
|}
 
|}
  
 +
 +
</div>
 +
 +
</div>
 
Based on these results, since both p-values are &gt;0.05 we do not reject the presumption of normality and can go on. If one or more of the p-values were less than 0.05 we would then use a Mann-Whitney test (also known as a Wilcoxon rank sum test) will be done, see below for more details.
 
Based on these results, since both p-values are &gt;0.05 we do not reject the presumption of normality and can go on. If one or more of the p-values were less than 0.05 we would then use a Mann-Whitney test (also known as a Wilcoxon rank sum test) will be done, see below for more details.
  
 +
<span id="testing-for-equal-variance"></span>
 
=== Testing for Equal Variance ===
 
=== Testing for Equal Variance ===
  
We generally use the [https://cran.r-project.org/web/packages/car/index.html car] package which contains code for [https://en.wikipedia.org/wiki/Levene%27s_test Levene's Test] to see if two groups can be assumed to have equal variance:
+
We generally use the [https://cran.r-project.org/web/packages/car/index.html car] package which contains code for [https://en.wikipedia.org/wiki/Levene%27s_test Levene’s Test] to see if two groups can be assumed to have equal variance. For more details see @car:
  
<pre class="r">#load the car package
+
<div class="cell">
library(car)
+
  
#runs the test, grouping by the Treatment variable
+
<syntaxhighlight lang="r">#load the car package
leveneTest(Result ~ Treatment, data=test.data) %&gt;% tidy %&gt;% kable</pre>
+
library(car)</syntaxhighlight>
{|
+
<div class="cell-output cell-output-stderr">
!align="right"| statistic
+
 
!align="right"| p.value
+
<pre>Loading required package: carData</pre>
!align="right"| df
+
 
!align="right"| df.residual
+
</div>
 +
<div class="cell-output cell-output-stderr">
 +
 
 +
<pre>
 +
Attaching package: 'car'</pre>
 +
 
 +
</div>
 +
<div class="cell-output cell-output-stderr">
 +
 
 +
<pre>The following object is masked from 'package:dplyr':
 +
 
 +
    recode</pre>
 +
 
 +
</div>
 +
<syntaxhighlight lang="r">#runs the test, grouping by the Treatment variable
 +
leveneTest(Result ~ Treatment, data=test.data) %>% tidy %>% kable(caption="Levene's test on test data")</syntaxhighlight>
 +
<div class="cell-output cell-output-stderr">
 +
 
 +
<pre>Warning in leveneTest.default(y = y, group = group, ...): group coerced to
 +
factor.</pre>
 +
 
 +
</div>
 +
<div class="cell-output-display">
 +
 
 +
{| class="wikitable"
 +
|+ Levene’s test on test data
 
|-
 
|-
|align="right"| 0.368
+
! style="text-align: right;"| statistic
|align="right"| 0.558
+
! style="text-align: right;"| p.value
|align="right"| 1
+
! style="text-align: right;"| df
|align="right"| 10
+
! style="text-align: right;"| df.residual
 +
|-
 +
| style="text-align: right;"| 0.368
 +
| style="text-align: right;"| 0.558
 +
| style="text-align: right;"| 1
 +
| style="text-align: right;"| 10
 
|}
 
|}
  
 +
 +
</div>
 +
 +
</div>
 +
<span id="performing-the-appropriate-pairwise-test"></span>
 
== Performing the Appropriate Pairwise Test ==
 
== Performing the Appropriate Pairwise Test ==
  
 
The logic to follow is:
 
The logic to follow is:
  
* If the Shapiro-Wilk test passes, do Levene's test. If it fails for either group, move on to a '''Wilcoxon Rank Sum Test'''.
+
* If the Shapiro-Wilk test passes, do Levene’s test. If it fails for either group, move on to a '''Wilcoxon Rank Sum Test'''.
* If Levene's test ''passes'', do a Student's ''t'' Test, which assumes equal variance.
+
* If Levene’s test ''passes'', do a Student’s ''t'' Test, which assumes equal variance.
* If Levene's test ''fails'', do a Welch's ''t'' Test, which does not assume equal variance.
+
* If Levene’s test ''fails'', do a Welch’s ''t'' Test, which does not assume equal variance.
  
=== Student's ''t'' Test ===
+
<span id="students-t-test"></span>
 +
=== Student’s ''t'' Test ===
  
<pre class="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be TRUE
+
<div class="cell">
t.test(Result~Treatment,data=test.data, var.equal=T) %&gt;% tidy %&gt;% kable</pre>
+
 
{|
+
<syntaxhighlight lang="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be TRUE
!align="right" width="8%"| estimate
+
t.test(Result~Treatment,data=test.data, var.equal=T) %>% tidy %>% kable(caption="Student's t test for test data")</syntaxhighlight>
!align="right" width="9%"| estimate1
+
<div class="cell-output-display">
!align="right" width="9%"| estimate2
+
 
!align="right" width="9%"| statistic
+
{| class="wikitable"
!align="right" width="7%"| p.value
+
|+ Student’s t test for test data
!align="right" width="9%"| parameter
+
!align="right" width="8%"| conf.low
+
!align="right" width="9%"| conf.high
+
!width="16%"| method
+
!width="11%"| alternative
+
 
|-
 
|-
|align="right"| -1.1
+
! style="text-align: right;"| estimate
|align="right"| 8.79
+
! style="text-align: right;"| estimate1
|align="right"| 9.89
+
! style="text-align: right;"| estimate2
|align="right"| -0.992
+
! style="text-align: right;"| statistic
|align="right"| 0.345
+
! style="text-align: right;"| p.value
|align="right"| 10
+
! style="text-align: right;"| parameter
|align="right"| -3.56
+
! style="text-align: right;"| conf.low
|align="right"| 1.37
+
! style="text-align: right;"| conf.high
| Two Sample t-test
+
! style="text-align: left;"| method
| two.sided
+
! style="text-align: left;"| alternative
 +
|-
 +
| style="text-align: right;"| -1.1
 +
| style="text-align: right;"| 8.79
 +
| style="text-align: right;"| 9.89
 +
| style="text-align: right;"| -0.992
 +
| style="text-align: right;"| 0.345
 +
| style="text-align: right;"| 10
 +
| style="text-align: right;"| -3.56
 +
| style="text-align: right;"| 1.37
 +
| style="text-align: left;"| Two Sample t-test
 +
| style="text-align: left;"| two.sided
 
|}
 
|}
  
=== Welch's ''t'' Test ===
 
  
<pre class="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be FALSE, or leave the default
+
</div>
t.test(Result~Treatment,data=test.data, var.equal=F) %&gt;% tidy %&gt;% kable</pre>
+
 
{|
+
</div>
!align="right" width="8%"| estimate
+
<span id="welchs-t-test"></span>
!align="right" width="9%"| estimate1
+
=== Welch’s ''t'' Test ===
!align="right" width="9%"| estimate2
+
 
!align="right" width="9%"| statistic
+
<div class="cell">
!align="right" width="7%"| p.value
+
 
!align="right" width="9%"| parameter
+
<syntaxhighlight lang="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be FALSE, or leave the default
!align="right" width="8%"| conf.low
+
t.test(Result~Treatment,data=test.data, var.equal=F) %>% tidy %>% kable(caption="Welch's t test for test data")</syntaxhighlight>
!align="right" width="9%"| conf.high
+
<div class="cell-output-display">
!width="20%"| method
+
 
!width="10%"| alternative
+
{| class="wikitable"
 +
|+ Welch’s t test for test data
 
|-
 
|-
|align="right"| -1.1
+
! style="text-align: right;"| estimate
|align="right"| 8.79
+
! style="text-align: right;"| estimate1
|align="right"| 9.89
+
! style="text-align: right;"| estimate2
|align="right"| -0.992
+
! style="text-align: right;"| statistic
|align="right"| 0.345
+
! style="text-align: right;"| p.value
|align="right"| 9.72
+
! style="text-align: right;"| parameter
|align="right"| -3.57
+
! style="text-align: right;"| conf.low
|align="right"| 1.38
+
! style="text-align: right;"| conf.high
| Welch Two Sample t-test
+
! style="text-align: left;"| method
| two.sided
+
! style="text-align: left;"| alternative
 +
|-
 +
| style="text-align: right;"| -1.1
 +
| style="text-align: right;"| 8.79
 +
| style="text-align: right;"| 9.89
 +
| style="text-align: right;"| -0.992
 +
| style="text-align: right;"| 0.345
 +
| style="text-align: right;"| 9.72
 +
| style="text-align: right;"| -3.57
 +
| style="text-align: right;"| 1.38
 +
| style="text-align: left;"| Welch Two Sample t-test
 +
| style="text-align: left;"| two.sided
 
|}
 
|}
  
 +
 +
</div>
 +
 +
</div>
 +
<span id="wilcoxon-rank-sum-test"></span>
 
=== Wilcoxon Rank Sum Test ===
 
=== Wilcoxon Rank Sum Test ===
  
<pre class="r"># no need to specify anything about variance
+
<div class="cell">
wilcox.test(Result~Treatment,data=test.data) %&gt;% tidy %&gt;% kable</pre>
+
 
{|
+
<syntaxhighlight lang="r"># no need to specify anything about variance
!align="right"| statistic
+
wilcox.test(Result~Treatment,data=test.data) %>% tidy %>% kable(caption="Mann-Whitney test for test data")</syntaxhighlight>
!align="right"| p.value
+
<div class="cell-output-display">
! method
+
 
! alternative
+
{| class="wikitable"
 +
|+ Mann-Whitney test for test data
 
|-
 
|-
|align="right"| 12
+
! style="text-align: right;"| statistic
|align="right"| 0.394
+
! style="text-align: right;"| p.value
| Wilcoxon rank sum exact test
+
! style="text-align: left;"| method
| two.sided
+
! style="text-align: left;"| alternative
 +
|-
 +
| style="text-align: right;"| 12
 +
| style="text-align: right;"| 0.394
 +
| style="text-align: left;"| Wilcoxon rank sum exact test
 +
| style="text-align: left;"| two.sided
 
|}
 
|}
  
 +
 +
</div>
 +
 +
</div>
 +
<span id="corrections-for-multiple-observations"></span>
 
= Corrections for Multiple Observations =
 
= Corrections for Multiple Observations =
  
 
The best illustration I have seen for the need for multiple observation corrections is this cartoon from XKCD (see http://xkcd.com/882/):
 
The best illustration I have seen for the need for multiple observation corrections is this cartoon from XKCD (see http://xkcd.com/882/):
  
[[File:http://imgs.xkcd.com/comics/significant.png|frame|none|alt=Significance by XKCD. Image is from http://imgs.xkcd.com/comics/significant.png|caption Significance by XKCD. Image is from http://imgs.xkcd.com/comics/significant.png]]
+
<div class="figure">
  
Any conceptually coherent set of observations must therefore be corrected for multiple observations. In most cases, we will use the method of Benjamini and Hochberg since our p-values are not entirely independent. Some sample code for this is here:
+
[[File:http://imgs.xkcd.com/comics/significant.png|Significance by XKCD. Image is from http://imgs.xkcd.com/comics/significant.png]]
  
<pre class="r">p.values &lt;- c(0.023, 0.043, 0.056, 0.421, 0.012)
+
</div>
data.frame(unadjusted = p.values, adjusted=p.adjust(p.values, method=&quot;BH&quot;))</pre>
+
Any conceptually coherent set of observations must therefore be corrected for multiple observations. In most cases, we will use the method of @Benjamini1995 since our p-values are not entirely independent. Some sample code for this is here:
<pre>##  unadjusted adjusted
+
 
## 1      0.023   0.0575
+
<div class="cell">
## 2      0.043   0.0700
+
 
## 3      0.056   0.0700
+
<syntaxhighlight lang="r">p.values <- c(0.023, 0.043, 0.056, 0.421, 0.012)
## 4      0.421   0.4210
+
data.frame(unadjusted = p.values, adjusted=p.adjust(p.values, method="BH")) %>%
## 5      0.012   0.0575</pre>
+
  kable(caption="Effects of adjusting p-values by the method of Benjamini-Hochberg")</syntaxhighlight>
 +
<div class="cell-output-display">
 +
 
 +
{| class="wikitable"
 +
|+ Effects of adjusting p-values by the method of Benjamini-Hochberg
 +
|-
 +
! style="text-align: right;"| unadjusted
 +
! style="text-align: right;"| adjusted
 +
|-
 +
| style="text-align: right;"| 0.023
 +
| style="text-align: right;"| 0.057
 +
|-
 +
| style="text-align: right;"| 0.043
 +
| style="text-align: right;"| 0.070
 +
|-
 +
| style="text-align: right;"| 0.056
 +
| style="text-align: right;"| 0.070
 +
|-
 +
| style="text-align: right;"| 0.421
 +
| style="text-align: right;"| 0.421
 +
|-
 +
| style="text-align: right;"| 0.012
 +
| style="text-align: right;"| 0.057
 +
|}
 +
 
 +
 
 +
</div>
 +
 
 +
</div>
 +
<span id="session-information"></span>
 
= Session Information =
 
= Session Information =
  
<pre class="r">sessionInfo()</pre>
+
<div class="cell">
<pre>## R version 4.4.1 (2024-06-14)
+
 
## Platform: x86_64-apple-darwin20
+
<syntaxhighlight lang="r">sessionInfo()</syntaxhighlight>
## Running under: macOS Monterey 12.7.6
+
<div class="cell-output cell-output-stdout">
##
+
 
## Matrix products: default
+
<pre>R version 4.4.1 (2024-06-14)
## BLAS:  /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRblas.0.dylib  
+
Platform: x86_64-apple-darwin20
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
+
Running under: macOS Sonoma 14.7
##
+
 
## locale:
+
Matrix products: default
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
+
BLAS:  /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRblas.0.dylib  
##
+
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## time zone: America/Detroit
+
 
## tzcode source: internal
+
locale:
##
+
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## attached base packages:
+
 
## [1] stats    graphics  grDevices utils    datasets  methods  base     
+
time zone: America/Detroit
##
+
tzcode source: internal
## other attached packages:
+
 
## [1] car_3.1-2           carData_3.0-5       broom_1.0.6        
+
attached base packages:
## [4] pwr_1.3-0            knitcitations_1.0.12 dplyr_1.1.4        
+
[1] stats    graphics  grDevices utils    datasets  methods  base     
## [7] tidyr_1.3.1         knitr_1.48        
+
 
##
+
other attached packages:
## loaded via a namespace (and not attached):
+
[1] car_3.1-2     carData_3.0-5 broom_1.0.6   dplyr_1.1.4   tidyr_1.3.1
## [1] jsonlite_1.8.8    compiler_4.4.1    tidyselect_1.2.1 Rcpp_1.0.13     
+
[6] knitr_1.48  
## [5] xml2_1.3.6        stringr_1.5.1     jquerylib_0.1.4  yaml_2.3.10     
+
 
## [9] fastmap_1.2.0     R6_2.5.1          plyr_1.8.9        generics_0.1.3    
+
loaded via a namespace (and not attached):
## [13] backports_1.5.0  tibble_3.2.1      RefManageR_1.4.0 lubridate_1.9.3 
+
  [1] vctrs_0.6.5      cli_3.6.3        rlang_1.1.4      xfun_0.47       
## [17] bslib_0.8.0       pillar_1.9.0     rlang_1.1.4       utf8_1.2.4      
+
  [5] purrr_1.0.2      generics_0.1.3    jsonlite_1.8.8    glue_1.7.0     
## [21] stringi_1.8.4    cachem_1.1.0     xfun_0.46        sass_0.4.9     
+
  [9] backports_1.5.0   htmltools_0.5.8.1 fansi_1.0.6      rmarkdown_2.28    
## [25] bibtex_0.5.1     timechange_0.3.0 cli_3.6.3        withr_3.0.0       
+
[13] abind_1.4-8      evaluate_0.24.0  tibble_3.2.1      fastmap_1.2.0  
## [29] magrittr_2.0.3   digest_0.6.36    lifecycle_1.0.4   vctrs_0.6.5     
+
[17] yaml_2.3.10       lifecycle_1.0.4  compiler_4.4.1    htmlwidgets_1.6.4
## [33] evaluate_0.24.0   glue_1.7.0        abind_1.4-5       fansi_1.0.6     
+
[21] pkgconfig_2.0.3  rstudioapi_0.16.0 digest_0.6.37    R6_2.5.1       
## [37] rmarkdown_2.27    purrr_1.0.2      httr_1.4.7        tools_4.4.1       
+
[25] tidyselect_1.2.1  utf8_1.2.4        pillar_1.9.0      magrittr_2.0.3   
## [41] pkgconfig_2.0.3  htmltools_0.5.8.1</pre>
+
[29] withr_3.0.1       tools_4.4.1      </pre>
 +
 
 +
</div>
 +
 
 +
</div>
 +
<span id="references"></span>
 
= References =
 
= References =
  
<a name=bib-pwr></a>[[#cite-pwr|[1]]] S. Champely. ''pwr: Basic Functions for Power Analysis''. R package version 1.3-0. 2020. URL: https://CRAN.R-project.org/package=pwr.
+
<div id="refs">
 +
 
 +
 
 +
 
 +
</div>

Revision as of 00:10, 16 October 2024

<syntaxhighlight lang="r">library(knitr)

  1. figures makde will go to directory called figures, will make them as both png and pdf files

opts_chunk$set(fig.path='figures/',dev=c('png','pdf')) options(scipen = 2, digits = 3)

  1. set echo and message to TRUE if you want to display code blocks and code output respectively

knitr::knit_hooks$set(inline = function(x) {

 knitr:::format_sci(x, 'md')

})


superpose.eb <- function (x, y, ebl, ebu = ebl, length = 0.08, ...)

 arrows(x, y + ebu, x, y - ebl, angle = 90, code = 3,
 length = length, ...)


se <- function(x) sd(x, na.rm=T)/sqrt(length(x))

  1. load these packages, nearly always needed

library(tidyr) library(dplyr)</syntaxhighlight>

Attaching package: 'dplyr'
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

<syntaxhighlight lang="r"># sets maize and blue color scheme color.scheme <- c('#00274c', '#ffcb05')</syntaxhighlight>

General Statistical Methods

There are several important concepts that we will adhere to in our group. These involve design considerations, execution considerations and analysis concerns. The standard for our field is null hypothesis significance testing, which means that we are generally comparing our data to a null hypothesis, generating an effect size and a p-value. As a general rule, we report both of these both within our Rmd/qmd scripts, and in our publications.

We generally use an <math display="inline">\alpha</math> of <math display="inline">p<0.05</math> to determine significance, which means that (if true) we are rejecting the null hypothesis. This is known as null hypothesis significance testing.

An alternative approach is to use a Bayesian approach, described in more detail in this document

Pairwise Testing

If you have two groups (and two groups only) that you want to know if they are different, you will normally want to do a pairwise test. This is not the case if you have paired data (before and after for example). The most common of these is something called a Student’s t-test, but this test has two key assumptions:

  • The data are normally distributed
  • The two groups have equal variance

Testing the Assumptions

Best practice is to first test for normality, and if that test passes, to then test for equal variance

Testing Normality

To test normality, we use a Shapiro-Wilk test (details on Wikipedia on each of your two groups). Below is an example where there are two groups:

<syntaxhighlight lang="r">#create seed for reproducibility set.seed(1265) test.data <- tibble(Treatment=c(rep("Experiment",6), rep("Control",6)),

          Result = rnorm(n=12, mean=10, sd=3))
  1. test.data$Treatment <- as.factor(test.data$Treatment)

kable(test.data, caption="The test data used in the following examples")</syntaxhighlight>

The test data used in the following examples
Treatment Result
Experiment 11.26
Experiment 8.33
Experiment 9.94
Experiment 11.83
Experiment 6.56
Experiment 11.41
Control 8.89
Control 11.59
Control 9.39
Control 8.74
Control 6.31
Control 7.82


Each of the two groups, in this case Test and Control must have Shapiro-Wilk tests done separately. Some sample code for this is below (requires dplyr to be loaded):

<syntaxhighlight lang="r">#filter only for the control data control.data <- filter(test.data, Treatment=="Control")

  1. The broom package makes the results of the test appear in a table, with the tidy command

library(broom)

  1. run the Shapiro-Wilk test on the values

shapiro.test(control.data$Result) %>% tidy %>% kable(caption="Shapiro-Wilk test for normality of control data")</syntaxhighlight>

Shapiro-Wilk test for normality of control data
statistic p.value method
0.968 0.88 Shapiro-Wilk normality test


<syntaxhighlight lang="r">experiment.data <- filter(test.data, Treatment=="Experiment") shapiro.test(test.data$Result) %>% tidy %>% kable(caption="Shapiro-Wilk test for normality of the test data")</syntaxhighlight>

Shapiro-Wilk test for normality of the test data
statistic p.value method
0.93 0.377 Shapiro-Wilk normality test


Based on these results, since both p-values are >0.05 we do not reject the presumption of normality and can go on. If one or more of the p-values were less than 0.05 we would then use a Mann-Whitney test (also known as a Wilcoxon rank sum test) will be done, see below for more details.

Testing for Equal Variance

We generally use the car package which contains code for Levene’s Test to see if two groups can be assumed to have equal variance. For more details see @car:

<syntaxhighlight lang="r">#load the car package library(car)</syntaxhighlight>

Loading required package: carData
Attaching package: 'car'
The following object is masked from 'package:dplyr':

    recode

<syntaxhighlight lang="r">#runs the test, grouping by the Treatment variable leveneTest(Result ~ Treatment, data=test.data) %>% tidy %>% kable(caption="Levene's test on test data")</syntaxhighlight>

Warning in leveneTest.default(y = y, group = group, ...): group coerced to
factor.
Levene’s test on test data
statistic p.value df df.residual
0.368 0.558 1 10


Performing the Appropriate Pairwise Test

The logic to follow is:

  • If the Shapiro-Wilk test passes, do Levene’s test. If it fails for either group, move on to a Wilcoxon Rank Sum Test.
  • If Levene’s test passes, do a Student’s t Test, which assumes equal variance.
  • If Levene’s test fails, do a Welch’s t Test, which does not assume equal variance.

Student’s t Test

<syntaxhighlight lang="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be TRUE t.test(Result~Treatment,data=test.data, var.equal=T) %>% tidy %>% kable(caption="Student's t test for test data")</syntaxhighlight>

Student’s t test for test data
estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high method alternative
-1.1 8.79 9.89 -0.992 0.345 10 -3.56 1.37 Two Sample t-test two.sided


Welch’s t Test

<syntaxhighlight lang="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be FALSE, or leave the default t.test(Result~Treatment,data=test.data, var.equal=F) %>% tidy %>% kable(caption="Welch's t test for test data")</syntaxhighlight>

Welch’s t test for test data
estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high method alternative
-1.1 8.79 9.89 -0.992 0.345 9.72 -3.57 1.38 Welch Two Sample t-test two.sided


Wilcoxon Rank Sum Test

<syntaxhighlight lang="r"># no need to specify anything about variance wilcox.test(Result~Treatment,data=test.data) %>% tidy %>% kable(caption="Mann-Whitney test for test data")</syntaxhighlight>

Mann-Whitney test for test data
statistic p.value method alternative
12 0.394 Wilcoxon rank sum exact test two.sided


Corrections for Multiple Observations

The best illustration I have seen for the need for multiple observation corrections is this cartoon from XKCD (see http://xkcd.com/882/):

Any conceptually coherent set of observations must therefore be corrected for multiple observations. In most cases, we will use the method of @Benjamini1995 since our p-values are not entirely independent. Some sample code for this is here:

<syntaxhighlight lang="r">p.values <- c(0.023, 0.043, 0.056, 0.421, 0.012) data.frame(unadjusted = p.values, adjusted=p.adjust(p.values, method="BH")) %>%

 kable(caption="Effects of adjusting p-values by the method of Benjamini-Hochberg")</syntaxhighlight>
Effects of adjusting p-values by the method of Benjamini-Hochberg
unadjusted adjusted
0.023 0.057
0.043 0.070
0.056 0.070
0.421 0.421
0.012 0.057


Session Information

<syntaxhighlight lang="r">sessionInfo()</syntaxhighlight>

R version 4.4.1 (2024-06-14)
Platform: x86_64-apple-darwin20
Running under: macOS Sonoma 14.7

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Detroit
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] car_3.1-2     carData_3.0-5 broom_1.0.6   dplyr_1.1.4   tidyr_1.3.1  
[6] knitr_1.48   

loaded via a namespace (and not attached):
 [1] vctrs_0.6.5       cli_3.6.3         rlang_1.1.4       xfun_0.47        
 [5] purrr_1.0.2       generics_0.1.3    jsonlite_1.8.8    glue_1.7.0       
 [9] backports_1.5.0   htmltools_0.5.8.1 fansi_1.0.6       rmarkdown_2.28   
[13] abind_1.4-8       evaluate_0.24.0   tibble_3.2.1      fastmap_1.2.0    
[17] yaml_2.3.10       lifecycle_1.0.4   compiler_4.4.1    htmlwidgets_1.6.4
[21] pkgconfig_2.0.3   rstudioapi_0.16.0 digest_0.6.37     R6_2.5.1         
[25] tidyselect_1.2.1  utf8_1.2.4        pillar_1.9.0      magrittr_2.0.3   
[29] withr_3.0.1       tools_4.4.1      

References