Lecture 8: Empirical p-values

DATA 352/551: Modeling and Simulation

Streakiness

Project one: fencing

Project two: game of n!àì

Streak

  • Try: run the sample code; enter a sequence of binary data and report your \(B_{10}\) value; is your data considered streaky?

Data can be due to random chance

\(B_{10}\) is a statistic (i.e., it’s calculated based on data). We need to evaluate its statistical significance by looking at p-values or confidence intervals.

  • If I randomly generate a data sequence (not streaky), is it possible that I get a \(B_{10}\) value greater than 1 (streaky) just by luck?

  • Idea: Say \(B_{10} = 1.35\) for my actual data. I will randomly sample none-streaky data and get their \(B_{10}\) values to see if \(1.35\) is considered extreme.

Exercise

Write code to randomly sample none-streaky data (consider randomly flipping a coin)

  • My actual data and its \(B_{10}\) value:
mydata
 [1] 1 1 1 1 0 1 0 1 0 1 0 0 0 0 1 1
f.B10(mydata)
[1] 1.345708
  • Randomly sample none-streaky data:
newdata <- sample(c(0,1),
                 size=              ,
                 replace=T,
                 prob =                      )
f.B10(newdata)

Hypothesis testing

  • Null hypothesis \(H_0\): data is not streaky
  • Alternative hypothesis \(H_1\): data is streaky
  • Test statistic: \(B_{10}\)
  • Reference distribution: If the null hypothesis is true (i.e., NOT streaky), what is the distribution of the test statistic \(B_{10}\)?

Exercise: Run a simulation (1000 times) to estimate the reference distribution: i.e., repeat the previous step 1000 times and record the \(B_{10}\) value each time.

Empirical p-value

  • p-value: probability that the test statistic is as extreme, or more extreme, than the observed value, under the null hypothesis (“extreme” in the sense of pointing to the alternative hypothesis).
  • The empirical p-value is calculated as:
(sum(result>=f.B10(mydata)) + 1)/(nsim + 1)
[1] 0.1978022

Lady tasting tea

Introduction

A classic example in statistics, due to R. A. Fisher: lady tasting tea

  • T: adding tea first; M: adding milk first
  • A lady claims that she can tell the difference by tasting
  • 8 cups are prepared: T T T T M M M M
  • The lady is asked to pick out 4 that she believes was made by adding tea first
Out of the 4 she picks, how many needs to be correct for you to believe that she actually has the ability to tell the difference?

Hypothesis test

  • Null hypothesis \(H_0\): The lady is just guessing (i.e., cannot tell the difference)
  • Alternative hypothesis \(H_1\): The lady can tell the difference
  • Test statistic: X = the number she gets correct (Q: what’s the range of X?)
  • Reference distribution: If the null hypothesis is true (i.e., she is only guessing), what is the distribution of the test statistic X?

Simulation

tea <- c('T','T','T','T','M','M','M','M')
choice <- sample(tea,size=4,replace=F); choice
[1] "M" "M" "T" "T"
nsim<-10000
result <- rep(NA,nsim)
for(i in 1:nsim){
  choice<-sample(tea,size=4,replace=F)
  result[i]<-sum(choice=='T')
}
table(result)
result
   0    1    2    3    4 
 157 2316 5071 2329  127 
  • If she guessed 4 out of 4 correct, the p-value is:
  • If she guessed 3 out of 4 correct, the p-value is:

Discussions

  • In the “lady tasting tea” and SES examples (see homework 8), the reference distribution is easy to derive mathematically. Simulation is more useful with difficult test statistics (like \(B_{10}\)).

  • For the lady tasting tea example, if she gets 4 out of 4 correct, we would reject the null and conclude that “we have enough evidence that she is not guessing.” This DOES NOT prove that she is definitely not guessing. All we are saying is that “it’s difficult to get 4 out of 4 correct just by random chance.”

  • …if she gets 3 out of 4 correct, we would “accept” the null and conclude that “she is guessing.” This DOES NOT prove that she is definitely guessing. All we are saying is that “it’s very possible to get 3 out of 4 correct just by random chance.”

  • Monte Carlo methods refer to a very broad class of methods that rely on using random sampling to solve otherwise difficult problems (example: bootstrapping, Monte Carlo integration, empirical p-value).