Introduction to Inferential Statistics
Example 1: NJ happiness
Suppose you conduct a survey of adults in New Jersey and ask each respondent to rate their happiness on a scale from 0 to 100. The following map shows the average happiness score for each county. What conclusions can you draw?
What’s the truth
Any pattern you see is actually due to random chance. The happiness score is randomly generated for each of the 21 counties using the same underlying distribution (rnorm(21,mean=60,sd=15)).
Example 2: US crime rates
This map is based on the USArrests dataset. What conclusions can you draw?
Murder Assault UrbanPop Rape
Alabama 13.2 236 58 21.2
Alaska 10.0 263 48 44.5
Arizona 8.1 294 80 31.0
Arkansas 8.8 190 50 19.5
California 9.0 276 91 40.6
Colorado 7.9 204 78 38.7
What’s the truth
Unknown. Any pattern you see can be due to random chance, or can be statistically significant.
Pattern you see in the data can be due to random chance!
sample(c('Success','Fail'),size=50,replace=T) [1] "Fail" "Fail" "Fail" "Success" "Success" "Fail" "Success"
[8] "Fail" "Fail" "Success" "Success" "Success" "Fail" "Fail"
[15] "Success" "Fail" "Fail" "Fail" "Success" "Fail" "Success"
[22] "Fail" "Fail" "Success" "Fail" "Success" "Success" "Success"
[29] "Success" "Success" "Fail" "Fail" "Fail" "Fail" "Success"
[36] "Success" "Success" "Success" "Fail" "Success" "Success" "Fail"
[43] "Success" "Fail" "Fail" "Fail" "Success" "Success" "Success"
[50] "Fail"
We see streaks of Fails and Successes, but these are all due to random chance!
Inferential statistics
We have been working with many different data and can describe the data in many ways (summary statistics, visualizations, correlations, etc.). However, data does NOT equal to the truth!
In any survey data: respondents are only a random sample of the population. If we see that, for instance, 23% of respondents in the
child.caredataset used childcare, it does not mean that exactly 23% of the entire U.S. population used childcare.In an NBA example: a player’s true ability (of shooting field goals, for instance) is a latent variable that cannot be directly measured. For instance, in the 2024 NBA season, Mikal Bridges has a higher field goal percentage than Jalen Green, but it doesn’t necessarily mean that Mikal Bridges is better than Jalen Green; maybe he is just lucky (i.e., his success is “due to random chance”).
library(Lock5Data)
NBAPlayers2024 %>%
filter(Starts > 80, MinPerGame > 30 ) %>%
select(Player, Pos, Team, Games, FGAttempt, FGMade, FGPct) %>%
arrange(desc(Team)) Player Pos Team Games FGAttempt FGMade FGPct
1 Domantas Sabonis C SAC 82 1068 634 0.594
2 Jalen Green SG HOU 82 1332 563 0.423
3 Michael Porter Jr. SF DEN 81 1079 522 0.484
4 Mikal Bridges SF BRK 82 1294 564 0.436
- Another way to think about this: most of the time, data is not reproducible: if I conduct the same survey again with different respondents, I will get a different percentage of respondents using childcare; Mikal Bridges is better than Jalen Green in 2024, but it is not a guarantee that he will also be better in 2025.
p-value and statistical significance
Descriptive statistics, also referred to as Exploratory Data Analysis (EDA), is about describing the data
Inferential statistics is about generalizing the conclusion to the entire population and/or the truth
Typically, a “hypothesis test” is conducted and a p-value is obtained
If p-value is large (>0.05): the conclusion is not significant and due to random chance
If p-value is small (<0.05): the conclusion is statistically significant
Definition of p-value: the probability of observing the data (i.e., test statistic) as extreme, or more extreme, than the one obtained, assuming the null hypothesis is true.
Example 3: drug effect on tumor size
Suppose a new drug is developed to treat a certain type of cancer and has been tested on 20 patients. Is the drug effective?
tumsize1 tumsize2 patient
1 15.454042 6.926204 patient 1
2 7.286632 14.213477 patient 2
3 11.628681 11.079898 patient 3
4 14.428575 10.482909 patient 4
5 9.524580 9.125362 patient 5
6 11.471811 9.149994 patient 6
7 9.493480 9.774322 patient 7
8 5.206460 10.899869 patient 8
9 10.554134 10.676206 patient 9
10 16.671018 9.649528 patient 10
11 15.437937 8.548358 patient 11
12 10.478542 10.441172 patient 12
13 14.385787 10.130264 patient 13
14 10.809951 9.408377 patient 14
15 7.876401 12.626095 patient 15
16 12.141398 8.267080 patient 16
17 11.616720 6.881314 patient 17
18 9.512484 8.905779 patient 18
19 11.369541 14.949419 patient 19
20 12.846685 9.973217 patient 20
Remember, the data alone does not tell us whether the drug is effective; we need to evaluate the statistical significance. We decide to conduct a paired samples t-test. What is the conclusion here?
t.test(df.drug$tumsize1, df.drug$tumsize2, paired = TRUE)
Paired t-test
data: df.drug$tumsize1 and df.drug$tumsize2
t = 1.3812, df = 19, p-value = 0.1833
alternative hypothesis: true mean difference is not equal to 0
95 percent confidence interval:
-0.6722583 3.2808597
sample estimates:
mean difference
1.304301
Solution
The p-value is large (0.1833 > 0.05), so we conclude that the drug is not statistically significant in reducing tumor size.
What’s the truth
Any pattern you see is indeed due to random chance. The tumor sizes before and after taking the drug are both randomly, and independently generated for each patient using the same underlying distribution (rgamma(20,shape=20,rate=2)).
Intuition behind p-value: a thought experiment
Suppose your data looks like this: Matt, he is wearing a blue t-shirt, a pair of jeans, some sneakers, sunglasses, and a baseball cap. He is holding his phone and listening to music on his headphones.
Question: which scenario is more likely?
- A: Matt is about to run a race.
- B: Matt is not about to run a race.
Key idea
IF Matt is about to run a race, what should he look like? In other words, IF scenario A is true, what should your data look like? This is called a reference distribution.
Reference distribution
If your data is very “unusual” or “extreme” compared to the reference distribution, you would reject scenario A on the basis of your data.
If not, i.e., your data looks pretty “likely” under the reference distribution, you would fail to reject scenario A.
How likely is your data under the reference distribution? This is essentially what the p-value measures! Hence: small p-value -> unlikely data -> reject the null hypothesis (scenario A).
Question: if you decide scenario B is more likely, did you prove with certainty that Matt is not about to run a race?
Key idea
No, you cannot prove that Matt is not about to run a race. You are just saying that the evidence is very strong against scenario A, meaning that your data is very unlikely under scenario A (i.e., it’s unlikely that Matt would wear jeans to a race).
Example 1: NJ happiness revisited
Consider two scenarios:
- A: every county in NJ is equally happy, on average.
- B: Morris county is more happy than all of the other counties.
You conduct your survey and look at your data (e.g., generate a map that looks like example 1), but how do you decide between scenario A and B?
Key idea: if scenario A is true, what should your data look like?
Reference distribution under scenario A
Now, if your data looks like this, would you reject scenario A?
What if your data looks like this?
Example 3: drug effect revisited
In the drug effect example, we can summarize the data by calculating the mean difference between the tumor sizes before and after treatment.
df.drug$tumsize1 - df.drug$tumsize2 [1] -6.92684415 -5.69340873 -4.74969350 -3.57987817 -0.28084238 -0.12207213
[7] 0.03736942 0.39921770 0.54878326 0.60670441 1.40157402 2.32181729
[13] 2.87346764 3.87431781 3.94566519 4.25552209 4.73540607 6.88957946
[19] 7.02148980 8.52783842
mean(df.drug$tumsize1 - df.drug$tumsize2)[1] 1.304301
We need to decide between:
- Scenario A: The drug has no effect (the null hypothesis)
- Scenario B: The drug has an effect (the alternative hypothesis)
Reference distribution: what should the mean difference look like if the drug has no effect?
(This reference distribution is a t-distribution, that’s why it’s called a paired samples t-test. The derivation is based on the Central Limit Theorem.)
Is our observed value (1.3) very extreme? What would you conclude?