Randomness & Statistical Inference

What a random sample can, and cannot, tell us

Merlin Schaeffer · Department of Sociology

2026-10-26

By the end of today you can …

1. explain why we rely on random samples, and how weights repair imperfect randomness;

2. see that every sample carries sampling error, and estimate it with a standard error;

3. use confidence intervals and hypothesis tests to say what a sample can, and cannot, generalise.

Random samples & weights

Part 1 of 3

All our data are samples. To generalise from them, we use one idea: randomness.

The research question of the day

Do better-educated people generally believe more strongly that the political system is responsive, that people have a say?

Political efficacy: the belief that one’s voice counts in politics.

Discuss: we only ever have data on some people, in some year. What has to be true about those people for us to trust a general answer?

Pupils protest the school reform outside Christiansborg, May 2016. Did it change anything? Photo: Ole Ryhl Olsson, CC BY-SA 4.0.

The photo could be them a few years ago: pupils on Slotspladsen against the 2014 school reform. Ask who has ever protested, signed something or voted, and whether it felt like it counted. That feeling is our outcome. Note the word “generally” in the question — that single word is what makes today necessary. We have 1,500-odd Danes from one year, and we want to say something about people in general. Let them answer the blue box; whatever they say will be some version of “the sample has to be like everyone else”, which is precisely Part 1.

Preparation

• Packages for today’s session
• Get the ESS data

pacman::p_load(
  tidyverse,    # Data manipulation and visualization
  haven,        # Read SPSS/Stata files & handle labelled data
  estimatr,     # OLS with robust standard errors (for weights)
  modelsummary  # Nicely formatted regression tables
)

Download ESS9e03_1.sav from Absalon into your course project folder, then:

ESS <- read_spss("ESS9e03_1.sav") %>%
  filter(cntry == "DK") %>% # Keep only the Danish respondents
  # A minimal set of variables for today
  select(idno, pspwght, gndr, eduyrs, agea,
         psppsgva, trstlgl, trstplc) %>%
  mutate(psppsgva = zap_labels(psppsgva), # Make numeric
         trstlgl = zap_labels(trstlgl),   # Make numeric
         trstplc = zap_labels(trstplc),   # Make numeric
         pspwght = zap_labels(pspwght),   # Make numeric
         eduyrs = case_when(              # Censor education at 9 and 21 years
           eduyrs > 21 ~ 21,
           eduyrs < 9 ~ 9,
           TRUE ~ as.numeric(eduyrs)),
         gndr = as_factor(gndr)) %>%
  drop_na() # Drop cases with missing values

Why do we study samples at all?

(1) Populations are too large to survey everyone.

(2) Even with data on everyone, we care about general patterns and social mechanisms, not just this year’s people in DK or today’s countries.

So we treat any data set (even a full register) as one sample drawn from an unobservable “super-population”.

Reason two is the surprising one, so spend the time there. Even with Danish register data on everybody, we would still be doing inference — because we care about the social mechanism, not about these particular Danes in this particular year. That is why the super-population idea matters and why “but I have the whole population” is not an escape from statistics.

Source: Wikipedia

Convenience samples mislead

A good sample is representative, it mirrors the super-population.

Discuss: you snowball-sample starting from your best friend. What bias creeps in, and who gets left out?

Your friends resemble you (age, education, city, politics). Whole groups are systematically missing; the sample is biased before you compute anything.

Make it personal — ask them who their snowball sample would actually reach, and who it would never touch. The key phrase for the reveal: the sample is biased before you compute anything. No amount of clever analysis repairs a sample that never contained the people you care about.

Randomness = fairness

Why do so many games use dice, coins, shuffled cards?

Fairness: a random draw gives everyone the same chance, regardless of who they are. No player, and no social group, can be systematically forgotten.

A random sample is the equivalent of stirring the soup before you taste it: one spoonful then represents the whole pot.

The soup metaphor is the one they will remember, so say it twice: stir the pot, and one spoonful tells you about the whole thing — and note that the size of the spoon matters far less than the stirring. Same logic as the coin in a fair game: randomness is not about being unpredictable, it is about being fair to every group.

Soup: Michael W. Kolton, CC BY-SA 3.0.

But participation is not random

Who answers a survey is never a perfect coin flip:
some groups are harder to reach, or more willing to respond.

We repair this after the fact with post-stratification weights.

Denmark, 2015. The last polls put Dansk Folkeparti at ~18% — it won 21.1%. On the other side, SF and Enhedslisten polled about a point above what they got.

Part of that gap is late deciders, not only refusals, but either way: who ends up in the data is not random.

Cartoon: Adam Zyglis, The Buffalo News / Cagle Cartoons

Start with the cartoon: the people who most distrust researchers are exactly the ones who hang up on them — so they are missing from the data, and they are not a random slice. Then reveal the Danish numbers. Dansk Folkeparti was underestimated by more than three points in 2015; Voxmeter’s own explanation was that new DF voters would not say who they backed. And it cuts both ways: SF and Enhedslisten polled above their result. Be honest that late deciders are part of the story too. This is the motivation for weights on the next slides.

What is a weight?

A weight reflects how much a case should count, to undo over- or under-sampling.

In a perfect random sample every case has weight 1. If older people are over-represented, we give them a weight below 1; under-represented groups get a weight above 1.

Discuss: the plot shows the ESS weights by age. Why might survey researchers down-weight older respondents here?

Define it in one line: a weight is how many people in the population this respondent stands for. Then the blue box — older people are easier to reach and more willing to answer, so they are over-represented and get weighted down. Ask what would happen to our education result if the over-represented group also differed in education; that is the whole reason weights exist.

How weights work: a worked example

Imagine a patriarchal ballot on women’s driving rights, where a man’s vote counts double a woman’s.

Gender	Vote	Voted yes	Weight
man	No	0	2
man	Yes	1	2
woman	Yes	1	1
woman	Yes	1	1
woman	No	0	1

Discuss: ignoring weights, what share voted “Yes”?

Three equivalent ways to get the weighted share of “Yes”:

• By hand
• With vectors
• Built-in

# (yes × weight) summed, over total weight
((0*2 + 1*2 + 1 + 1 + 0*1) / (2 + 2 + 1 + 1 + 1)) * 100
# [1] 57.1

yes    <- c(0, 1, 1, 1, 0)
weight <- c(2, 2, 1, 1, 1)
sum(yes * weight) / sum(weight) * 100
# [1] 57.1

weighted.mean(x = yes, w = weight) * 100
# [1] 57.1

Get the unweighted answer out of them first — 60% voted yes — then run the weighted version and watch it drop. The deliberately unpleasant framing (a man’s vote counting double) makes the mechanics obvious: weighting is just counting some rows more than others. Three routes to the same number, so nobody has to take weighted.mean() on faith.

Weights can be powerful

With the right weights, even a wildly non-representative sample can forecast well.

Wang, Rothschild, Goel, and Gelman (2015) predicted the 2012 US election from Xbox gamers (young, male, unrepresentative) reweighted to the electorate.

Source: Wang, Rothschild, Goel et al. (2015)

A genuinely fun result — Xbox gamers are about as unrepresentative of the US electorate as a sample can get, young and overwhelmingly male, and reweighted they still called the 2012 election well. Use it to make the honest point and its limit: weights repair imbalance on characteristics you can measure and know the population totals for. They cannot conjure back people the sample never reached at all.

Using weights in regression

Most R functions take a weights argument. For weighted OLS use estimatr::lm_robust().

Weights make the residuals heteroscedastic, which breaks a standard OLS assumption. lm_robust() fixes the standard errors for us.

• Table
• R code

	Pol. Efficacy (unweighted)	Pol. Efficacy (weighted)
(Intercept)	2.199	2.378
eduyrs	0.039	0.025
Num.Obs.	1511	1511
R2	0.026	0.010

mod_unw <- lm_robust(psppsgva ~ eduyrs, data = ESS)
mod_wt  <- lm_robust(psppsgva ~ eduyrs, data = ESS, weights = pspwght)

modelsummary(
  list("Pol. Efficacy \n (unweighted)" = mod_unw, 
       "Pol. Efficacy \n (weighted)" = mod_wt),
  statistic = NULL,
  gof_map = c("nobs", "r.squared"),
  output = "kableExtra"
)

Practical takeaway: add weights = and switch to lm_robust(). Explain why the switch is needed — weighting makes the residuals heteroscedastic, which breaks a standard OLS assumption, and lm_robust() repairs the standard errors. Compare the two columns: the coefficients move a little, which is exactly the correction doing its job.

Sampling error & the standard error

Part 2 of 3

Randomness removes bias, but a random sample is still noisy. How noisy, exactly?

A thought experiment: pretend we see everyone

Suppose the Danish ESS (1511 people) were our whole super-population.

Then we could calculate the true slope of political efficacy on education (blue line): \[\beta = 0.039\]

No weights here; this is a make-believe population.

Set the fiction up carefully, because the next six slides depend on it: pretend these 1,500 Danes are the entire world. Under that pretence we can compute the true β — the blue line — which we never actually get in real research. Keep pointing back at the blue line; it is the fixed truth everything else will scatter around.

Now draw one small sample

We rarely see the population, usually just a sample. Take 50 people at random:

set.seed(1261990) # Make randomness reproducible
ESS_sample <- ESS %>% sample_n(50) # Sample 50 cases at random

The sample’s own regression line (red) gives \(\hat\beta = 0.042\); not the true 0.039.

Another sample, another answer

Draw a different 50 people and the line moves again: \(\hat\beta = -0.008\).

Beware: each sample gives a different estimate. This scatter of estimates around the truth is sampling error; not a mistake, but the price of randomness and seeing only a sample.

Nothing was done wrong, and that is the lesson — same population, same code, different 50 people, different answer. Insist on the vocabulary: this is sampling error, not a mistake. It is the unavoidable price of not seeing everyone, and the rest of the lecture is about measuring it rather than eliminating it.

… and again, and again

Keep drawing fresh samples of 50. Every red line is a new random sample; the blue line is the unchanging truth.

The lines fan out around the true line. That fan is the sampling error: in this thought experiment visible, but normally (as we’ll see) predictable.

Let the animation run twice and just narrate it — every red line is one researcher’s honest study. Two things to point out: they fan out around the blue truth rather than drifting off it, and the fan has a definite width. Both matter: unbiased, but noisy, and the noise is regular enough to be measured.

Do it 1000 times: the sampling distribution

• The idea
• Watch it build
• Histogram
• … its R code
• It’s Normal!
• Small samples: the t

Repeat “draw 50, estimate \(\hat\beta\)” 1000 times and collect all the estimates. Their spread is the sampling error; and it has a beautiful, predictable shape.

set.seed(1261990)
sample_betas <- c() # An empty vector to collect the 1000 slopes
for (draw in 1:1000) {                     # 1000 repetitions of …
  s <- ESS %>% sample_n(50)                # … draw 50 people
  sample_betas[draw] <- coef(lm(psppsgva ~ eduyrs, data = s))["eduyrs"] # … store the slope
}
sd(sample_betas) # The true sampling error
# [1] 0.0367

Each dot is one sample’s slope. As they pile up, a bell curve emerges around the true \(\beta\).

true_beta <- coef(lm(psppsgva ~ eduyrs, data = ESS))["eduyrs"] %>% unname()
sd_b <- sd(sample_betas)

ggplot(tibble(b = sample_betas), aes(x = b)) + # One row per sample's slope
  geom_histogram(binwidth = 0.008, fill = "#901A1E", alpha = 0.85) +
  geom_vline(xintercept = true_beta, color = "#425570", linewidth = 1.2) +
  geom_vline(xintercept = true_beta + c(-1.96, 1.96) * sd_b,
             color = "#b5892c", linetype = "longdash", linewidth = 1) +
  labs(x = expression("1000 sample estimates of "*hat(beta)),
       y = "Count",
       caption = "Blue = true β · gold dashed = ±1.96 × sampling error")

The estimates pile up in a Normal (bell) curve around the truth. We know its areas exactly:

• \(\pm 1.65 \times\) SD covers the middle 90%
• \(\pm 1.96 \times\) SD covers the middle 95%

Those multipliers, 1.65 and 1.96, are the critical values: how far out from the centre you must go, counted in standard deviations, to cover a given share of the curve. Want more coverage, walk further out.

We will soon multiply a standard error by a critical value to build confidence intervals and tests.

Source: Veaux, Velleman, and Bock (2021)

With small \(n\), the curve has slightly fatter tails: the \(t\)-distribution. Its critical value depends on the degrees of freedom (\(n\) − parameters):

# 95% critical value, df = 50 − 2
qt(p = 0.975, df = 48)
# [1] 2.01

Source: Wikipedia

The crux of the lecture — go slowly and use the “watch it build” tab. Each dot is one whole study; say that out loud, because students routinely think a dot is a person. As the dots pile up a bell curve appears around the truth, and once we know it is Normal we know its areas exactly: ±1.96 SD covers the middle 95%. The \(t\) tab is a footnote — same idea, slightly fatter tails when \(n\) is small.

Gauss’s gift

Discuss: in real research we get one sample. We can’t draw 1000. So how could we possibly know the sampling error?

Thanks to Carl Friedrich Gauss, we can estimate the sampling error from a single sample. That estimate is the standard error \(\hat\sigma\).

Carl Friedrich Gauss · 1777 – 1855

For an OLS slope: \[\hat{\sigma}(\hat\beta) = \sqrt{\frac{1}{n-2}\,\frac{\text{SD}(e)}{\text{SD}(x)}}\]

	Pol. Efficacy (50-person sample)
(Intercept)	1.965
	(0.530)
eduyrs	0.042
	(0.036)
Num.Obs.	50
R2	0.027

The estimated SE from this one sample (0.036) is remarkably close to the true sampling error from 1000 samples (0.037).

Ask the blue-box question and let it land as a real problem: everything so far needed 1000 samples, and in real research we get one. Then the payoff — Gauss showed the spread of that 1000-sample distribution can be estimated from a single sample. Put the two numbers side by side and let them see how close they are. That is what a standard error is, and it is why the SE column in every table they have ever seen is not magic.

Your turn: samples & weights

Open exercise 1 in a new tab ↗

Break

Confidence intervals & hypothesis tests

Part 3 of 3

The standard error turns a single estimate into an honest statement of uncertainty.

95% confidence intervals

Combine the estimate with its standard error:

\[\hat\beta \pm \text{critical value} \times \hat\sigma\]

What “95%” means: if we drew samples over and over and built this interval each time, 95% of those intervals would contain the true \(\beta\) (the blue line). It is a statement about the procedure, not any single interval.

Watch the intervals accumulate: only about 1 in 20 (the red ones) misses the truth.

Be pedantic here, because the wrong version is everywhere: 95% is a property of the procedure, not of the one interval in front of you. Run the animation and count the red misses — about 1 in 20, exactly as advertised. The sentence to forbid: “there is a 95% chance the true value is in this interval.” The true β is fixed; it is our intervals that jump around.

Confidence intervals, hands-on

Play with samples, coverage, and interval width yourself:

Uncertainty, drawn

geom_smooth(method = "lm") shades the 95% CI around the line.

ggplot(ESS_sample, aes(y = psppsgva, x = eduyrs)) +
  geom_jitter(alpha = 1/3,
              width = 0.1, height = 0.1) +
  geom_smooth(method = "lm", color = "#901A1E") + # 95% CI band
  ylim(1, 5) +
  labs(y = "Political efficacy",
       x = "Years of education")

The band is exactly the region where those sampled regression lines plausibly lie. Watch them fall inside it.

This connects the grey band they have been drawing since Lecture 2 to everything we did today. The animation is the point: the accumulating sample lines are the same red lines from earlier, and the band is where they live. Also note the shape — wide where the data are thin, narrow where they are dense, which is just sampling error depending on how much information sits at each x.

Hypothesis tests: argue with an opponent

The debate of Socrates and Aspasia

Research is a debate with a sceptic who dismisses your finding:

“Sure, your educated respondents scored higher. But that’s just a fluke of your 5-person sample. Really there’s no difference at all.”

\[\underbrace{H_{0}}_{\text{the } H_{\text{opponent}}}:\ \beta_{\text{education}} = 0\]

We test whether the data can overturn this null.

Frame testing as an argument rather than a ritual: there is a sceptic in the room who says your finding is just a fluke of your sample. \(H_0\) is their position, not yours. Everything that follows is the machinery for deciding whether the data embarrass them. Students who see the null as an opponent stop mixing up which hypothesis is being tested.

The (weird) logic of a test

• \(H_0\): the sun is shining.

• I see 5 kids walking to school with umbrellas. How likely is that if the sun is really shining?

• If that probability is below 5%, I stop believing \(H_0\), and bet it’s raining.

Discuss: can you phrase your own everyday \(H_0\)-and-evidence example?

We never prove our \(H_1\)! We show the sceptic’s \(H_0\) makes the data too surprising to keep believing in \(H_0\).

The umbrella example is worth acting out. Notice the direction of reasoning — we never prove it is raining; we show that the umbrellas would be too surprising if the sun were shining. Say the last line twice, because it is the sentence they get wrong in exams: a significant result is evidence against \(H_0\), never proof of \(H_1\).

A test, step by step

1 The sceptic’s claim: \(H_0: \beta = 0\).

2 The ruler: our sample’s standard error, \(\hat\sigma = 0.043\).

3 The sampling distribution: If \(H_0\) were true, samples of 50 would scatter around 0 like this (\(t\)-distribution, df = 48).

4 The test interval: 95% of those chance estimates land within \(0 \pm 2.01 \times \hat\sigma\).

5 The \(t\)-value: Our estimate** \(\hat\beta = 0.042\) is \(t = 0.98\) standard errors from 0; here inside the test interval.

6 The \(p\)-value: 67% of chance estimates are closer to 0 than ours. And the other \(p = 0.33\) are at least as far out → the probability \(p\) to draw a sample like ours if $H_0# is true, is \(p = 0.33\). Not surprising → we can’t reject \(H_0\).

Click through slowly — one idea per click, and each picture only adds to the last. (1) Start from the sceptic’s claim, not ours. (2) The standard error is the ruler: it tells us how far estimates stray by chance alone. (3) Put the two together and we know exactly how estimates would scatter if the sceptic were right — the same bell curve as the 1000 samples, only now centred on 0. (4) 95% of those chance estimates fall in the gold interval. (5) Only now bring in our estimate: about one standard error out, comfortably inside. (6–7) The blue share is closer to 0 than ours; the red tails are at least as extreme — both tails, because a fluke could go either way. The red share is the \(p\)-value. A third of the sceptic’s world looks like our data, so the umbrellas are not surprising.

Put it into practice

• \(t\) and \(p\)
• Small vs. full sample
• The full sample

\(t\)-value \(= \dfrac{\hat\beta}{\hat\sigma}\): how many standard errors the estimate sits from 0.

\(p\)-value: if \(H_0\) were true, the probability of an estimate this far from 0 (or further).

A small \(p\) (by convention \(< 0.05\)) means the data are hard to reconcile with \(H_0\); evidence against it, not proof of \(H_1\).

ols <- lm_robust(psppsgva ~ eduyrs,
                 data = ESS_sample)
tidy(ols) %>%                    # Tidy result table
  filter(term == "eduyrs") %>%
  select(estimate, std.error,
         statistic, p.value)
#   estimate std.error statistic p.value
# 1    0.042     0.043     0.976   0.334

50 people: the estimate lands inside the test interval → \(p = 0.33\), can’t reject \(H_0\).

All 1,511 people in DK: a smaller estimate, but far outside a much narrower interval → \(p = 0.0015\), reject \(H_0\).

Same question, same axis. Only the standard error shrank.

	Political efficacy
(Intercept)	2.378***
	(0.114)
eduyrs	0.025**
	(0.008)
Num.Obs.	1511
R2	0.010
+ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001

The \(t\)-value is just “how many standard errors from zero” — no more mysterious than that; the numbers in the R output are exactly the ones from the step-by-step picture. The comparison tab makes the real point: the same question is not significant in the 50-person sample and clearly significant in the full one — even though the full-sample estimate is smaller. The null’s bell curve simply got much narrower. Nothing about the world changed, only how much sampling error we were carrying. Non-significant does not mean “no effect”; it can simply mean “not enough people”.

Learning goal achieved

Using the whole weighted sample, education predicts political efficacy:

• \(\hat\beta = 0.025\) per year of education,
• \(\hat\sigma =0.008\),
• \(t = 3.2\), \(p = 0.0015\).

Answer: Yes, better-educated people believe significantly more in the responsiveness of Denmark’s political system.

Close the loop on the opening question and make them read all four numbers — estimate, SE, \(t\), \(p\) — as one sentence. Then add the honest caveats, since both matter: the effect is small per year of education, and significant is not the same as large. And it is not causal — education is not randomly assigned, which is where the next lectures go.

Your turn: inference

Open exercise 2 in a new tab ↗

Check yourself: today’s goals

Look back at the goals from the start of the lecture. Can you tick all three?

• Explain to your neighbour why a random sample beats a convenience sample, and what a weight repairs.
• Say, in one sentence, what a standard error estimates; use the word sampling distribution.
• Read a regression’s \(t\)- and \(p\)-value and state what they do (and don’t) let you conclude about \(H_0\).

Anything feel shaky? That is what this week’s Absalon quiz and the Friday exercise class are for.

Today’s important functions

1. haven::read_spss() + zap_labels(): read SPSS data and turn labelled columns numeric.
2. dplyr::sample_n(): draw a random sample of rows.
3. weighted.mean(), and the weights = argument: analyses that respect survey weights.
4. estimatr::lm_robust(..., weights =): weighted OLS with correct standard errors.
5. qt(): critical values from the \(t\)-distribution.
6. geom_smooth(method = "lm"): an OLS line with its 95% confidence band.
7. broom::tidy(): pull estimates, standard errors, \(t\)- and \(p\)-values into a tibble.

References

Veaux, D., Velleman, and Bock (2021). Stats: Data and Models, Global Edition. Pearson Higher Ed.

Wang, W., D. Rothschild, S. Goel, et al. (2015). “Forecasting elections with non-representative polls”. In: International Journal of Forecasting, pp. 980-991.