Selection Bias

Why correlation ≠ causation
Potential outcomes & confounding

Merlin Schaeffer · Department of Sociology

2026-10-26

By the end of today you can …

1. state a causal hypothesis as a counterfactual comparison: the potential-outcomes framework, its fundamental problem of causality and avergage causal effects.
2. explain why a naïve comparison of two groups is usually not the avergage causal effect: selection bias;
3. read a DAG, spot a confounder on a backdoor path, and use a balance test to expose it.

Frame the day: our goal is not describing patterns but explaining them — and today is about avoiding false explanations. Walk the arc once: a real-world puzzle, then a simple but flawed analysis, then the theory that fixes it (potential outcomes), then the tool that draws it (DAGs).

The research question of the day

The integration paradox: immigrants and their descendants who are better integrated (more educated, longer resident) often report more discrimination, not less (Schaeffer and Kas, 2024).

One proposed mechanism: the better integrated consume more news, become more aware of discrimination, and so report it more often.

Research question: does consuming news increase how often immigrant minorities report discrimination?

Mesut Özil, world champion in 2014, quit Germany’s national team in 2018: “I am German when we win, but an immigrant when we lose.”

Photo: Steindy, CC BY-SA 3.0.

280 estimates from 42 studies. Source: (Schaeffer and Kas, 2024)

Start with the pictures, before any numbers. Özil is the paradox in one person: born in Gelsenkirchen, world champion with Germany, as “integrated” as anyone could be, and the one who said publicly that he was only German when Germany won. Ask whether that is an exception — then click: it is the rule. Say the paradox out loud, because it is genuinely counterintuitive: the better integrated report more discrimination, not less. Point at the forest plot — 280 estimates from 42 studies, so this is a robust pattern, not one odd survey. Today we test one proposed mechanism: awareness, via news consumption.

From hypothesis to comparison

Part 1 of 3

Every causal claim is a comparison.

A hypothesis is a comparison

A causal claim implies a counterfactual: things would have been different had \(X\) (not) happened.

\(H_1:\) news consumption increases how often minorities report discrimination:

\[0 < \text{Avg}_n[\text{Discr.} \mid \text{news} = 1] - \text{Avg}_n[\text{Discr.} \mid \text{news} = 0]\]

\(H_0:\) news consumption does not change it: the difference of the two group averages is \(\leq 0\):

\(0 \leq \text{Avg}_n[\text{Discr.} \mid \text{news} = 1] - \text{Avg}_n[\text{Discr.} \mid \text{news} = 0]\)

Discuss: what comparison is implied by each classic claim? Marx: capitalist labour alienates people and reduces mental health. Simmel: city life increases nervous stimulation and makes residents distant.

The key idea: testing a hypothesis always implies a comparison — causality is a counterfactual, a “what if”. Give them two minutes on Marx and Simmel, then collect: Marx compares capitalist against non-capitalist labour, Simmel urban against rural life. Every causal claim names two worlds, even when it doesn’t say so.

Why not just ask people?

The obvious “study”: ask people “How does reading the news affect you?”

Image: UZH News

Discuss: why is simply asking a bad way to test our hypothesis?

People have inaccurate perceptions of what shapes them because of social desirability, faulty memory, little self-insight. Statements by research subjects are valuable, but they are not credible causal evidence.

We need a comparison, not an introspection.

Ask first — give them two minutes before you click the reveal. Expect: social desirability, poor memory, lack of self-awareness. The line to land: people tell interesting stories about their lives, they do not record causal facts about themselves. So we need a comparison, not an introspection.

Preparation

• Packages
• The APAD survey
• Get & prepare the data

pacman::p_load(
  tidyverse,    # Data manipulation and visualization
  estimatr,     # OLS with robust / weighted standard errors
  modelsummary  # Regression AND balance tables
)

Schaeffer, Kas, and Hagedorn (2023): 1,093 immigrants and children of immigrants in Berlin, Hamburg, Munich, Frankfurt & Cologne, interviewed August-September 2021.

1. “On a typical day, about how much time do you spend watching, reading, or listening to news about politics and current affairs?” (hours + minutes)

2. “How often were you personally discriminated in the following situations here in Germany?”

• when looking for work,
• at work,
• in education,
• looking for housing,
• with officials,
• out in public. (1) Never … (5) Very often

Discrimination means that a person is treated worse than others including specific reasons for this behavior and no factual justification. People use different modes of discrimination like insult, ostracism, or sexual harassment. Rules and laws disadvantaging people are also discrimination.

load("APAD.RData") # Load APAD dataset

APAD <- APAD %>%
  mutate(
    news    = news_hrs * 60 + news_mins,          # total daily minutes
    news_yn = case_when(                          # binary reads news?
      news < 5 ~ 0,                               # 0 if less than 5 minutes
      news >= 5 ~ 1),                             # 1 if 5 minutes or more
    dis_index = rowMeans(                         # average across the 6 domains of discrimination
      select(., dis_trainee, dis_job, dis_school,
             dis_house, dis_gov, dis_public),
      na.rm = TRUE
    )
  )

Keep this short — it is scaffolding, not the point. Two things only: the outcome dis_index averages the six discrimination domains into one index, and the cause comes in two forms, total daily minutes and a “reads news” dummy: at least 5 minutes a day. Then move on; the payoff is the next slide.

The naïve comparison

• The picture
• … its R code
• OLS estimation
• OLS table

ggplot(APAD, aes(y = dis_index, x = news)) +        # Discrimination (Y) by news minutes (X)
  geom_point(aes(size = gewFAKT), alpha = 1/3) +    # One dot per respondent; larger weight = bigger dot
  geom_smooth(aes(weight = gewFAKT), method = "lm", # Weighted OLS regression line ...
              color = "#901A1E") +                # ... in KU red, with its 95% confidence band
  scale_y_continuous(breaks = 1:5,                  # Label the 1-5 answer scale
    labels = c("Never", "Rarely", "Sometimes", "Often", "Very often")) +
  labs(y = "Reported discrimination",               # Axis titles
       x = "Daily minutes of news consumption") +
  theme_minimal(base_size = 14) +                   # Clean look, larger font
  theme(legend.position = "none")                   # No legend for the dot sizes

# Weighted OLS: the discrimination index on the news dummy (1 = reads 5+ minutes a day)
ols <- lm_robust(dis_index ~ news_yn, # Outcome ~ predictor
                 weights = gewFAKT,   # Survey weights: the sample mirrors the population
                 data = APAD)         # Robust standard errors are lm_robust()'s default

# Show the result as a formatted regression table
modelsummary(list("Reported Discrimination" = ols),     # Column title = model name
             stars = TRUE,                              # Significance stars
             coef_rename = c("news_yn" = "Reads news"), # Readable label for the coefficient
             gof_map = c("nobs", "r.squared"),          # Report only N and R²
             output = "kableExtra")                     # Table format for the slides

	Reported Discrimination
(Intercept)	2.320***
	(0.146)
Reads news	−0.237
	(0.155)
Num.Obs.	1085
R2	0.008
+ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001

News readers report, if anything, 0.24 points less discrimination, and the difference is not significant (\(t \approx -1.5\)). Taken at face value, \(H_1\) looks wrong. But is this a fair comparison?

Ask them to interpret the scatter before you show the table — two minutes with a neighbour. The line is flat, if anything slightly negative, so visually it does not support our hypothesis. Then the regression: about −0.24, not significant. Let them draw the obvious conclusion that the hypothesis is wrong — and then ask whether this is a fair comparison. That question is the rest of the lecture.

Small recap: hypothesis tests, \(t\)- and \(p\)-values

1 The sceptic’s claim: reading news makes no difference, \(H_0: \beta = 0\).

2 The sampling distribution: if \(H_0\) were true, samples like ours would still scatter around 0 by chance. With our standard error \(\hat\sigma = 0.155\), 95% of them land within \(0 \pm 1.96 \times \hat\sigma\).

3 Our estimate: \(\hat\beta = -0.237\) is \(t = -1.53\) standard errors from 0; inside the test interval.

4 The \(p\)-value: 87% of chance estimates are closer to 0 than ours; \(p = 0.13\) are at least as far out. Not surprising if \(H_0\) were true → we can’t reject \(H_0\).

A two-minute recap of Lecture 3, on today’s number, so “not significant” on the previous slide means something. Click one idea at a time. (1) Start from the sceptic: news makes no difference. (2) If the sceptic were right, samples like ours would still scatter around 0; the standard error is the ruler, and 95% of chance estimates fall inside the gold interval. (3) Only now bring in our −0.24: about one and a half standard errors out, still inside. (4) The red tails are the chance estimates at least as extreme as ours, in both directions: that share is the \(p\)-value. About one sample in eight would look like ours even if news made no difference, so we cannot reject \(H_0\). Then back to the real question: is this comparison even fair?

Break

Your turn: exercise 1

Run the naïve analysis yourself, on a different outcome: support for an anti-discrimination law.

Open exercise 1 in a new tab ↗

Potential outcomes & selection bias

Part 2 of 3

To see why the comparison can mislead, we need the language of potential outcomes.

Two potential futures for one person

For Ferda, two potential outcomes :

\[\text{Effect}_{\text{Ferda}} = Y_{1,\text{Ferda}} - Y_{0,\text{Ferda}}\]

Ferda
\(Y_0\): discrimination without news	2 (rarely)
\(Y_1\): discrimination with news	4 (often)
Personal causal effect	+2

The fundamental problem of causal inference: we only ever see one of Ferda’s two futures. The other potential outcome is always missing — so a personal causal effect can never be observed directly.

Point at the two pictures: the same Ferda, same armchair, same evening — the only thing that differs is the newspaper. Ferda makes it concrete: 4 with news, 2 without, so her personal causal effect is +2. Then click. Now the hard part — in reality we see only one of those two rows; the other is gone forever. That is the fundamental problem of causal inference, and it is exactly why we cannot fix this by asking people to imagine the other path.

Goal: The average causal effect (ATE)

\[\text{ATE} = \text{Avg}_n[Y_{1i} - Y_{0i}] = \frac{1}{n}\sum_{i=1}^{n} Y_{1i} - \frac{1}{n}\sum_{i=1}^{n} Y_{0i}\]

The average treatment effect (ATE) is just the average of everyone’s personal causal effects.

It compares both potential worlds: one where everyone reads the news, against one where no one does. Neither world is ever fully observed! But under the right conditions, we can estimate the average.

We give up on personal effects and aim for the average instead. Conceptually it compares two whole worlds — everyone reads the news, nobody does — and neither is ever observed. But the average is estimable under the right conditions, and what those conditions are is essentially the rest of this course.

Apples and oranges: Ferda vs. Tuki

We can’t see Ferda’s missing future. So what if we compare her to Tuki, who doesn’t read news?

	Ferda	Tuki
\(Y_0\): without news	(2)	5
\(Y_1\): with news	4	(5)
true effect	(+2)	(0)

Grey = the counterfactual we never see.

\[\underbrace{Y_{1,\text{Ferda}}}_{4} - \underbrace{Y_{0,\text{Tuki}}}_{5} = -1\]

They do not compare. Photo: Michael Johnson, CC BY 2.0.

Discuss: Ferda’s true effect is \(+2\), yet comparing her to Tuki gives \(-1\).

Why is comparing different people so misleading here?

Since we cannot see Ferda’s missing future, we substitute a different person — that is what every observational comparison does. The result is −1, a terrible estimate of either person’s true effect. Give them two minutes on “why is this so wrong?”; the answer you are steering to is that they were different people to begin with.

Observed vs. potential outcomes

\[D_i = \begin{cases} 1, & \text{if } i \text{ reads news (like Ferda)} \\ 0, & \text{otherwise (like Tuki)} \end{cases}\]

Observed outcomes: what the data show.

• \(Y_{1i} \mid D_i = 1\): outcome of those who read news, with news. Ferda: 4
• \(Y_{0i} \mid D_i = 0\): outcome of those who don’t, without news. Tuki: 5

Potential outcomes we never observe (counterfactuals).

• \(Y_{0i} \mid D_i = 1\): news readers, had they not read news. Ferda: (2)
• \(Y_{1i} \mid D_i = 0\): non-readers, had they read news. Tuki: (5)

Everyone has both potential outcomes, \(Y_{1i}\) and \(Y_{0i}\). The data reveal only the one that matches what the person actually did. A naïve comparison uses the blue box only.

Introduce the notation slowly, one click per box. \(D_i\) is simply who reads news. Blue first: these are the two numbers we can actually compute from data, one per group — Ferda’s 4, Tuki’s 5. Then red: the two missing pieces of the puzzle, what each group would have reported in the other world — Ferda’s grey 2, Tuki’s grey 5. Land the green sentence: everyone has both potential outcomes, we only ever see one, and the naïve comparison works with the blue box alone.

The hidden term: selection bias

Same staircase, different paths. Photo: Massimo Strazzeri, “Diverging paths” (Flickr).

	Ferda	Tuki
\(Y_0\): without news	(2)	5
\(Y_1\): with news	4	(5)
true effect	(+2)	(0)

What we can compare: \[\underbrace{Y_{1,\text{Ferda}}}_{4} - \underbrace{Y_{0,\text{Tuki}}}_{5} = -1\]

Step 1: add and subtract Ferda’s unobserved \(Y_{0,\text{Ferda}}\). That changes nothing, because \(-2 + 2 = 0\): \[= Y_{1,\text{Ferda}} \; \color{gray}{- \; Y_{0,\text{Ferda}} \; + \; Y_{0,\text{Ferda}}} \; - \; Y_{0,\text{Tuki}}\]

Step 2: regroup the four terms into two pairs: \[= \underbrace{Y_{1,\text{Ferda}} - Y_{0,\text{Ferda}}}_{\text{Ferda's true effect: } 4 - 2 \, = \, +2} \; + \; \underbrace{Y_{0,\text{Ferda}} - Y_{0,\text{Tuki}}}_{\text{Selection bias: } 2 - 5 \, = \, -3}\]

Selection bias: \(Y_{0,\text{Ferda}} \neq Y_{0,\text{Tuki}}\). Even without any news, Tuki would report more discrimination than Ferda. They took different paths before anyone read a newspaper.

Go slowly; this is the key move of the lecture, and it is only arithmetic. Start from the one comparison we can make: Ferda’s 4 minus Tuki’s 5, −1. Click: we add and subtract the same thing, Ferda’s unobserved Y₀ — overall zero, so nothing changed, but it lets us split the −1. Click: regroup into two pairs — Ferda’s true effect, +2, plus something else worth −3. Click: name it. Selection bias — the two were different before the treatment, like two people who took different branches of the staircase long before we met them.

Selection is everywhere

Discuss: people choose their “treatments”: a dating app, a hospital, the gym, a private school, therapy …

Find examples where those who select in or select out of treatment already differ in the outcome before any treatment (\(Y_0\)). Which way would a naïve comparison go wrong?

Who makes a dating profile?

Who sits in a hospital waiting room? Photo: Kolforn, CC BY-SA 4.0.

Give them three minutes in pairs to collect examples before you reveal anything; the best ones come from their own lives (dating apps, the gym, a private school, therapy). For each, push with one question: would the people who chose it have differed on the outcome anyway? Then the reveal, and point at the two pictures: whoever makes a dating profile was already keen to meet someone before downloading the app; the hospital patients were ill before they ever arrived. Same mechanism, opposite signs — selection bias can make a treatment look better, worse, or harmful.

From two persons to two groups

The same trick, now with group averages. What we can compare: \[\underbrace{\text{Avg}_n[Y_{1i} \mid D_i = 1] - \text{Avg}_n[Y_{0i} \mid D_i = 0]}_{\text{Difference in observed group means}}\]

Step 1: add and subtract the unobserved \(\text{Avg}_n[Y_{0i} \mid D_i = 1]\), news readers had they not read news: \[= \text{Avg}_n[Y_{1i} \mid D_i = 1] \; \color{gray}{- \; \text{Avg}_n[Y_{0i} \mid D_i = 1] \; + \; \text{Avg}_n[Y_{0i} \mid D_i = 1]} \; - \; \text{Avg}_n[Y_{0i} \mid D_i = 0]\]

Step 2: regroup into two pairs: \[= \underbrace{\text{Avg}_n[Y_{1i} \mid D_i = 1] - \text{Avg}_n[Y_{0i} \mid D_i = 1]}_{\text{ATT: average effect on the treated}} \; + \; \underbrace{\text{Avg}_n[Y_{0i} \mid D_i = 1] - \text{Avg}_n[Y_{0i} \mid D_i = 0]}_{\text{Selection bias: the gap in the baseline } Y_0}\]

The ATT is not the ATE

The first pair on the last slide is the average effect on the treated (ATT):

\[\text{ATT} = \text{Avg}_n[Y_{1i} - Y_{0i} \mid D_i = 1]\]

It averages the effect only over those who chose the treatment.

The average treatment effect (ATE) from slide 14 averages over everyone:

\[\text{ATE} = \text{Avg}_n[Y_{1i} - Y_{0i}]\]

	Ferda	Tuki
reads news?	\(D = 1\)	\(D = 0\)
true effect \(Y_1 - Y_0\)	\(+2\)	\(0\)

• ATT: if only Ferda chose news \(\rightarrow\) ATT \(= +2\)
• ATT: if only Tuki chose news \(\rightarrow\) ATT \(= 0\)
• ATE: both of them \(\rightarrow\) ATE \(= \frac{+2 + 0}{2} = +1\)

Observed difference = ATT + selection bias. So even if selection bias were zero, comparing groups tells us the effect for the kind of people who selected in, not for everyone.

Slow down here; students routinely treat “the effect” as one number. Start from the ATT, the first pair of the previous slide: the effect averaged over those who actually read news. Click: contrast it with the ATE from slide 14, averaged over everyone. Click: Ferda and Tuki make the gap concrete — only Ferda chose the news, so the ATT is her +2, while the ATE averages her +2 with Tuki’s 0. Click: the moral. A naïve group comparison is ATT plus selection bias, so even with no selection bias at all it answers a narrower question: what does news do to the people who choose to read it? Ask them whether that could differ from the effect on people who avoid the news.

Selection bias and the integration paradox

Think: if minorities who read the news had not read them, would they report discrimination as often as those who never read news? If not, the selection bias is not zero.

Same arithmetic as for Ferda and Tuki, now for whole groups: nothing new, only averages instead of persons. Start with what we compute: the difference in observed group means. Click: add and subtract the unobserved average — news readers, had they not read news. Click: regroup. The second pair is selection bias, the baseline gap; the first pair is the ATT, which the next slide unpacks. Let the comic sit while they think about the blue box: why might news readers differ at baseline — education, age, language, citizenship?

(Im)balance of observed variables

• The idea
• The table

We can never see the unobserved baseline \(Y_0\). But if the two groups differ on things we can measure (age, citizenship, generation) that is a strong hint they differ on \(Y_0\) too. Balance on covariates indicates but does not prove that a naive comparison gives us a causal effect.

A balance test compares the groups on those background variables.

APAD %>%
  select(news_yn, age, german, imor,
         nbh_exposed, gewFAKT) %>%
  rename(weights = gewFAKT) %>% # so it is used as a weight
  datasummary_balance(~ news_yn, data = .,
                      output = "kableExtra")

		0		1
		Mean	Std. Dev.	Mean	Std. Dev.	Diff. in Means	Std. Error
age		37.3	13.9	43.0	15.1	5.8	3.3
nbh_exposed		4.1	0.7	4.2	1.0	0.2	0.1
		N	Pct.	N	Pct.
german	Ja	48	39.7	602	61.9
	Nein	73	60.3	370	38.1
imor	Child of immigrant	37	30.6	426	43.8
	Immigrant	84	69.4	546	56.2

The groups are not balanced. News readers are markedly older (≈ 43 vs. 37) and more often German citizens. Our naïve comparison is confounded, and these are only the observed differences.

The move here: we can never compare the unobserved baseline, but we can compare observed characteristics — and if the groups differ on those, they almost certainly differ on \(Y_0\) too. Walk the code briefly, then read the table: readers are around 43 versus 37 and more often German citizens. State the conclusion flatly — the groups are imbalanced, so the naïve comparison is definitely biased. And stress: these are only the differences we can see.

DAGs

Part 3 of 3

A directed acyclic graph (DAG) draws our causal assumptions.

Reading a DAG

• Nodes = variables; arrows = causal effects.
• A backdoor path links \(D\) and \(Y\) starting with an arrow into \(D\) (Pearl, Glymour, and Jewell, 2016; Breen, 2022).
• A confounder \(C\) sits on a backdoor path; it makes \(D\) and \(Y\) correlate without any causal arrow between them.
  • Confounder bias is selection bias!

The red dashed arrow is not really part of the DAG. It just marks the spurious \(D\)–\(Y\) correlation that the backdoor path through \(C\) produces.

A DAG maps your causal assumptions, not your data. The puppet metaphor works well: grab a node, shake it, and the movement travels along the arrows. Shake \(C\) and both \(D\) and \(Y\) move together — that is the spurious correlation. Shake \(D\) and \(Y\) does not move at all, because there is no arrow between them. A confounder is just a variable sitting on such a backdoor path.

Our case: a confounded correlation

Plausibly, German citizenship shapes both whether people follow German news and how much discrimination they encounter.

It opens a backdoor path, so the raw news–discrimination correlation is (at least partly) spurious, as the selection bias the balance table revealed.

Now make it concrete with our own case. German citizenship plausibly drives both whether people follow German news and how much discrimination they actually encounter. Trace the backdoor path with your finger: reads news ← German citizen → discrimination. That path on its own could produce the negative correlation we saw — no causal arrow required. And this is exactly the imbalance the balance table found.

The verdict: correlation ≠ causation

Correlation ≠ causation.

The baseline \(Y_0\) differs between the groups we compare, because confounders sort people into and out of how much they read the news.

Ice-cream sales and shark attacks correlate, but temperature drives both.

Close the loop: they now know why this flat line cannot be read causally — confounders sort people into and out of reading the news, so the baselines differ. Ice cream and shark attacks is the throwaway example if anyone still resists. Then bridge to next week: if the problem is that the groups differ to begin with, the fix is to make them comparable — which is what randomisation does.

Your turn: exercise 2

You draw the DAG and run a balance test yourself — is racial appearance balanced across news readers?

Open exercise 2 in a new tab ↗

Today’s general lessons

1. A causal claim is a counterfactual claim. It compares an outcome with a treatment to the same outcome without it.

2. Potential outcomes (\(Y_0\), \(Y_1\)) make this precise; a personal causal effect is \(Y_1 - Y_0\). The fundamental problem: one of them is always missing.

3. The average causal effect is the average of those personal effects.

4. A naïve comparison of groups usually is not the causal effect, because it adds selection bias: the groups differ in their baseline \(Y_0\).

5. A balance test exposes observed imbalance; a DAG shows why: a confounder on a backdoor path. Confounder bias is selection bias.

Run through this fast — they are tired by now. If only one sentence survives the day, make it this one: an observed difference in means is a causal effect plus selection bias, and everything we do from here on is about forcing that second term to zero.

Check yourself: today’s goals

• Write a causal hypothesis as a difference of two potential-outcome averages, and say which term is never observed.
• Explain, using Ferda & Tuki, how an observed group difference splits into a causal effect plus selection bias.
• Draw the news → discrimination DAG with a confounder, and say what a balance test would show.

Shaky on any of these? That is what this week’s Absalon quiz and the Friday exercise class are for.

Today’s important functions

• rowMeans(select(., ...)): average several items into one index (same units).
• estimatr::lm_robust(y ~ d, weights = ...): weighted OLS with robust standard errors.
• modelsummary::datasummary_balance(~ group, data = ...): a balance table across two groups (rename your weight column to weights).

References

Breen, R. (2022). “Causal inference with observational data”. In: Handbook of Sociological Science: Contributions to Rigorous Sociology. Ed. by K. Gërxhani, N. de Graaf and W. Raub. Edward Elgar Publishing, pp. 272-286.

Pearl, J., M. Glymour, and N. P. Jewell (2016). Causal Inference in Statistics - A Primer. 1.th ed. Chichester, West Sussex: Wiley.

Schaeffer, M. and J. Kas (2024). “The integration paradox: Does awareness of the extent of ethno-racial discrimination increase reports of discrimination?” In: Political Psychology.

Schaeffer, M., J. Kas, and P. Hagedorn (2023). “The Association between Actual and Perceived Discrimination (APAD): Technical Report”. In: SocArXiv.