Welcome to PSYC 434

Conducting Research Across Cultures | Trimester 1, 2026

Prof Joseph Bulbulia | Victoria University of Wellington

Assessments

Contact

Course Description

From the VUW course catalogue:

This course focuses on theoretical and practical challenges for conducting research involving individuals from more than one cultural background or ethnicity. Topics include defining and measuring culture; developing culture-sensitive studies; choice of language and translation; communication styles and bias; questionnaire and interview design; qualitative and quantitative data analysis for cultural and cross-cultural research; minorities, power, and ethics in cross-cultural research; and ethno-methodologies and indigenous research methodologies. Appropriate background for this course: PSYC 338.

Course Learning Objectives

In this advanced course, students develop foundational skills in cross-cultural psychological research with a strong emphasis on causal inference.

1. Programming in R. Students will learn the basics of programming in the statistical language R, gaining essential computational tools for psychological research. These skills lay the foundation for applying data analysis techniques in a causal inference framework and beyond. This year we will try something different, introducing you to coding with LLM agents.

2. Understanding causal inference. Students will develop a robust understanding of causal inference concepts and approaches, with particular emphasis on how they mitigate common pitfalls in cross-cultural research. We focus, first, on how to ask questions in comparative psychology, and (only then), how to answer them. This takes to the design of studies, analysing data, and drawing appropriately confident conclusions about cause-and-effect relationships in comparative research.

3. Understanding measurement in comparative settings. Quite a lot of this course is devoted to concepts of measurement in psychological research. Although we will cover classical techniques for constructing and validating psychometrically sound measures across cultures, we will also clarify why statistical tests alone are insufficient to ensure we are measuring as we hope.

Licence

© 2026 Joseph Bulbulia. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence.

Schedule

Weekly Schedule (2026)

Assessments

Overview

Assessment 1: Lab Diaries (10%)

Nine weekly diaries, one per lab (weeks 1–6 and 8–10). There are no labs in week 7 (test 1), week 11 (test 2), or week 12 (presentations). Your best eight diaries count (8 × 1.25%), so you may miss one without penalty. Each diary is graded satisfactory/not satisfactory. You receive full credit for submitting a satisfactory entry. Diaries are due by the end of the lab session.

What to write

Each diary is a short reflection (~150 words) covering:

1. What the lab covered and what you did.
2. A connection to the week's readings or lecture content.
3. One thing you found useful, surprising, or challenging.

Format

Write each diary as a plain markdown (.md) file named by week number: lab-01.md, lab-02.md, …, lab-10.md (there is no lab-07.md). Use GitHub-flavoured markdown formatting: headings, paragraphs, bold, italics, and lists. Because you push diaries to GitHub, your files will render there automatically. These submissions build your markdown fluency; later in the course you will use Quarto to extend markdown to PDF and Word.

Submission

Push your diary files to the lab diary GitHub repository created in Lab 1. The commit timestamp is your submission record.

# Lab 01: Introduction to R

This week we installed R and RStudio, then ran our first script.
The exercise connected to the lecture on **causal questions** by
showing how we structure data for analysis.

I found the following steps useful:

- Creating an RStudio project
- Writing a short R script
- Pushing changes to GitHub

Assessment 2: In-Class Test 1 (20%) — 22 April

Covers material from weeks 1–6 (causal diagrams, confounding, ATE, effect modification). Test duration is 50 minutes. The allocated time is 1 hour 50 minutes. Required: pen/pencil. No devices permitted.

Assessment 3: In-Class Test 2 (20%) — 20 May

Covers material from weeks 8–10 (heterogeneous treatment effects, machine learning, resource allocation, policy trees, and classical measurement theory). Same format and conditions as test 1.

Assessment 4: In-Class Presentation (10%) — 27 May

10-minute presentation summarising the student's study. Assessment criteria: clarity, efficiency, and quality of presentation.

Assessment 5: Research Report (40%) — Due 30 May

# install (once)
install.packages("remotes")
remotes::install_github("go-bayes/causalworkshop@v0.2.1")

# generate data
library(causalworkshop)
d <- simulate_nzavs_data(n = 5000, seed = 2026)

Option A: Research Report

Quantify the Average Treatment Effect of a specific exposure on a specific outcome using the synthetic NZAVS panel dataset generated by causalworkshop::simulate_nzavs_data(). You choose one of three exposures and one of four outcomes (see the data generation instructions above).

• Introduction: 1,500-word limit.
• Conclusion: 1,500-word limit.
• Methods/Results: concise, no specific word limit.
• APA style. Submit as a single PDF with R code appendix.

Assessment Criteria (Option A)

Stating the problem. State your question clearly. Explain its scientific importance. Frame it as a causal question. Identify any subgroup analysis (e.g. cultural group). Explain the causal inference framework for non-specialists. Describe ethics/policy relevance. Confirm data are from the NZAVS simulated dataset using three waves.

Determining the outcome. Define outcome variable $Y$. Assess multiple outcomes if applicable. Explain how outcomes relate to the question. Address outcome type (binary and rare?) and timing (after exposure).

Determining the exposure. Define exposure variable $A$. Explain relevance, positivity, consistency, and exchangeability. Specify exposure type (binary or continuous) and whether you contrast static interventions or modified treatment policies. Confirm exposure precedes outcome.

Accounting for confounders. Define baseline confounders $L$. Justify how they could affect both $A$ and $Y$. Confirm they are measured before exposure. Include baseline measures of exposure and outcome in confounder set. Assess sufficiency; explain sensitivity analysis (E-values) if needed. Address confounder type (z-scores for continuous; one-hot encoding for categorical with three or more levels).

Drawing a causal diagram. Include both measured and unmeasured confounders. Describe potential measurement error biases. Add time indicators ensuring correct temporal order.

Identifying the estimand. State your causal contrast clearly.

Source and target populations. Explain how your sample relates to your target populations.

Eligibility criteria. State the eligibility criteria for the study.

Sample characteristics. Provide descriptive statistics for baseline demographics. Describe magnitude of exposure change from baseline. Include references for more information about the sample. Make clear the data are simulated.

Missing data. Check for missing data. Describe how you will address the problem (IPCW).

Model approach. Decide between G-computation, IPTW, or doubly-robust estimation. Specify model. Explain how machine learning works. Address outcome specifics (logistic regression for rare binary outcomes; z-scores for continuous). Describe sensitivity analysis (E-values).

Option B: Marsden Fund Expression of Interest (EOI)

Write a first-round Marsden Fund Expression of Interest (EOI) following the RSNZ 2026 guidelines. Your research question must use the causal inference framework taught in this course. Assume an Ecology, Human Behaviour, and Evolution (EHB) panel.

Formatting: 12-point Times New Roman, single spacing, 2 cm margins. Submit as a single PDF.

Required Sections

Section numbers follow the 2026 RSNZ EOI form.

1a. Research Title (max 25 words). Plain language, no jargon. The title should be accessible to a scientifically literate non-specialist.

1d. Research Summary (max 200 words). State the current state of the field, the aims of your research, the methods you will use, and the expected outcome. This summary must be standalone: assessors outside your discipline will read it.

2. Vision Mātauranga (max 200 words). Describe how the proposed research relates to the four Vision Mātauranga (VM) themes: (i) indigenous innovation, drawing on Māori knowledge, resources, and people; (ii) taiao, achieving environmental sustainability through iwi and hapū relationships with land and sea; (iii) hauora/oranga, improving health and social wellbeing; (iv) mātauranga, exploring indigenous knowledge and its contribution to NZ research. If none of the themes apply, you may state "not applicable" with a considered justification.

3a. Abstract (max 1 page). Cover the following: aims of the research; importance of the research area; novelty, originality, insight, and ambition of the proposed work; potential impact; methodology; and your capacity to deliver.

3b. Benefit Statement (max 400 words, 1 page). Describe the economic, environmental, or health benefit of the research to New Zealand. Explain why NZ is the right place for this research and describe potential impacts for Māori. In a student context the benefit case may be aspirational, but it must be concrete.

3c. References (max 3 pages). Bold your own name. Include article titles and full author lists (up to 12 authors; use "et al." thereafter).

3d. Roles and Resources (max 1 page). Describe the contributions of each team member, the resources required, and any ethical considerations. Use the Roles and Resources form.

Assessment Criteria (Option B)

Research. Quantifiable impact potential through novelty, originality, insight, and ambition. Rigorous methods grounded in prior research. Ability and capacity to deliver.

Benefit. Economic, environmental, or health benefit to New Zealand. Rationale for NZ-based research. In a student context the benefit case may be aspirational but must be concrete.

Vision Mātauranga. Relation to VM themes; where relevant, engagement with Māori. "Not applicable" is acceptable with considered justification.

Causal reasoning (course-specific). Well-defined causal question, clearly stated causal estimand, appropriate identification strategy. This criterion carries substantial weight.

For the full Marsden Fund assessment criteria, see the RSNZ 2026 EOI Guidelines (pdf).

Extensions and Materials

Extensions

Negotiate a new due date by writing (email) before the mid-term test. Every reasonable request will be accepted (e.g. too many assignments falling in the same week, you want another week to complete).

Penalties

Late submissions incur a one full grade-per-week penalty: e.g. if late by one day, B becomes C; one week later, C becomes D. Over-length assignments will be penalised.

Unforeseeable Events

Extensions will require evidence (e.g. medical certificate).

Materials and Equipment

Bring a laptop with R and RStudio installed for data analysis sessions. Contact the instructor if you lack computer access. For in-class tests, bring a writing utensil. Electronic devices are not permitted during tests.

Test 1: Study Sheet

Comprehensive study sheet covering lectures 1–6 (weeks 1–6).

Practice Quiz (2024, with Answers)

Last year's in-class quiz covering key concepts and causal diagrams. Includes answers.

Practice Quiz (2025, Workshop Diagnostic)

A diagnostic quiz from the 2025 SASP Causal Inference Workshop covering causal diagrams, confounding, and identification assumptions.

Simulation Guide

Simulations are pedagogical tools that let us see what causal inference methods do when we know the truth. In observational research, we never know the true causal effect. In a simulation, we build the data-generating process ourselves, so we can compare each method's estimate against the ground-truth parameter. The four simulations in this course illustrate distinct threats to valid causal inference and distinct strategies for addressing them.

Download the R script

Generalisability and transportability

Connects to Week 4: external validity and selection bias.

This simulation creates two populations that differ in the prevalence of an effect modifier $Z$. In the sample, $Z=1$ is rare ($p=0.1$); in the target population, $Z=1$ is common ($p=0.5$). The treatment effect depends on $Z$: individuals with $Z=1$ benefit more from treatment. Because the sample under-represents these high-benefit individuals, the naive sample Average Treatment Effect (ATE) underestimates the population ATE.

The simulation fits three models: an unweighted model on the sample, a weighted model on the sample (using inverse-probability-of-sampling weights), and an oracle model on the full population. The regression coefficients are nearly identical across all three models, yet the marginal ATEs differ. This dissociation is the central lesson: model coefficients describe conditional associations, but the ATE is a marginal quantity that depends on the distribution of effect modifiers in the target population. Weighting the sample to match the population distribution of $Z$ recovers the correct ATE.

The script also includes a manual calculation section that shows exactly what stdReg does under the hood: create counterfactual datasets in which everyone receives $A=0$ and everyone receives $A=1$, predict outcomes under each scenario, and take the mean difference. This "g-computation" procedure makes the marginalisation step explicit.

Cross-sectional data ambiguity

Connects to Week 3: confounding versus mediation.

This simulation generates data in which $A$ causes $L$, and $L$ causes $Y$. The variable $L$ is therefore a mediator, not a confounder. Two models are fit: one that conditions on $L$ (treating it as a confounder) and one that omits $L$ (treating it as a mediator). The model that conditions on $L$ returns a near-zero estimate for the effect of $A$ on $Y$ because it blocks the very path through which $A$ operates. The model that omits $L$ correctly recovers the total effect.

The crux of the problem is that with cross-sectional data alone, the investigator cannot distinguish a confounder from a mediator. Both the fork $A\leftarrow L\to Y$ and the chain $A\to L\to Y$ produce the same observed association between $A$, $L$, and $Y$. The correct modelling decision depends on the assumed causal structure, which the data themselves do not reveal.

Confounding control strategies

Connects to Weeks 3–4: conditioning choices and the backdoor criterion.

This simulation builds a three-wave panel structure with a baseline covariate $L_0$, a prior outcome $Y_0$, a prior exposure $A_0$, an unmeasured confounder $U$, a treatment $A_1$, and an outcome $Y_2$. The true treatment effect is ${\delta }_{A_1}=0.3$, and the outcome also depends on $Y_0$, $A_0$, $L_0$, their interactions, and $U$.

Three models are compared. The "no control" model regresses $Y_2$ on $A_1$ alone and overestimates the effect because it leaves all confounding paths open. The "standard" model adds $L_0$ but still omits $Y_0$ and $A_0$, leaving residual confounding. The "interaction" model conditions on $L_0$, $A_0$, $Y_0$, and their interactions with $A_1$, recovering an estimate close to the true value.

The simulation uses the clarify package to obtain simulation-based confidence intervals for each ATE. The progressive improvement from no control to standard to interaction control illustrates that closing more backdoor paths moves the estimate closer to the truth, but only conditioning on the right set of variables eliminates confounding entirely.

Causal forest estimation

Connects to Week 8: machine learning for heterogeneous treatment effects.

This simulation uses the same data-generating process as the confounding control simulation, then fits a causal forest (from the grf package) with $L_0$, $A_0$, and $Y_0$ as covariates. The causal forest is a non-parametric method that estimates individual-level treatment effects $\hat{\tau }(x)$ by partitioning the covariate space adaptively. Unlike the parametric models above, the causal forest does not require the investigator to specify interaction terms; it discovers them from the data.

The simulation reports the causal forest's ATE estimate and its standard error. Comparing this estimate to the parametric interaction model illustrates two points. First, the causal forest can recover the ATE without requiring the analyst to guess the correct functional form. Second, the causal forest provides a standard error that accounts for the adaptive splitting, making valid inference possible even in the non-parametric setting.

Running the simulations

To run the full script, open R and execute:

source("scripts/simulations.R")

Each section prints its results to the console. You can also run individual sections by selecting and executing the relevant code blocks. The script sets random seeds so that results are reproducible.

The Causal Workflow: Nine Steps

This page summarises a nine-step framework for conducting causal inference with observational data. Each step addresses a potential threat to valid causal interpretation. The framework draws on Hernan and Robins (What If, 2024), VanderWeele (2022), and Bulbulia (2024).

Step 1: State a well-defined treatment

Specify the hypothetical intervention precisely enough that every member of the target population could, in principle, receive it. "Weight loss" is too vague because people lose weight via exercise, diet, depression, surgery, and more. A clearer intervention is: "engage in vigorous physical activity for at least 30 minutes per day."

Precision here underwrites the causal consistency assumption (Step 5). If the treatment is vaguely defined, different people effectively receive different interventions, and the potential outcome $Y(a)$ is not well-defined.

See Week 5 for the formal definition of causal consistency.

Step 2: State a well-defined outcome

Define the outcome so the causal contrast is meaningful and temporally anchored. "Wellbeing" is underspecified; "psychological distress one year post-intervention, measured with the Kessler-6" is interpretable and reproducible. Include timing, scale, and instrument.

See Week 10 for how measurement choices affect causal identification.

Step 3: Clarify the target population

Say exactly who you aim to inform. Eligibility rules define the source population, but sampling and participation can yield a study population with a different distribution of effect modifiers. If you intend to generalise beyond the source population (transportability), articulate the additional conditions required.

See Week 6 for how effect modification interacts with population composition.

Step 4: Evaluate exchangeability

Make the case that potential outcomes are independent of treatment conditional on covariates: $Y(a)\perp \perp A\mid X$. Use DAGs, subject-matter arguments, pre-treatment covariate balance checks, and overlap diagnostics. If exchangeability is doubtful, redesign (e.g., stronger measurement, alternative identification strategies) rather than rely solely on modelling.

See Week 5 for the formal definition of exchangeability.

Step 5: Ensure causal consistency

Consistency requires that, for individuals receiving a treatment version compatible with level $a$, the observed outcome equals $Y(a)$. It also presumes well-defined versions and no interference between units. When multiple versions exist, either refine the intervention so versions are irrelevant to $Y(a)$, or condition on version-defining covariates.

See Week 5 for examples of consistency violations.

Step 6: Check positivity (overlap)

Each treatment level must occur with non-zero probability at every covariate profile needed for exchangeability:

$$P(A=a\mid L=l)>0$$

Diagnose limited overlap using propensity score distributions or extreme weights. Consider design-stage remedies (trimming, restriction, adaptive sampling) before estimation.

See Week 5 for the formal positivity assumption.

Step 7: Ensure measurement aligns with the scientific question

Verify that constructs are captured by instruments whose error structures do not distort the causal contrast of interest. Be explicit about forms of measurement error (classical, Berkson, differential, misclassification) and their structural implications for bias.

See Week 10 for how measurement error creates collider bias in DAGs.

Step 8: Preserve representativeness

End-of-study analyses should reflect the target population's distribution of effect modifiers. Differential attrition, non-response, or measurement processes tied to treatment and outcomes can induce selection bias. Plan strategies such as inverse probability weighting for censoring, multiple imputation, and sensitivity analyses for missing-not-at-random data.

See Week 4 for selection bias structures.

Step 9: Document transparently

Make assumptions, disagreements, and judgement calls legible. Register or timestamp your analytic plan. Include identification arguments (DAGs), code, and data where possible. Report robustness and sensitivity analyses. Transparent reasoning is a scientific result in its own right.

Summary table

Potential Outcomes and Causal Inference

This page introduces the fundamental problem of causal inference, the potential outcomes framework, and the three identification assumptions needed to estimate causal effects from data. It draws on the Women's Health Initiative (WHI) hormone replacement therapy case study as a motivating example.

Motivating example: hormone replacement therapy

Observational evidence (1980s–1990s)

Throughout the 1980s and 1990s, observational studies suggested that oestrogen therapy reduced all-cause mortality in postmenopausal women by roughly 30% (hazard ratio $\approx 0.68$ for current users vs. never users). Professional bodies endorsed hormone replacement therapy (HRT) on this basis:

• 1992, American College of Physicians: "Women who have coronary heart disease or who are at increased risk ... are likely to benefit from hormone therapy."
• 1996, American Heart Association: "ERT does look promising as a long-term protection against heart attack."

The experiment disagreed

The Women's Health Initiative (WHI) was a large randomised, double-blind, placebo-controlled trial enrolling 16,000 women aged 50–79. Participants were assigned to oestrogen plus progestin therapy and followed for up to eight years.

The experimental hazard ratio for all-cause mortality was 1.23 (initiators vs. non-initiators), the opposite direction from the observational finding.

What went wrong?

The discrepancy was not a failure of causal assumptions. It was a failure of study design: the observational studies did not correctly emulate a target trial. Specifically, they failed to align "time zero" (the start of follow-up) with the moment of treatment initiation, introducing survivor bias. When investigators re-analysed the observational data using a target trial emulation framework that matched treatment initiation to the start of follow-up, the observational estimates aligned with the experimental findings.

The fundamental problem of causal inference

Causality is never directly observed. To quantify a causal effect, we need to compare two states of the world for the same individual, but each individual can experience only one.

Notation

Let $A$ denote a binary exposure ($A=1$: treated, $A=0$: untreated) and $Y$ denote the outcome.

• $Y_i(1)$: the potential outcome for individual $i$ under treatment.
• $Y_i(0)$: the potential outcome for individual $i$ under control.

The individual causal effect is:

$${\tau }_i=Y_i(1)-Y_i(0)$$

We say there is a causal effect when $Y_i(1)-Y_i(0)\ne 0$.

The missing-data problem

At most one potential outcome is observed for each individual. The unobserved outcome is the counterfactual:

• If $A_i=1$ is observed, then $Y_i(0)$ is counterfactual.
• If $A_i=0$ is observed, then $Y_i(1)$ is counterfactual.

Individual-level causal effects are therefore generally unidentifiable. However, under certain assumptions, we can identify average causal effects at the population level.

Three identification assumptions

1. Causal consistency

The potential outcome corresponding to the exposure an individual actually receives equals their observed outcome:

$$Y_i(a)=Y_i\mid A_i=a$$

This assumption requires that treatment is well-defined (no hidden versions of treatment) and that there is no interference between units (one person's treatment does not affect another's outcome).

2. Exchangeability

The potential outcomes are independent of treatment assignment. In a randomised experiment, this holds by design. In observational studies, we require conditional exchangeability: after conditioning on a set of measured covariates $L$, treatment assignment is independent of potential outcomes:

$$Y(a)\perp \perp A\mid L$$

When exchangeability holds, the Average Treatment Effect (ATE) is identified:

$$\text{ATE}=\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]=\mathbb{E}(Y\mid A=1)-\mathbb{E}(Y\mid A=0)$$

In observational settings with confounders $L$:

$$\text{ATE}=\sum _l[\mathbb{E}(Y\mid A=1,L=l)-\mathbb{E}(Y\mid A=0,L=l)]\mathrm{Pr}(L=l)$$

3. Positivity

Every individual has a non-zero probability of receiving each treatment level, conditional on their covariates:

$$P(A=a\mid L=l)>0\text{for all }a\text{ and }l$$

Positivity is the only assumption that can be verified with data. Violations occur when certain subgroups never receive a particular treatment level, making causal effect estimates for those subgroups extrapolations rather than identifiable quantities.

From experiments to observational data

Randomised experiments address the fundamental problem by balancing confounders across treatment groups. Random assignment satisfies exchangeability by design, and controlled treatment administration satisfies consistency. Although individual causal effects remain unobservable, random assignment allows inference about average (marginal) causal effects.

In observational data, we must satisfy these three assumptions through study design, covariate adjustment, and sensitivity analysis. The remainder of this course develops the tools for doing so: causal diagrams (weeks 2–4), estimation methods (weeks 5–6, 8–9), and measurement considerations (week 10).

Discussion questions

1. Where in your own research would an average treatment effect be the right causal estimand, and when would it mask disparities that stakeholders care about?
2. Which of the three identification assumptions is most fragile in your field, and what designs or measurements could strengthen it?
3. What are examples of post-treatment variables you have been tempted to adjust for, and how would doing so bias the total effect?

Further reading

• Hernan MA, Robins JM. Causal Inference: What If. Chapman & Hall/CRC, 2025. Chapters 1–3. Book site
• See the Course Readings page for a chapter-by-week guide.

Reporting Guide

This guide covers how to report results from causal inference analyses. It follows the nine-step causal workflow and demonstrates best practices for communicating average treatment effects, heterogeneous effects, and sensitivity analyses. This resource directly supports the research report assessment (40%).

The nine-step causal inference workflow

Before reporting results, ensure each step has been addressed.

Steps 1–3: problem definition

1. Well-defined treatment. Specify the exposure precisely, including the contrast (e.g., "weekly religious service attendance vs. less than weekly").
2. Well-defined outcome. State the outcome measure, its scale, and when it was assessed.
3. Target population. Define who the results apply to, including any weighting for population representativeness.

Steps 4–6: identification strategy

4. Exchangeability. Describe how conditional independence was achieved (e.g., "rich baseline covariate control including 32 covariates").
5. Consistency. Explain why the treatment is well-defined and uniform across individuals.
6. Positivity. Report verification through transition tables showing exposure variation across covariate strata.

Steps 7–9: implementation

7. Measurement validity. Note the psychometric properties of outcome scales.
8. Attrition handling. Describe how panel dropout was addressed (e.g., inverse probability of censoring weights).
9. Transparent reporting. Document all analytical decisions and assumptions.

Reporting average treatment effects

Standard ATE table format

Include these elements for each outcome:

Key reporting elements

1. Effect sizes: report in standard deviation units for continuous outcomes.
2. Confidence intervals: show uncertainty around each estimate.
3. E-values: indicate robustness to unmeasured confounding.
4. Sample size: total analysed after exclusions and weighting.

Example results text

"Weekly religious service attendance showed positive causal effects across all cooperation measures. The largest effects were observed for social outcomes: sense of belonging ($\beta =0.18$, 95% CI: 0.14–0.22) and social support ($\beta =0.15$, 95% CI: 0.11–0.19). All effects were robust to moderate unmeasured confounding (E-values > 1.6)."

Reporting heterogeneous treatment effects

RATE results summary

When heterogeneity is detected, report it systematically:

Example heterogeneity text

"We found substantial heterogeneity in treatment effects for social outcomes (RATE-Qini > 0.08, $p<0.05$), but limited heterogeneity for behavioural outcomes. This suggests that while religious service benefits most people socially, individual responses vary considerably in magnitude."

Reporting policy tree results

Present policy trees with both standardised and original-scale interpretations.

Subgroup identification with data-scale effects

High-response subgroups for charitable donations:

1. Older adults with high agreeableness (age > 45, agreeableness > +1 SD)

   • Standardised effect: $\beta =0.28$ (95% CI: 0.21–0.35)
   • Data-scale effect: NZ$847 additional annual donations (95% CI: NZ$635–1,058)
   • Sample proportion: 23%

2. Parents with medium conscientiousness (parent = yes, conscientiousness > 0)

   • Standardised effect: $\beta =0.22$ (95% CI: 0.16–0.28)
   • Data-scale effect: NZ$665 additional annual donations (95% CI: NZ$484–846)
   • Sample proportion: 31%

3. All others

   • Standardised effect: $\beta =0.08$ (95% CI: 0.04–0.12)
   • Data-scale effect: NZ$242 additional annual donations (95% CI: NZ$121–363)
   • Sample proportion: 46%

Example policy tree text

"Policy tree analysis identified two subgroups with enhanced treatment response for charitable donations. Older adults (45+) with high agreeableness showed the largest increase ($\beta =0.28$ SD, equivalent to NZ$847 additional annual donations), representing 23% of the sample. In practical terms, targeted interventions toward these subgroups could generate 2.8–3.5 times more charitable giving than population-wide approaches."

Sensitivity analysis: E-values

Interpretation

E-values quantify robustness to unmeasured confounding. The E-value is the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need with both the treatment and the outcome to explain away the observed effect.

Example sensitivity text

"To assess robustness to unmeasured confounding, we calculated E-values for all estimates. The observed effect on sense of belonging (E-value = 2.4) would require an unmeasured confounder associated with both religious service and belonging by a risk ratio of 2.4 each to explain away the result."

Methods section template

A complete methods section following the nine-step workflow should include:

1. Treatment definition: what the exposure is, how it is coded, and the contrast of interest.
2. Outcome definition: measures used, timing of assessment, any transformations applied.
3. Target population: sampling frame, weighting strategy, eligibility criteria.
4. Causal identification: covariates conditioned on, justification for conditional exchangeability.
5. Statistical analysis: estimation method, key tuning parameters (e.g., number of trees, minimum node size).
6. Attrition handling: censoring weights, stages of dropout addressed.
7. Heterogeneity assessment: RATE metrics, false discovery rate correction, policy tree depth.
8. Sensitivity analysis: E-values for all primary estimates.

Reporting checklist

Do report

• Both standardised and data-scale effects
• Effect sizes with confidence intervals
• Sample sizes after exclusions and weighting
• E-values for sensitivity analysis
• Clear practical interpretation of effect magnitudes
• Subgroup sizes and effect magnifications
• Target trial framework and causal question
• Explicit treatment and outcome definitions

Do not report

• Model coefficients without interpretation
• p-values alone without effect sizes
• Only standardised effects for policy-relevant outcomes
• Technical details that obscure main findings
• Causal claims beyond your identification strategy

Figure presentation

ATE plots

• Use forest plots with confidence intervals.
• Order by effect magnitude or E-value.
• Include sample sizes.

Policy tree plots

• Show decision rules clearly.
• Include sample proportions in each node.
• Provide plain-language interpretation alongside the tree.

Glossary and DAG Hand-outs

A reference page covering causal inference terminology and links to hand-out PDFs on directed acyclic graphs (DAGs), confounding, selection bias, and measurement error.

Causal inference glossary

Causal inference rests on mathematical foundations that enjoy broad agreement, but the terminology varies across disciplines. The same word sometimes carries different (even opposite) meanings in different literatures. Terms to watch include "selection", "fixed effects", "standardisation", "moderator", "structural equation model", and "identification".

Core concepts

Graphical concepts

Estimation and sensitivity

DAG hand-outs

The following hand-outs cover DAG conventions, common structures, and specific forms of bias. All PDFs are available for download from the hand-outs folder (Dropbox).

Foundations

Common applications

Time series and confounding

Advanced topics

Supplementary materials

Course Readings

Primary text

Hernan MA, Robins JM. Causal Inference: What If. Chapman & Hall/CRC, 2025.

• Book website
• Free PDF (Dropbox)

Chapters 1–9 are the required reading for this course. The book is freely available from the link above.

Chapter guide

Supplementary references

These are not required but provide additional depth on specific topics covered in the course.

• Neal B. Introduction to Causal Inference. 2020. Chapters 1–2. Covers the same foundations as Hernan and Robins with a machine-learning orientation.
• Pearl J, Glymour M, Jewell NP. Causal Inference in Statistics: A Primer. Wiley, 2016. Compact introduction to the graphical (structural) approach to causation.

Week 1: How to Ask a Question in Psychological Science?

Lab: Git and GitHub

Lecture: Introduction to the Course

Two foundational questions

Every causal inquiry in psychological science begins with two questions. First: what do I want to know? Second: for which population does this knowledge generalise? These questions sound simple, yet most published studies answer neither one clearly. A regression table reports associations, but associations are not causes. A convenience sample of undergraduates describes that sample, not the population we care about. This course teaches you to state your question precisely before you touch any data.

The first question forces you to specify an intervention. "Does green space affect happiness?" is vague. A sharper version: "What is the causal effect of moving from a neighbourhood with no accessible park to one with an accessible park on self-reported life satisfaction?" Here the treatment is well-defined ($A=1$ for park access, $A=0$ for no park access), the outcome is specified ($Y$ = life satisfaction), and someone could, in principle, run an experiment to test it.

The second question forces you to specify a target population. Results obtained from a sample of university students in Wellington may not generalise to elderly residents in rural Southland. The distinction between the sample population (who you studied) and the target population (who you want to learn about) is central to external validity, which we revisit formally in Week 4.

The fundamental problem of causal inference

Consider a person who moves to a neighbourhood with a park. She reports high life satisfaction. Would she have reported the same life satisfaction had she stayed in her old neighbourhood without a park? We cannot know, because she experienced only one of the two conditions. This is the fundamental problem of causal inference: for any individual, we can observe at most one potential outcome.

We formalise this with potential outcomes notation. Let $Y_i(1)$ denote the outcome that person $i$ would experience under treatment ($A=1$), and $Y_i(0)$ the outcome under control ($A=0$). The individual causal effect is:

$${\delta }_i=Y_i(1)-Y_i(0)$$

Because we observe only one of $Y_i(1)$ or $Y_i(0)$, the individual causal effect ${\delta }_i$ is never directly observable. This is not a limitation of our methods; it is a logical constraint. No amount of data collection, no statistical technique, and no machine learning algorithm can reveal both potential outcomes for the same person at the same time.

From individuals to populations

If individual effects are unobservable, what can we learn? We can learn about average effects across a population. The Average Treatment Effect (ATE) is defined as:

$$\text{ATE}=\mathbb{E}[Y(1)-Y(0)]$$

This is the expected difference in the outcome if everyone in the target population experienced treatment versus if everyone experienced control. The ATE is a population-level quantity. It tells us what would happen on average, not what would happen to any particular person.

The shift from individual to population is not just a concession to practicality. Causal inference contrasts counterfactual states at the population or subpopulation level. When we say "green space causes happiness," we mean that on average, across a defined population, access to green space raises life satisfaction relative to the counterfactual of no access.

How randomisation solves the problem

If we randomly assign people to neighbourhoods with or without parks, treatment assignment is independent of all other characteristics, both observed and unobserved. The healthy, the unhealthy, the wealthy, the poor, the optimists, and the pessimists are all equally probable in both groups. Under randomisation, the average outcome among the treated group estimates $\mathbb{E}[Y(1)]$, and the average outcome among the control group estimates $\mathbb{E}[Y(0)]$. Their difference estimates the ATE.

Randomisation works because it satisfies a condition called exchangeability: the potential outcomes are independent of treatment assignment. We write this as $Y(a)\perp \perp A$. In words, the treatment and control groups are exchangeable: had the treated group received control instead, their average outcome would have matched the control group's observed average, and vice versa.

We develop this idea formally in Week 2, where we also introduce the three assumptions that underpin all causal inference (causal consistency, conditional exchangeability, and positivity). For now, the key insight is that randomisation transforms the unobservable population-level counterfactual contrast into an observable comparison between groups.

What comes next

Most psychological research cannot randomise the variables we care about. We cannot randomly assign people to experience trauma, adopt a religion, or lose a job. This course is about what to do when randomisation is impossible. The answer involves stating our causal assumptions explicitly (using causal diagrams), checking whether those assumptions are sufficient for identification (using the backdoor criterion), and estimating the causal effect using methods that respect the structure of the problem.

Lab materials: Lab 1: Git and GitHub

Week 2: Causal Diagrams — Five Elementary Structures

Seminar

Overview

• Understand basic features of causal diagrams: definitions and applications
• Introduction to the five elementary causal structures
• Lab: gentle introduction to simulation and regression

Review

1. Psychological research begins with two questions:

1. What do I want to know?
2. For which population does this knowledge generalise?

This course considers how to ask psychological questions that pertain to populations with different characteristics.

2. In psychological research, we typically ask questions about the causes and consequences of thought and behaviour: "What if?" questions (Hernán & Robins, 2025).

3. The following concepts help us describe two distinct failure modes:

• External validity: the extent to which findings generalise to other situations, people, settings, and time periods. We want to know if our findings carry beyond the sample population to the target population. We fail when our results do not generalise as we think, or when we have not clearly defined our question or target population.

• Internal validity: the extent to which the associations we obtain from data reflect causality. When asking "What if?" questions, we want to understand what would happen if we intervened. In this course, we use "treatment" or "exposure" to denote the intervention, and "outcome" to denote the effect of an intervention.

4. During the first part of the course, our primary focus is on challenges to internal validity from confounding bias.

How randomisation identifies causal effects

Last week we introduced the fundamental problem of causal inference: we cannot observe both potential outcomes $Y_i(1)$ and $Y_i(0)$ for the same individual. Randomisation solves this problem at the population level. When treatment is randomly assigned ($\mathcal{R}\to A$), treatment assignment is independent of all potential outcomes:

$$Y(a)\perp \perp A$$

This condition is called unconditional exchangeability. It means that the treated and untreated groups are, on average, identical in every respect except the treatment they received. Any difference in average outcomes between groups can therefore be attributed to the treatment itself. The ATE is then identified by a simple difference in group means:

$$\widehat{\text{ATE}}=\hat{\mathbb{E}}[Y\mid A=1]-\hat{\mathbb{E}}[Y\mid A=0]$$

Experiments are the benchmark for causal inference because randomisation eliminates confounding without requiring the investigator to know or measure the confounders. However, most psychological questions cannot be answered by experiment alone: we cannot randomly assign people to experience grief, adopt a religion, or grow up in poverty. For these questions, we need observational methods that achieve conditional exchangeability through careful adjustment. The causal diagrams we learn today are the tools for deciding what to adjust for.

Definitions

Internal validity is compromised if the association between the treatment and outcome in a study does not consistently reflect causality in the sample population as defined at baseline.

External validity is compromised if the association between the treatment and outcome in a study does not consistently reflect causality in the target population as defined at baseline.

Confounding bias exists if there is an open back-door path between the treatment and outcome, or if the path between the treatment and outcome is blocked.

Today, our purpose is to clarify the meaning of each term in this definition. To that end, we introduce the five elementary graphical structures employed in causal diagrams, then explain the four elementary rules that allow investigators to identify causal effects from the asserted relations in a causal diagram.

Introduction to Causal Diagrams

Causal diagrams (also called causal graphs, Directed Acyclic Graphs, or Causal DAGs) are graphical tools whose primary purpose is to enable investigators to detect confounding biases.

Remarkably, causal diagrams are rarely used in psychology!

The meaning of our symbols

For us:

• $X$ denotes a variable without reference to its role.
• $A$ denotes the "treatment" or "exposure" variable: the variable for which we seek to understand the effect of intervening on it. It is the "cause."
• $Y$ denotes the outcome or response of an intervention. It is the "effect."
• $Y(a)$ denotes the counterfactual or potential state of $Y$ in response to setting the level of the exposure to $A=a$. To consistently estimate causal effects we need to evaluate counterfactual states of the world. This might seem like science fiction, but we are already familiar with methods for obtaining such contrasts: randomised controlled experiments.
• $L$ denotes a measured confounder or set of confounders: a variable which, if conditioned upon, closes an open back-door path between $A$ and $Y$.
• $U$ denotes an unmeasured confounder: a variable that may affect both the treatment and the outcome but for which we have no direct measurement.
• $M$ denotes a mediator: a variable along the path from exposure to outcome. Conditioning on a mediator when estimating the total effect of $A$ on $Y$ will bias that estimate.
• $\bar{X}$ denotes a sequence of variables, for example a sequence of treatments.
• $\mathcal{R}$ denotes a randomisation or a chance event.

Elements of our causal graphs

Time indexing

In our causal diagrams, we implement two conventions for temporal order. First, the layout is structured left to right to reflect the sequence of causality. Second, we index nodes according to the relative timing of events. If $X_0$ precedes $X_1$, the indexing indicates this chronological order.

Representing uncertainty in timing

When the sequence of events is ambiguous (particularly in cross-sectional data), we use $X_{\varphi t}$ to propose a temporal order without clear time-specific measurements. We denote an event presumed to occur first as $X_{\varphi 0}$ and a subsequent event as $X_{\varphi 1}$.

Arrows

• Black arrows denote causality.
• Red arrows reveal an open backdoor path.
• Dashed black arrows denote attenuation.
• Red dashed arrows denote bias in a true causal association.
• A blue arrow with a circle point denotes effect-measure modification.
• $\mathcal{R}\to A$ denotes random treatment assignment.

Boxes

• A black box denotes conditioning that reduces confounding or is inert.
• A red box describes conditioning that introduces confounding bias.
• A dashed circle denotes a latent variable (not measured or not conditioned upon).

Terminology for conditional independence

• Statistical independence ($\coprod$): $A\coprod Y(a)$ means treatment assignment is independent of the potential outcomes.
• Statistical dependence ($\cancel{\coprod }$): $A\cancel{\coprod }Y(a)$ means treatment assignment is related to potential outcomes, potentially introducing bias.
• Conditioning ($|$): specifies contexts under which independence or dependence holds.
  • Conditional independence: $A\coprod Y(a)|L$ means that once we account for $L$, treatment and potential outcomes are independent.
  • Conditional dependence: $A\cancel{\coprod }Y(a)|L$ means potential outcomes and treatments are not independent after conditioning on $L$.

The Five Elementary Structures of Causality

Judea Pearl proved that all elementary structures of causality can be represented graphically (Pearl, 2009).

Two variables:

1. Causality absent: no causal effect between $A$ and $B$. They are statistically independent: $A\coprod B$.
2. Causality: $A$ causally affects $B$. They are statistically dependent: $A\cancel{\coprod }B$.

Three variables:

3. Fork: $A$ causally affects both $B$ and $C$. Variables $B$ and $C$ are conditionally independent given $A$: $B\coprod C|A$.
4. Chain: $C$ is affected by $B$, which is affected by $A$. Variables $A$ and $C$ are conditionally independent given $B$: $A\coprod C|B$. Here $B$ mediates the effect of $A$ on $C$.
5. Collider: $C$ is affected by both $A$ and $B$, which are independent. Conditioning on $C$ induces an association: $A\cancel{\coprod }B|C$.

Once we understand these basic relationships, we can build more complex causal diagrams. These structures help us see how statistical independences and dependencies emerge from the data, allowing us to clarify confounders so that $Y(a)\coprod A|L$.

The Three Identification Assumptions

Every causal inference, whether from an experiment or an observational study, rests on three assumptions. When all three hold, the causal effect of $A$ on $Y$ is identified: it can be estimated from the observed data.

These three assumptions form the foundation for everything that follows in this course. Causal diagrams help us evaluate whether the second assumption (conditional exchangeability) holds: if we can identify a set $L$ that blocks all backdoor paths between $A$ and $Y$, and if we condition on $L$, then conditional exchangeability is satisfied (given that our diagram is correct).

You might wonder: "If not from the data, where do our assumptions about causality come from?" Our assumptions are based on existing knowledge. Otto Neurath, an Austrian philosopher, used the metaphor of a ship rebuilt at sea:

We are like sailors who on the open sea must reconstruct their ship but are never able to start afresh from the bottom. Where a beam is taken away a new one must at once be put there, and for this the rest of the ship is used as support. In this way, by using the old beams and driftwood, the ship can be shaped entirely anew, but only by gradual reconstruction. (Neurath, 1973, p. 199)

The Four Rules of Confounding Control

1. Condition on common cause or its proxy. When $A$ and $Y$ share common causes, conditioning on these common causes blocks the open backdoor paths.

2. Do not condition on a mediator. When $L$ mediates $A\to Y$, conditioning on $L$ biases the total causal effect estimate.

3. Do not condition on a collider. When $L$ is a common effect of $A$ and $Y$, conditioning on $L$ induces a spurious association. For example, if marriage causes wealth and happiness causes wealth, conditioning on wealth induces an association between marriage and happiness even without a causal connection.

4. Proxy rule: conditioning on a descendant is akin to conditioning on its parent. When $L^{\prime }$ is an effect of $L$, conditioning on $L^{\prime }$ is approximately equivalent to conditioning on $L$. This can introduce bias (if $L$ is a collider) or reduce bias (if $L$ is an unmeasured common cause and $L^{\prime }$ is a measured proxy).

Lab materials: Lab 2: Install R and RStudio

Week 3: Causal Diagrams — The Structures of Confounding Bias

Date: 11 Mar 2026

Seminar

Learning outcomes

By the end of this week, you will be able to:

1. Define confounding bias formally and identify it in a causal diagram.
2. State and apply the backdoor criterion to determine which variables to condition on.
3. Explain why good regression model fit does not rule out confounding.
4. Distinguish confounding problems that longitudinal data can resolve from those it cannot.
5. Define M-bias and explain why conditioning on a pre-treatment collider introduces spurious associations.

What is confounding?

Last week we introduced the five elementary causal structures and the four rules of confounding control. This week we apply those tools to the central problem of observational causal inference: confounding.

Confounding exists when a common cause of both the treatment $A$ and the outcome $Y$ creates a non-causal (backdoor) path between them. If this path is left open, the observed association between $A$ and $Y$ conflates the true causal effect with a spurious correlation induced by the shared cause.

Consider a simple example. Suppose we want to know whether exercise ($A$) causes lower blood pressure ($Y$). Health consciousness ($L$) is a common cause: health-conscious people exercise more and eat better, which independently lowers blood pressure. The DAG is $A\leftarrow L\to Y$, with a direct arrow $A\to Y$. The path $A\leftarrow L\to Y$ is a backdoor path. If we do not condition on $L$, our estimate of the effect of $A$ on $Y$ will be inflated because it includes the spurious association transmitted through $L$.

The backdoor criterion

How do we know which variables to condition on? Pearl's backdoor criterion provides the answer.

The backdoor criterion translates the causal problem into a graphical one. Instead of reasoning about unobservable counterfactuals, we draw a DAG, list all backdoor paths, and check whether our measured covariates block them. If they do, the causal effect is identified. If they do not, we have unresolved confounding, and our estimate will be biased regardless of sample size or statistical sophistication.

Confounding and regression

Regression is the standard tool for "controlling for" confounders. When we write $Y={\beta }_0+{\beta }_{1}A+{\beta }_{2}L+\epsilon$, we are conditioning on $L$ and interpreting ${\beta }_1$ as the effect of $A$ on $Y$ holding $L$ constant.

A critical point for this course: good model fit does not indicate absence of confounding. A regression can explain 90% of the variance in $Y$ (high $R^2$) while conditioning on the wrong variables. If you condition on a mediator instead of a confounder, the model may fit well but the coefficient for $A$ will be biased. If you condition on a collider, the model may again fit well but will introduce a spurious association that was not there before. Model fit is a statistical property of the relationship between predictors and outcome. Confounding is a structural (causal) property of the data-generating process. The two are logically independent.

Confounding problems resolved by time-series data

Many confounding problems arise because cross-sectional data cannot distinguish cause from effect. When we measure $A$ and $Y$ at the same time, we cannot tell whether $A$ caused $Y$, $Y$ caused $A$, or a shared cause $L$ produced both. Longitudinal data collection, in which variables are measured at distinct time points, resolves several of these ambiguities.

The following figure presents seven confounding scenarios. In each case, the structural problem can be addressed by measuring the relevant variables in the correct temporal order: confounders before treatment, treatment before outcome.

The seven scenarios include reverse causation (where the apparent cause is actually the effect), confounding by a measured common cause (which can be conditioned on once it is measured before treatment), and cases where the temporal ordering of measurement clarifies which variables are mediators versus confounders. In each case, the solution is the same: measure confounders at baseline ($t_0$), treatment at wave 1 ($t_1$), and outcome at wave 2 ($t_2$). This three-wave panel design is the minimum structure for most observational causal questions in psychology.

Confounding problems not resolved by time-series data alone

Not all confounding problems can be solved by longitudinal data collection. The next figure presents six examples where time-series data are insufficient. Study these carefully before seminar.

M-bias is counterintuitive because $L$ is measured before treatment and appears to be a plausible confounder. The instinct to "control for everything measured at baseline" can introduce bias rather than remove it. The only way to diagnose M-bias is through a causal diagram: it is invisible in the data.

Other scenarios in the figure include residual confounding (where unmeasured variables affect changes in confounders over time even after baseline adjustment), treatment-confounder feedback (where past treatments affect future confounders, creating cycles that standard regression cannot handle), and intermediary confounding in mediation analysis (where a variable that confounds the mediator-outcome relationship is itself affected by treatment, making natural indirect effects unidentifiable).

Worked example: the assumptions in causal mediation

The following figure illustrates the assumptions required for causal mediation analysis. When we decompose the total effect of $A$ on $Y$ into a direct effect and an indirect effect through $M$, we need not only the standard three assumptions (causal consistency, conditional exchangeability, positivity) but also the absence of treatment-induced confounding of the mediator-outcome relationship.

If a confounder of the $M\to Y$ path is itself affected by treatment $A$, conditioning on it blocks part of the treatment effect (mediator bias) while failing to condition on it leaves confounding open. This dilemma has no solution within the standard regression framework and is one reason why causal mediation analysis requires stronger assumptions than total-effect estimation.

Lab materials: Lab 3: Regression, Graphing, and Simulation

Week 4: Interaction, Measurement Bias, and Selection Bias

Seminar

Learning outcomes

By the end of this week, you will be able to:

1. Distinguish effect modification from confounding and from interaction.
2. Identify the four types of measurement error bias in a causal diagram.
3. Explain how selection bias arises from conditioning on a collider at baseline.
4. Define the distinction between a target population and a source population, and explain why it matters for transportability.

Effect modification

Effect modification occurs when the causal effect of $A$ on $Y$ differs across strata of a third variable $V$. In our DAG conventions, effect modification is indicated by a blue arrow with a circle point from $V$ to the $A\to Y$ path. Importantly, effect modification is not a source of bias. It is a feature of the world: some treatments work differently for different people.

Effect modification differs from confounding. A confounder is a common cause of $A$ and $Y$ that must be conditioned on to eliminate bias. An effect modifier is a variable that changes the magnitude (or direction) of the causal effect. We do not adjust for effect modifiers to remove bias; we examine them to understand for whom the treatment works.

Effect modification also differs from statistical interaction, although the two are related. Interaction is a property of a statistical model (e.g., a product term $A\times V$ in a regression). Effect modification is a causal property of the data-generating process. A statistical interaction may reflect effect modification, confounding, or an artefact of the scale on which the outcome is measured. Distinguishing these requires a causal diagram.

Common causal questions presented as causal graphs

The following figure shows how several standard research designs can be represented as causal DAGs. Each design answers a different causal question, and the DAG makes explicit what is being estimated and what assumptions are required.

A typology of measurement error bias

Measurement error is pervasive in psychology. We rarely observe the true value of a variable; instead, we observe a measured version that may differ from the truth. When measurement error is systematic rather than purely random, it can introduce bias into our causal estimates. The following figure presents four structural types of measurement error, classified along two dimensions.

Understanding these types matters because the standard advice to "use reliable measures" addresses only the simplest case (independent, uncorrelated error). The other three types require structural reasoning: you need a DAG to determine whether measurement error in your study is a source of bias and, if so, in which direction.

Selection bias and transportability

Selection bias arises when the composition of the study sample differs systematically from the target population in ways that affect the treatment-outcome relationship. The most common structural source of selection bias is conditioning on a collider at baseline.

Consider a study of the effect of exercise on blood pressure. Suppose health-conscious people are more probable to both exercise and volunteer for health studies, and suppose high socioeconomic status (SES) independently predicts both better health and study participation. Participation is a collider of health consciousness and SES. By restricting analysis to study participants, we condition on this collider and induce a spurious association between health consciousness and SES, which can distort the estimated effect of exercise on blood pressure.

The acronym "WEIRD" (Western, Educated, Industrialised, Rich, Democratic) has drawn attention to the narrow demographic base of much psychological research. From a causal perspective, the problem is not that WEIRD samples are unrepresentative per se, but that effect modifiers may be distributed differently in the target population. If the treatment effect varies across levels of an effect modifier, and that modifier's distribution differs between sample and target, the sample ATE will not transport to the target. The simulation guide for this course demonstrates this point concretely using a transportability simulation.

Measurement invariance (which we cover in Week 10) connects to measurement error bias in cross-cultural research. If the same questionnaire measures a construct differently in different populations, then what appears to be a cross-population difference in the outcome may instead be a difference in how the outcome is measured. This is a form of correlated, directed measurement error, and it cannot be detected by standard reliability statistics.

Lab materials: Lab 4: Regression and Confounding Bias

Week 5: Causal Inference: Average Treatment Effects

Date: 25 Mar 2026

Seminar

Learning Outcomes

• You will understand why causation is never directly observed.
• You will understand how experiments address this "causal gap."
• You will understand how applying three principles from experimental research allows human scientists to close this "causal gap" when making inferences about a population as a whole, that is, inferences about "marginal effects."

Opening

Introduction: The Estrogen Therapy Paradox

Motivating Example

Consider the following cross-cultural question:

Does bilingualism improve cognitive abilities in children?

There is evidence that bilingual children perform better at cognitive tasks, but is this improvement truly caused by learning more than one language, or is it confounded by other factors (e.g., cultural environment, parental influence)? How can we know? Each child might answer, like the traveller in Frost's poem:

"And sorry I could not travel both. And be one traveler $\dots$"

Part 1: The Fundamental Problem of Causal Inference as a Missing Data Problem

The fundamental problem of causal inference is that causality is never directly observed.

Let $Y$ and $A$ denote random variables.

We formulate a causal question by asking whether experiencing an exposure $A$, when this exposure is set to level $A=a$, would lead to a difference in $Y$, compared to what would have occurred had the exposure been set to a different level, say $A=a^{\prime }$ will lead to a difference in outcome $Y$. For simplicity, we imagine binary exposure such that $A=1$ denotes receiving the "bilingual" exposure and $A=0$ denotes receiving the "monolingual" exposure. Assume these are the only two exposures of interest:

Let:

• $Y_i(a=1)$ denote the cognitive ability of child $i$ if the child were bilingual (potential outcome when $A_i=1$).
• $Y_i(a=0)$ denote the cognitive ability of child $i$ if the child were monolingual (potential outcome when $A_i=0$).

What does it mean to quantify a causal effect? We may define the individual-level causal effect of bilingualism on cognitive ability for child $i$ as the difference between two states of the world: one for which the child experiences a bilingual exposure and the other for which the child does not. We write this contrast by referring to the potential outcomes under different levels of exposure:

$${\text{Causal Effect}}_i=Y_i(1)-Y_i(0).$$

We say there is a causal effect of the bilingual exposure if

$$Y_i(1)-Y_i(0)\ne 0.$$

Because each child experiences only one exposure condition in reality, we cannot directly compute this difference from any dataset. The missing observation is called the counterfactual:

• If $Y_i|A_i=1$ is observed, then $Y_i(0)|A_i=1$ is counterfactual.
• If $Y_i|A_i=0$ is observed, then $Y_i(1)|A_i=1$ is counterfactual.

"And sorry I could not travel both / And be one traveler, long I stood $\dots$"

In short, individuals cannot simultaneously experience both exposure conditions, so one outcome is inevitably missing.

How can we make contrasts between counterfactual (potential) outcomes?

Fundamental Assumption 1: Causal Consistency

Causal consistency means that the potential outcome corresponding to the exposure an individual actually receives is exactly what we observe. In other words, if individual $i$ receives exposure $a$, then the potential outcome (or equivalently the counterfactual outcome under a given level of exposure $A=a$, that is $Y_i(a)$) is equivalent to the observed outcome: $Y_i\mid A_i\equiv a$. Where the symbol $\equiv$ means "equivalent to", when we assume that the causal consistency assumption is satisfied, we assume that:

$$\begin{array}{rl}\underset{\text{counterfactual}}{\underbrace{Y_i(1)}}& \equiv \underset{\text{observable}}{\underbrace{(Y_i\mid A_i=1)}},\\ \underset{\text{counterfactual}}{\underbrace{Y_i(0)}}& \equiv \underset{\text{observable}}{\underbrace{(Y_i\mid A_i=0)}}.\end{array}$$

Notice however that we cannot generally obtain individual causal effects because at any given time, each individual may only receive at most one level of an exposure. Where the symbol $⟹$ means "implies," at any given time, receiving one level of an exposure precludes receiving any other level of that exposure:

$$Y_i|A_i=1⟹Y_i(0)|A_i=1\text{is counterfactual}$$

Likewise:

$$Y_i|A_i=0⟹Y_i(1)|A_i=1\text{is counterfactual}$$

Because of the laws of physics (above the atomic scale), an individual can experience only one exposure level at any moment. Consequently, we can observe only one of the two counterfactual outcomes needed to quantify a causal effect. This is the fundamental problem of causal inference. Counterfactual contrasts cannot be individually observed.

However, because of the causal consistency assumption, we can nevertheless recover half of the missing counterfactual (or "potential") outcomes needed to estimate average treatment effects. We may do this if two other assumptions are satisfied.

Fundamental Assumption 2: Exchangeability

Exchangeability justifies recovering unobserved counterfactuals from observed outcomes and averaging them. By accepting that $Y_i(a)=Y_i$ if $A_i=a$, we can estimate population-level average potential outcomes. In an experiment where exposure groups are comparable, we define the Average Treatment Effect (ATE) as:

$$\begin{array}{rl}\text{ATE}& =\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]\\ & =\mathbb{E}(Y\mid A=1)-\mathbb{E}(Y\mid A=0).\end{array}$$

Because randomisation ensures that missing counterfactuals are exchangeable with those observed, we can still estimate $\mathbb{E}[Y(a)]$. For instance, assume:

$$\underset{\text{counterfactual}}{\underbrace{\mathbb{E}[Y(1)\mid A=1]}}=\underset{\text{unobservable}}{\underbrace{\mathbb{E}[Y(1)\mid A=0]}}=\underset{\text{observed}}{\underbrace{(Y_i\mid A_i=1)}}$$

which lets us infer the average outcome if everyone were treated. Likewise, if

$$\underset{\text{counterfactual}}{\underbrace{\mathbb{E}[Y(0)\mid A=0]}}=\underset{\text{unobservable}}{\underbrace{\mathbb{E}[Y(1)\mid A=0]}}=\underset{\text{observed}}{\underbrace{\mathbb{E}(Y\mid A_i=0)}}$$

then we can infer the average outcome if everyone were given the control. The difference between these two quantities gives the ATE:

$$\text{ATE}=[\stackrel{\begin{array}{c}\text{by consistency:}\\ \equiv \text{ observed }\mathbb{E}[Y\mid A=1]\end{array}}{\overbrace{\mathbb{E}[Y(1)\mid A=1]}}+\stackrel{\begin{array}{c}\text{by exchangeability:}\\ \text{unobservable, yet }\equiv \mathbb{E}[Y\mid A=1]\end{array}}{\overbrace{\mathbb{E}[Y(1)\mid A=0]}}]-[\stackrel{\begin{array}{c}\text{by consistency:}\\ \equiv \text{observed }\mathbb{E}[Y\mid A=0]\end{array}}{\overbrace{\mathbb{E}[Y(0)\mid A=0]}}+\stackrel{\begin{array}{c}\text{by exchangeability:}\\ \text{unobservable, yet }\equiv \mathbb{E}[Y\mid A=0]\end{array}}{\overbrace{\mathbb{E}[Y(0)\mid A=1]}}]$$

We have it that $\mathbb{E}[Y\mid A=1]$ and $\mathbb{E}[Y\mid A=0]$ and $\mathbb{E}[Y(1)\mid A=0]$ are observed. If both consistency and exchangeability are satisfied then we may use these observed quantities to identify contrasts of counterfactual quantities.

Thus, although individual-level counterfactuals are missing, the consistency assumption and the exchangeability assumption allow us to identify the average effect of treatment using observed data. Randomised controlled experiments allow us to meet these assumptions. Randomisation warrants the exchangeability assumption. Control warrants the consistency assumption.

Fundamental Assumption 3: Positivity

There is one further assumption, called positivity. It states that treatment assignments cannot be deterministic. That is, for every covariate pattern $L=l$, each individual has a non-zero probability of receiving every treatment level to be compared:

$$P(A=a\mid L=l)>0.$$

Randomised experiments achieve positivity by design, at least for the sample that is selected into the study. In observational settings violations occur if some subgroups never receive a particular treatment. If treatments occur but are rare, we may have sufficient data from which to obtain convincing causal inferences.

Positivity is the only assumption that can be verified with data. We will consider how to assess this assumption using data when we develop our data analytic workflows in the second half of this course.

Challenges with Observational Data

1. Satisfying Causal Consistency is Difficult in Observational Settings

Below are some ways in which real-world complexities can violate causal consistency in observational studies. Causal consistency requires there is no interference between units (also called "SUTVA" or "Stable Unit Treatment Value"). Causal consistency also requires that each treatment level is well-defined and applied uniformly. If these conditions fail, then $Y(a)$ may not reflect a consistent exposure across individuals. We are then comparing apples with oranges. Consider some examples:

1. Cultural differences: one group's "second-language exposure" may differ qualitatively from another's if cultural norms shape how, when, or by whom the second language is taught. A child in a bilingual community may receive diverse and immersive language experiences all of which are coded as $A=1$. Yet the treatments might be quite different. We might be comparing apples with oranges.

2. Age of acquisition: the developmental effect of learning a second language may vary by when the child is exposed. Comparing acquisition at, say, age two with acquisition at, say, age twelve might be comparing apples with oranges.

3. Language variation: sign languages, highly tonal languages, or unwritten languages may demand different cognitive tasks than spoken, nontonal, or widely documented languages. Lumping them together as "learning a second language" can obscure the fact that these distinct exposures might produce fundamentally different outcomes. Again, comparisons here would be of apples with oranges.

These sources of heterogeneity reveal why careful delineation of treatments is crucial. If the actual exposures differ across individuals, then consistency $(Y_i(a)=Y_i\mid A_i=a)$ may fail, because $A=a$ is not the same phenomenon for everyone.

2. Conditional Exchangeability (No Unmeasured Confounding) Is Difficult to Achieve

In theory, we can identify a causal effect from observational data if all confounders $L$ are measured. Formally, we need the potential outcomes to be independent of treatment once we condition on $L$. One way to express this assumption is: $Y(a)\coprod A\mid L$. If the potential outcomes are independent of treatment assignment, we can identify the Average Treatment Effect (ATE) as:

$$\text{ATE}=\sum _l\left[\mathbb{E}\right(Y\mid A=1,L=l)-\mathbb{E}(Y\mid A=0,L=l\left)\right]\mathrm{Pr}(L=l).$$

In randomised experiments, conditioning is automatic because $A$ is unrelated to potential outcomes by design. In observational studies, ensuring or approximating such conditional exchangeability is often difficult. For example, bilingualism research would need to consider:

• Cultural histories: cultures that value language acquisition might also value knowledge acquisition. Associations might arise from culture, not causation.
• Personal values: families who place a high priority on bilingualism may also promote other developmental resources.

If important confounders go unmeasured or are poorly measured, these differences can bias causal effect estimates.

3. The Positivity Assumption May Fail: Treatments Might Not Exist for All

Positivity requires that each individual could, in principle, receive any exposure level. In real-world observational settings, some groups have no access to bilingual education (or no reason to be monolingual), making certain treatment levels impossible for them. If a treatment level does not appear in the data for a given subgroup, any causal effect estimate for that subgroup is purely an extrapolation (Westreich & Cole, 2010; Hernan & Robins, 2023).

Summary

We introduced the fundamental problem of causal inference by distinguishing correlation (associations in the data) from causation (contrasts between potential outcomes, of which only one can be observed for each individual).

Randomised experiments address this problem by balancing confounding variables across treatment levels. Although individual causal effects are unobservable, random assignment allows us to infer average causal effects, also called marginal effects.

In observational data, inferring average treatment effects demands that we satisfy three assumptions that are automatically satisfied in (well-conducted) experiments: causal consistency, exchangeability, and positivity. These assumptions ensure that we can compare like-with-like (that the population-level treatment effect is consistent across individuals), that there are no unmeasured common causes of the exposure and outcomes that may lead to associations in the absence of causality, and that every exposure level is a real possibility for each subgroup.

Lab materials: Lab 5: Average Treatment Effects

Appendix A: Notation Variants

Consider that in the causal inference literature, we may write the contrast of two potential outcomes under treatment as:

$${\text{Causal Effect}}_i=Y_i^{a=1}-Y_i^{a=0}$$

or

$${\text{Causal Effect}}_i=Y_i^1-Y_i^0$$

or:

$${\text{Causal Effect}}_i=Y_1-Y_0$$

Where subscripts are dropped. You will soon encounter many notational variants across sources.

Week 6: Effect Modification and CATE

Date: 2 Apr 2026

If you learn nothing else from this course...

To answer psychological questions properly, we first need to state them very clearly. Causal inference gives us the tools to do this.

Causal inference asks: "What if?"

The core idea is to compare what actually happened with what could have happened under different conditions (these "what ifs" are called counterfactuals).

Imagine an outcome we care about, like student test scores ($Y$). We might compare the score if everyone got a new teaching method (let's call this condition $a^{\ast }$) versus if everyone got the old method (condition $a$).

The difference in the potential outcome ($Y$) under these two scenarios is the causal effect for one person: $Y(a^{\ast })-Y(a)$.

Since we can't see both scenarios for the same person, we often look at the average effect across a group. The Average Treatment Effect (ATE) is the average difference in potential outcomes across the whole population:

$$\text{ATE}=\mathbb{E}[Y(a^{\ast })-Y(a)]$$

This asks: "On average, how much would scores change if we switched everyone from the old method ($a$) to the new method ($a^{\ast }$)?".

A big challenge is dealing with confounders, other factors that mix up the relationship between the treatment ($A$) and the outcome ($Y$), potentially misleading us about the true causal effect. We need to account for these.

What do 'Interaction' and 'Effect Modification' mean in Causal Inference?

Words like 'moderation' and 'interaction' are often used loosely. Causal inference needs precise terms.

We'll focus on two specific ideas:

1. Interaction: About the effect of combining interventions.
2. Effect Modification: About how the effect of one intervention changes for different types of people.

Interaction: The Effect of Teamwork (or Lack Thereof)

Interaction in causal inference is about joint interventions. We look at what happens when we apply two or more different treatments at the same time.

Let's say we have two treatments, $A$ and $B$, and an outcome $Y$.

• $Y(a)$ is the potential outcome if we set treatment $A$ to level $a$.
• $Y(b)$ is the potential outcome if we set treatment $B$ to level $b$.
• $Y(a,b)$ is the potential outcome if we set $A$ to $a$ and $B$ to $b$ simultaneously.

To figure out these effects from observational data, we usually need assumptions like:

• Consistency: The outcome we see for someone who got treatment $a$ is the same as their potential outcome $Y(a)$.
• Conditional Exchangeability (No Unmeasured Confounding): We can make the groups receiving different treatments comparable by adjusting for measured confounders ($L$ for $A\to Y$, and $Q$ for $B\to Y$). The sets $L$ and $Q$ might overlap.
• Positivity: The exposures to be compared occur in all subgroups.

To study the interaction between $A$ and $B$, we need to be able to estimate the effect of $A$ and the effect of $B$, which means we need to adjust for all confounders in both $L$ and $Q$ (i.e., their union $L\cup Q$).

Defining Interaction: Does 1 + 1 = 2?

Let's use an education example:

• $A$: New teaching method (1=New, 0=Old)
• $B$: Extra tutoring (1=Yes, 0=No)
• $Y$: Test score

Is the boost in scores from getting both the new method and tutoring simply the sum of the boost from only the new method and the boost from only tutoring?

We define causal interaction on the additive scale (looking at differences) by comparing the effect of the joint intervention to the sum of the individual effects (all compared to getting neither):

$$\underset{\text{Effect of Both}}{\underbrace{(\mathbb{E}[Y(1,1)]-\mathbb{E}[Y(0,0)])}}\text{vs}\underset{\text{Effect of A only}}{\underbrace{(\mathbb{E}[Y(1,0)]-\mathbb{E}[Y(0,0)])}}+\underset{\text{Effect of B only}}{\underbrace{(\mathbb{E}[Y(0,1)]-\mathbb{E}[Y(0,0)])}}$$

Interaction exists if these are not equal. This simplifies to checking if the following is non-zero (see Appendix A):

$$\underset{\text{Both}}{\underbrace{\mathbb{E}[Y(1,1)]}}-\underset{\text{A only}}{\underbrace{\mathbb{E}[Y(1,0)]}}-\underset{\text{B only}}{\underbrace{\mathbb{E}[Y(0,1)]}}+\underset{\text{Neither}}{\underbrace{\mathbb{E}[Y(0,0)]}}\ne 0$$

• If this is positive: Synergy (the combination is better than expected).
• If this is negative: Antagonism (the combination is worse than expected).

(We could also look at interaction on other scales, like ratios, which might give different answers. Always state the scale you're using; we'll come back to this in later lectures.)

Finding Causal Interaction in Data

To estimate this interaction, we need valid estimates for all four average potential outcomes:

$$\mathbb{E}[Y(0,0)],\mathbb{E}[Y(1,0)],\mathbb{E}[Y(0,1)],\mathbb{E}[Y(1,1)]$$

This means we must control for confounders of both the $A\to Y$ link and the $B\to Y$ link.

The following figure shows this. $L_A$ are confounders for $A\to Y$, and $L_B$ are confounders for $B\to Y$. We need to block the backdoor paths (red arrows).

The next figure shows we need to condition on (adjust for) both $L_A$ and $L_B$.

In our education example:

• $L_A$ (Confounders for Teaching Method $\to$ Score): Prior achievement, motivation, family background (SES), school quality, teacher differences (if not randomly assigned).
• $L_B$ (Confounders for Tutoring $\to$ Score): Prior achievement, motivation, family background (SES, paying for tutoring), student availability, specific learning needs.

Notice that prior achievement and motivation are in both $L_A$ and $L_B$. We need to measure and adjust for all important factors in $\boxed{L_A}$ and $\boxed{L_B}$ to get a reliable estimate for interaction.

Effect Modification: Different Effects for Different People

Unlike interaction (about combining treatments), effect modification is about whether the causal effect of a single intervention ($A$) on an outcome ($Y$) is different for different subgroups in the population. These subgroups are defined by baseline characteristics (like age, sex, prior history; let's call these $G$ or $X$).

Effect modification helps us understand who benefits most (or least) from an intervention. We explore this using ideas like Heterogeneous Treatment Effects (HTE) and Conditional Average Treatment Effects (CATE).

Heterogeneous Treatment Effects (HTE): The Idea of Variation

Heterogeneous Treatment Effects (HTE) just means that the effect of a treatment ($A$ on $Y$) isn't the same for everyone. The effect varies. This variation is effect modification.

Why does it vary?

• Differences in things we can measure (like age, sex, baseline health, our $X$ variables).
• Differences in things we can't easily measure (like genetics, unmeasured background factors).

HTE is the reality; treatments rarely work identically for all.

Conditional Average Treatment Effect (CATE): Measuring Variation with Data $\tau (x)$

To study HTE using data, we focus on the Conditional Average Treatment Effect (CATE). CATE is a specific causal question (estimand): What is the average treatment effect for the subgroup of people who share specific measured characteristics $X=x$?

$$\tau (x)=\text{CATE}(x)=\mathbb{E}[Y(1)-Y(0)|X=x]$$

Here, $Y(1)$ is the potential outcome with treatment, $Y(0)$ without. $\tau (x)$ tells us the average effect specifically for people with characteristics $X=x$. By looking at how $\tau (x)$ changes for different $x$, we quantify effect modification by the characteristics we measured in X.

Comparing Effects Across Defined Groups

A simple way to check for effect modification by a category $G$ (like comparing males vs females, or different locations) is to estimate the Average Treatment Effect (ATE) separately within each group. This is like comparing CATEs where $X$ is just the group variable $G$.

Let's say $A$ is the treatment (0=control, 1=treated) and $G$ is the potential modifier (e.g., $g=$female, $g^{\prime }=$male).

We compare:

1. The average effect for females ($G=g_1$): ${\delta }_{g_1}=\mathbb{E}[Y(1)|G=g_1]-\mathbb{E}[Y(0)|G=g_1]$
2. The average effect for males ($G=g_2$): ${\delta }_{g_2}=\mathbb{E}[Y(1)|G=g_2]-\mathbb{E}[Y(0)|G=g_2]$

Effect modification by $G$ exists if these are different: $\gamma ={\delta }_{g_1}-{\delta }_{g_2}\ne 0$.

If our estimate $\hat{\gamma }$ is far from zero, it suggests the treatment effect differs between males and females.

Finding Effect Modification in Data

To estimate these group-specific effects (${\delta }_g$) and their difference ($\gamma$) correctly, we need to control for confounders ($L$) of the $A\to Y$ relationship within each group defined by $G$. Note that we are not estimating the causal effect of $G$. As such, we do not need to control for things that cause $G$ itself, unless they also confound the $A\to Y$ relationship (i.e., are also in $L$).

Look at the following figure. To estimate the $A\to Y$ effect within levels of $G$, we need to adjust for the confounders $L_0$. However, this will partially block the effect-modification of $G$ on $Y$ because $L_0$ is a mediator for that path. Moreover, if we were identifying the causal effect of $G$ on $Y$, after conditioning on $L$, we would find that a backdoor path opens from $G\to Y$ because $\boxed{L}$ is a collider. $G$ does not have a causal interpretation in this model. However we would be wrong to think that $G$ is not an effect modifier of the effect of $A$ on $Y$. (See Appendix C.)

Thus it is essential to understand that when we control for confounding along the $A\to Y$ path, we do not identify the causal effects of effect-modifiers.

To clarify:

1. If the statistical model correctly identifies causal effect modification (by appropriately handling confounders and avoiding collider bias), then it has prognostic value regarding the differential outcomes expected under intervention $A=1$ vs $A=0$ depending on $G=g$.

2. If the statistical model contains interaction terms that are artefacts of bias (like conditioning on a collider) or reflect a different target (like conditioning on a mediator when the total effect was intended), its causal prognostic value is compromised or needs careful interpretation. It might still predict $Y$ well given $A$, $G$, and $L$ in an observational setting, but it wouldn't accurately predict the results of intervening on A differently for different $G$ groups. (See Appendix B.)

The choice of variables fundamentally determines which causal question (if any) the statistical model is estimating. As with the average treatment effect, we interpret evidence for effect modification in the context of our assumptions about the causal relationships that obtain in the world. This is because the statistical interaction we observe is highly sensitive to model choices. Any interpretation as causal effect modification, and therefore any reliable prognostic value for intervention effects within different segments of the population, depends entirely on whether and how our statistical model accounts for the causal structure.

Estimating How Effects Vary: Getting $\hat{\tau }(x)$ from Data

We defined the Conditional Average Treatment Effect (CATE), $\tau (x)$, as the true average effect for a subgroup with specific features $X=x$:

$$\tau (x)=\mathbb{E}[Y(1)-Y(0)|X=x]$$

Now, we want to estimate this from our actual data. We call our estimate $\hat{\tau }(x)$. For any person $i$ in our study with features $X_i$, the value $\hat{\tau }(X_i)$ is our data-based prediction of the average treatment effect for people like person i.

"Personalised" Effects vs. True Individual Effects

Wait: didn't we say we can't know the true effect for one specific person, $Y_i(1)-Y_i(0)$? Yes, that's still true.

So what does $\hat{\tau }(X_i)$ mean?

• Individual Causal Effect (Unknowable): $Y_i(1)-Y_i(0)$. This is the true effect for person $i$. We can't observe both $Y_i(1)$ and $Y_i(0)$.
• Estimated CATE ($\hat{\tau }(X_i)$) (What we calculate): This is our estimate of the average effect, $\mathbb{E}[Y(1)-Y(0)]$, for the subgroup of people who share the same measured characteristics $X_i$ as person $i$.

When people talk about "personalised" or "individualised" treatment effects in this context, they usually mean $\hat{\tau }(x)$. It's "personalised" because the prediction uses person $i$'s specific characteristics $X_i=x$. But remember, it's an estimated average effect for a group, not the unique effect for that single individual.

People Have Many Characteristics

People aren't just in one group; they have many features at once. A student might be:

• Female
• 21 years old
• From a low-income family
• Did well on previous tests
• Goes to a rural school
• Highly motivated

All these factors ($X_i$) together might influence how they respond to a new teaching method.

Trying to figure this out with traditional regression by manually adding interaction terms (like A*gender*age*income*...) becomes impossible very quickly:

• Too many combinations, not enough data in each specific combo.
• High risk of finding "effects" just by chance (false positives).
• Hard to know which interactions to even include.
• Can't easily discover unexpected patterns.

Thus, while simple linear regression with interaction terms (lm(Y ~ A * X1 + A * X2)) can estimate CATEs if the model is simple and correct, it often fails when things get complex (many $X$ variables, non-linear effects).

Causal forests (using the grf package in R; Athey, Tibshirani, and Wager, 2024) are a powerful, flexible alternative designed for this task. They build decision trees that specifically aim to find groups with different treatment effects.

Demo: Why Functional Form Matters (Even Without Confounding)

The following simulation makes this concrete. Treatment is randomised, so there is no confounding at all. The only question is whether each method can recover the true effect surface $\tau (x)$, which is deliberately non-linear (it includes sinusoidal, threshold, and interaction components that regression cannot capture).

# install once
# remotes::install_github("go-bayes/causalworkshop@v0.2.1")
library(causalworkshop)

# simulate data with non-linear heterogeneous effects
d <- simulate_nonlinear_data(n = 2000, seed = 2026)

# compare four estimation methods
results <- compare_ate_methods(d)

# summary table: ATE and individual-level RMSE
results$summary

# plot: predicted vs true treatment effects
results$plot_comparison

# plot: estimated effect as a function of x1
results$plot_by_x1

All four methods recover the correct ATE (because treatment is randomised, a simple difference in means suffices). The RMSE column tells a different story: it measures how well each method recovers the true effect for each individual. OLS with linear interactions compresses heterogeneity to a flat plane. Polynomial regression adds curvature but remains constrained by its parametric form. The GAM and causal forest learn the effect surface non-parametrically and achieve substantially lower RMSE.

The lesson: when treatment effects vary non-linearly across individuals, the functional form you assume determines whether you can detect that variation, even in a randomised experiment. Regression does not fail because of confounding here; it fails because it assumes a shape that the data do not follow.

We'll learn how to use grf after the mid-term break. It will allow us to get the $\hat{\tau }(x)$ predictions and then think about how to use them, for instance, to prioritise who gets a treatment if resources are limited.

Summary

Let's revisit the core concepts.

Interaction:

• Think: Teamwork effect.
• What: Effect of two or more different interventions ($A$ and $B$) applied together.
• Question: Is the joint effect $\mathbb{E}[Y(a,b)]$ different from the sum of individual effects?
• Needs: Control confounders for all interventions involved ($L\cup Q$).

Effect Modification / HTE / CATE:

• Think: Different effects for different groups.
• What: Effect of a single intervention ($A$) varies depending on people's baseline characteristics ($G$ or $X$).
• Question (HTE): Does the effect vary? (The phenomenon.)
• Question (CATE $\tau (x)$): What is the average effect for a specific subgroup with features $X=x$? (The measure.)
• Needs: Control confounders for the single intervention ($L$) within subgroups.

Estimated "Individualised" Treatment Effects ($\hat{\tau }(x)$):

• Think: Personal profile prediction.
• What: Our estimate of the average treatment effect for the subgroup of people sharing characteristics $X_i$.
• How: Calculated using models (like causal forests) that use the person's full profile $X_i$.
• Important: This is not the true effect for that single person (which is unknowable). It's an average for people like them.
• Use: Explore HTE, identify subgroups, potentially inform targeted treatment strategies.

Keeping these concepts distinct helps us ask clear research questions and choose the right methods.

Course Review So Far: A Quick Recap

Let's quickly review the main ideas of causal inference we've covered.

The Big Question: Does A cause Y?

Causal inference helps us answer if something (like a teaching method, $A$) causes a change in something else (like test scores, $Y$).

Core Idea: "What If?" (Counterfactuals)

We compare what actually happened to what would have happened in a different scenario.

• $Y(1)$: Score if the student had received the new method.
• $Y(0)$: Score if the student had received the old method.

The Average Treatment Effect (ATE) = $\mathbb{E}[Y(1)-Y(0)]$ is the average difference across the whole group.

This Lecture Clarified Concepts of Interaction vs. Effect Modification vs. Individual Predictions

Interaction (Think: Teamwork Effects)

• About: Combining two different interventions (A and B).
• Question: Does using both A and B together give a result different from just adding up their separate effects? (e.g., new teaching method + tutoring).
• Needs: Analyse effects of A alone, B alone, and A+B together. Control confounders for both A and B.

Effect Modification (Think: Different Effects for Different Groups)

• About: How the effect of one intervention (A) changes based on people's characteristics (X, like prior grades).
• Question: Does the teaching method (A) work better for high-achieving students (X=high) than low-achieving students (X=low)?
  • HTE: The idea that effects differ.
  • CATE $\tau (x)$: The average effect for the specific group with characteristics $X=x$.
• Needs: Analyse effect of A within different groups (levels of X). Control confounders for A.

Estimated Individualised Effects ($\hat{\tau }(X_i)$) (Think: Personal Profile Prediction)

• About: Using a person's whole profile of characteristics ($X_i$, age, gender, background, etc.) to predict their probable response to treatment A.
• How: Modern methods (like causal forests) take all of $X_i$ and estimate $\hat{\tau }(X_i)$.
• Result: this $\hat{\tau }(X_i)$ is not the true unknowable effect for person $i$. It is the estimated average effect for people similar to person i (sharing characteristics $X_i$).
• Use: helps explore if tailoring treatment based on these profiles ($X_i$) could be beneficial.

Simple Summary

• Interaction: Do A and B work together well/badly?
• Effect Modification: Does A's effect depend on who you are (based on X)?
• $\hat{\tau }(X_i)$: Can we predict A's average effect for someone based on their specific profile $X_i$?

Understanding these differences is key to doing good causal research.

Lab materials: Lab 6: CATE and Effect Modification

Appendix A: Simplification of Additive Interaction Formula

We start with the definition of additive interaction based on comparing the joint effect relative to baseline versus the sum of individual effects relative to baseline:

$$(\mathbb{E}[Y(1,1)]-\mathbb{E}[Y(0,0)])-\left[\right(\mathbb{E}[Y(1,0)]-\mathbb{E}[Y(0,0)])+(\mathbb{E}[Y(0,1)]-\mathbb{E}[Y(0,0)]\left)\right]$$

First, distribute the negative sign across the terms within the square brackets:

$$\mathbb{E}[Y(1,1)]-\mathbb{E}[Y(0,0)]-(\mathbb{E}[Y(1,0)]-\mathbb{E}[Y(0,0)])-(\mathbb{E}[Y(0,1)]-\mathbb{E}[Y(0,0)])$$

Now remove the parentheses, flipping the signs inside them where preceded by a minus sign:

$$\mathbb{E}[Y(1,1)]-\mathbb{E}[Y(0,0)]-\mathbb{E}[Y(1,0)]+\mathbb{E}[Y(0,0)]-\mathbb{E}[Y(0,1)]+\mathbb{E}[Y(0,0)]$$

Next, combine the $\mathbb{E}[Y(0,0)]$ terms:

• We have $-\mathbb{E}[Y(0,0)]$
• Then $+\mathbb{E}[Y(0,0)]$ (these two cancel each other out)
• And another $+\mathbb{E}[Y(0,0)]$ remains.

The expression simplifies to:

$$\mathbb{E}[Y(1,1)]-\mathbb{E}[Y(1,0)]-\mathbb{E}[Y(0,1)]+\mathbb{E}[Y(0,0)]$$

This is the standard definition of additive interaction, often called the interaction contrast. If this expression equals zero, there is no additive interaction; a non-zero value indicates an interaction effect.

This shows clearly that interaction is the deviation of the joint effect from the sum of the separate effects, adjusted for the baseline.

Appendix B: Evidence for Effect-Modification is Relative to Inclusion of Other Variables in the Model

The 'sharp-null hypothesis' states there is no effect of the exposure on the outcome for any unit in the target population. Unless the sharp-null hypothesis is false, there may be effect-modification. For any study worth conducting, we cannot evaluate whether the sharp-null hypothesis is false. If we could, the experiment would be otiose. Therefore, we must assume the possibility of effect-modification. Whether a variable is an effect-modifier also depends on which other variables are included in the model. That is, just as for the concept of a 'confounder', where a variable is an 'effect-modifier' cannot be stated without reference to an assumed causal order and an explicit statement about which other variables will be included in the model (VanderWeele, 2012).

As illustrated in the following figure, the marginal association between $A$ and $Y$ is unbiased. Here, exposure $A$ is unconditionally associated with $Y$. We use the convention that a solid blue box (e.g., $\boxed{G}$) denotes effect-modification with conditioning and a dashed blue circle (e.g., Z in a dashed circle) indicates effect-modification without conditioning.

The next figure presents the same randomised experiment as in the previous causal diagram. We again assume that there is no confounding of the marginal association between the exposure, $A$, and the outcome, $Y$. However, suppose we were to adjust for $Z$ and ask: does the conditional association of $A$ on $Y$ vary within levels of $G$, after adjusting for $Z$? That is, does $G$ remain an effect-modifier of the exposure on the outcome? VanderWeele (2007) proved that for effect-modification to occur, at least one other arrow besides the treatment must enter into the outcome. According to the diagram, the only arrow into $Y$ other than $A$ arrives from $Z$. Because $Y$ is independent of $G$ conditional on $Z$, we may infer that $G$ is no longer an effect modifier for the effect of $A$ on $Y$. Viewed another way, $G$ no longer co-varies with $Y$ conditional on $Z$ and so cannot act as an effect-modifier.

The following figure presents the same randomised experiment as in the previous graph. We assume a true effect of $A\to Y$. If we do not condition on $B$, then $G$ will not modify the effect of $A\to Y$ because $G$ will not be associated with $Y$. However, if we were to condition on $B$, then both $B$ (an effect modifier by proxy) and $G$ may become effect-modifiers for the causal effect of $A$ on $Y$. In this setting, both $B$ and $G$ are conditional effect-modifiers.

Note that causal graphs help us to evaluate classifications of conditional and unconditional effect modifiers. They may also help to clarify conditions in which conditioning on unconditional effect-modifiers may remove conditional effect-modification. However, we cannot tell from a causal diagram whether the ancestors of an unconditional effect-modifier will be conditional effect-modifiers for the effect of the exposure on the outcome; see VanderWeele (2007), also Suzuki et al. (2013). Causal diagrams express non-parametric relations. I have adopted an off-label colouring convention to denote instances of effect-modification to highlight possible pathways for effect-modification, which may be relative to other variables in a model.

The next figure reveals the relativity of effect-modification. If investigators do not condition on $B$, then $G$ cannot be a conditional effect-modifier because $G$ would then be independent of $Z$ because $B$ is a collider. However, as we observed in the previous figure, conditioning on $B$ (a collider) may open a path for effect-modification of $G$ by $Z$. Both $B$ and $G$ are conditional effect modifiers.

The final figure considers the implications of conditioning on $Z$, which is the only unconditional effect-modifier on the graph. If $Z$ is measured, conditioning on $Z$ will remove effect-modification for $B$ and $G$ because $B,G\perp \perp Y|Z$. This example again reveals the context dependency of effect-modification. Here, causal diagrams are useful for clarifying features of dependent and independent effect modification. For further discussion, see Suzuki et al. (2013) and VanderWeele (2009).

Appendix C: Further Clarification on Effect Modification Without Statistical Evidence for It

Again, look at the effect modification DAG from the seminar. Suppose the investigator models the effect of $A$ on $Y$. Suppose $G$ is not associated with $Y$ conditional on $L$. The DAG provides structural clarification for why concluding there is no effect modification by $G$ is incorrect.

To clarify:

• We have a DAG structure: $G\to L$; $L\to A$, $L\to Y$
• We're interested in the conditional average treatment effect (CATE): $\tau (g)=E[Y(1)-Y(0)|G=g]$
• To identify the effect of $A$ on $Y$, we must condition on $\boxed{L}$
• $G$ is (partially or fully) d-separated from $Y$ conditional on $\boxed{L}$

Even though $G$ is d-separated from $Y$ given $L$, this doesn't mean $G$ can't modify the effect of $A$ on $Y$. The CATE for a specific value of $G$ can be expressed as:

$$\tau (g)=E[Y(1)-Y(0)|G=g]=E[E[Y(1)-Y(0)|L,G=g]|G=g]$$

Since $G$ is d-separated from $Y$ given $L$:

$$\tau (g)=E[E[Y(1)-Y(0)|L]|G=g]$$

This means $\tau (g)$ is a weighted average of the L-specific treatment effects, where the weights are determined by the distribution of L given $G=g$.

If two conditions are met:

1. The effect of $A$ on $Y$ varies across levels of $L$
2. The distribution of $L$ varies with $G$ (which it does, since $G\to L$)

Then $\tau (g)$ will vary with $g$, indicating effect modification by $G$.

The investigators would be wrong to equate d-separation with absence of effect modification. Although $G$ doesn't directly affect $Y$ after conditioning on $L$, $G$ can still modify the effect of $A$ on $Y$ through its influence on the distribution of $L$.

Week 7: In-Class Test 1 (20%)

22 April 2026

This week is the first in-class test, covering material from weeks 1–6.

What is covered

• Causal diagrams: elementary structures (week 2)
• Confounding bias structures (week 3)
• Interaction, measurement bias, and selection bias (week 4)
• Average treatment effects and the three fundamental assumptions (week 5)
• Effect modification and CATE (week 6)

Format

• Duration: 50 minutes (allocated time: 1 hour 50 minutes)
• Closed book, no devices
• Bring a pen or pencil
• Location: EA120 (the usual seminar room)

Week 8: Heterogeneous Treatment Effects and Machine Learning

Date: 29 Apr 2026

Why worry about heterogeneity?

Relying on the average treatment effect (ATE) is a bit like handing out size-nine shoes to an entire student body: on average they might fit, but watch the tall students hobble and the small ones trip.

Today we focus on the causal question: "What would be the effects on multi-dimensional well-being if everyone spent at least one hour a week socialising with their community?"

A one-hour boost in weekly community socialising could send some students' sense of belonging soaring while leaving others unmoved. Spotting that spread, measuring how large it really is, and deciding whether it is worth tailoring an exposure to individual "shoe sizes" are the three practical goals of heterogeneous treatment effects analysis.

1. Start with estimating the average treatment effect (ATE)

Assume the three fundamental assumptions of causal inference (consistency, exchangeability, positivity) are met. Suppose we wish to estimate the average treatment effect for socialising with one's community.

We begin with the most straightforward (and secretly impossible) counterfactual: run two parallel universes, one where everyone gets the treatment, another where no one does, and compare the final scores. The resulting difference is the average treatment effect:

$$\text{ATE}=E[Y(1)-Y(0)].$$

This gives us the average response. You have seen this before.

2. Do effects differ across people?

Variation is captured by the conditional average treatment effect (CATE),

$$\tau (x)=E[Y(1)-Y(0)\mid X=x],$$

where $X$ gathers pre-treatment covariates: age, baseline wellbeing, personality, and whatever else we have measured and included in our model. Normally these will be our baseline confounders.

If $\tau (x)$ turns out to be flat, we say there is no evidence for heterogeneity worth targeting.

People differ in countless, overlapping ways. Think of age, baseline wellbeing, personality traits, study habits, and more. A linear interaction model tests whether the treatment works differently along one straight dimension, such as gender, by fitting a straight line. But real-world data often twist and turn. If the true relationship bends like a garden hose, a straight line will miss the curve.

Regression forests fix this by letting the data place splits wherever the shape changes, so they can follow any bends that appear (Wager and Athey, 2018). Straight-line models are fine for simple patterns, but regression forests can trace the curves that simple lines overlook.

Causal forests are based on regression forests, where the splitting attempts to maximise differences in causal effect estimates. What this means will soon be clear.

3. From straight lines to trees

Traditional "parametric" models (like simple regression) guess a single functional shape, often a straight line, before seeing the data. A non-parametric model, by contrast, lets the data decide the shape. A regression tree is the simplest non-parametric learner we will use.

Regression tree

The idea is to split the covariate space by asking yes/no questions ("Age $\le$ 20?", "Baseline wellbeing $>0.3$?") until each terminal leaf is fairly homogeneous. Inside a leaf the predicted outcome is just the sample mean, so the tree builds a piecewise-constant surface instead of a global line. Think of tiling a garden with stepping stones: each stone is flat, but taken together they follow the ground's contours.

Regression forest

A single tree is quick and interpretable but unstable: small changes in the data can move the splits and shift predictions. A random forest grows many trees on bootstrap samples and averages their outputs. Averaging cancels much of the noise (Breiman, 2001).

Causal forest

To estimate treatment effects rather than outcomes, each tree plays a two-step "honest" game (Wager and Athey, 2018). First, one half of its sample is used to choose splits that separate treated from control units. Second, the other half is used to compute treatment-control differences within every leaf.

For a new individual with covariates $x_i$, each tree supplies a noisy leaf-level effect; the forest reports the average, written

$$\hat{\tau }(x)=E[Y(1)-Y(0)\mid X=x].$$

Because the noisy estimates point in many directions, their average is markedly less variable. The wisdom of trees is a wisdom of crowds.

Straight-line models suit simple patterns; regression forests flex to any bends; causal forests add a third dimension, namely variation in treatment responses.

For more about causal forests, see the lecture by Susan Athey (starting around 18 minutes in): https://www.youtube.com/watch?v=YBbnCDRCcAI

4. Building honest trees: avoiding overfitting

Sample splitting means partitioning your data into training and testing sets. This avoids overfitting the model to observations. Remember, we seek to estimate parameters for an entire population under two different exposures, at most only one of which is observed on any individual. Sample splitting is a feature of estimation in causal forests: we separate model selection from estimation.

Moreover, the forest adds a second safeguard: out-of-bag (OOB) prediction. Each $\hat{\tau }(x_i)$ is averaged only over trees that never used $i$ in their split phase. Together, honesty and OOB prediction deliver reliable uncertainty estimates even in high-dimensional settings (i.e. settings with many covariates).

5. Handling missing data

The grf package adopts Missing Incorporated in Attributes (MIA) splitting. "Missing" can itself become a branch, so cases are neither discarded nor randomly imputed. This pragmatic approach keeps all observations in play while preserving the forest's interpretability.

6. Is the heterogeneity actionable? RATE statistics

Once we have a personalised score $\hat{\tau }(x)$ for every unit, the practical question is whether targeting high scorers delivers a benefit large enough to justify the extra effort. The tool of choice is the Targeting-Operator Characteristic (TOC) curve:

$$G(q)=\frac{1}{n}\sum _{i=1}^{\lfloor qn\rfloor }{\hat{\tau }}_{(i)},0\le q\le 1,$$

where ${\hat{\tau }}_{(1)}\ge {\hat{\tau }}_{(2)}\ge \cdots$ are the estimated effects sorted from largest to smallest. The horizontal axis $q$ is the fraction of the population we would treat; the vertical axis $G(q)$ is the cumulative gain we expect from treating that top slice.

Two integrals of the TOC curve summarise how lucrative targeting could be.

RATE AUTOC (Area Under the TOC) puts equal weight on every $q$. This answers: if benefits are concentrated among the very best prospects, how much can we harvest by cherry-picking them?

RATE Qini applies heavier weight to the mid-range of $q$. This is the go-to metric when investigators face a fixed, moderate-sized budget, say, "we can afford to treat 40% of individuals; will targeting help?" (Yadlowsky et al., 2021). We will evaluate the curve at treatment of 20% and 50% of the population.

To quantify the economic or policy value of heterogeneity, rank units by $\hat{\tau }(x)$ and draw a TOC curve that plots cumulative gain against the fraction $q$ of the population treated.

Summary

Our workflow in this lecture answers three questions in sequence:

1. Is there substantial heterogeneity? Reject $H_0:\tau (x)$ constant if RATE AUTOC or RATE Qini is positive and statistically reliable.
2. Does targeting pay at realistic budgets? Inspect the slope of the Qini curve around plausible coverage levels.
3. Can we express the targeting rule in a few defensible steps? Fit and validate a shallow policy tree.

In the lab section you will reproduce each stage on a simulated dataset. Next week we turn to the applied side of RATE statistics: interpreting AUTOC curves, Qini curves, and building policy trees from causal forest output.

Lab materials: Lab 8: RATE and QINI Curves

Week 9: Resource Allocation and Policy Trees

Date: 6 May 2026

Recap: RATE Statistics and the TOC Curve

In Week 8 we introduced the machinery for deciding whether heterogeneity in treatment effects is large enough to act on. The central tool is the Targeting-Operator Characteristic (TOC) curve, which plots cumulative gain on the vertical axis against the fraction $q$ of the population treated on the horizontal axis, after ranking individuals by their estimated CATE $\hat{\tau }(x)$ from largest to smallest:

$$G(q)=\frac{1}{n}\sum _{i=1}^{\lfloor qn\rfloor }{\hat{\tau }}_{(i)},0\le q\le 1.$$

Two summary statistics integrate the TOC curve into a single number:

• RATE AUTOC (Area Under the TOC) weights every fraction $q$ equally, answering: if benefits concentrate among the best prospects, how much can we harvest by selecting them?
• RATE Qini weights the mid-range of $q$ more heavily, suiting investigators who face a fixed, moderate-sized budget (Yadlowsky et al. 2021). We evaluate the curve at treatment of 20% and 50% of the population.

A positive, statistically reliable RATE statistic tells us that targeting outperforms blanket treatment. This week we apply these statistics to worked examples and extend the pipeline with Qini curves and policy trees.

Section 7: RATE AUTOC Example

Although out-of-bag (OOB) predictions are "out-of-sample" for individual trees, the full forest still reuses information. A simple remedy when estimating the RATE AUTOC and Qini is to split the data, training the forest on one fold and testing RATE/Qini on the other. This explicit splitting blocks optimistic bias and yields honest test statistics, including confidence intervals (GRF Labs 2024).

The RATE AUTOC curve for the effect of hours socialising on sense of meaning illustrates the typical pattern. The curve rises steeply at first: a small, correctly targeted programme could deliver large gains. Past about 30% of the sample the curve dips below zero, meaning that beyond that threshold targeting the CATE may perform worse than simply applying the ATE to everyone. The initial steepness reflects concentration of benefit among a minority of individuals, while the subsequent decline shows that extending the programme to those with small or negative estimated effects dilutes its value.

Section 8: Visualising Policy Value with the Qini Curve

A Qini curve displays cumulative benefit on the vertical axis and treatment coverage (percentage of the population treated) on the horizontal axis. As with the AUTOC curve, we use a held-out test fold to validate the response curve.

For the effect of hours socialising on social belonging, the Qini curve shows that focussing on the top 20% of individuals nets a gain of roughly 0.08 units (95% CI 0.04--0.12). Widening coverage to 50% increases the cumulative gain to approximately 0.13 units (95% CI 0.07--0.19). Beyond that point the curve flattens: once we have treated everyone who offers a meaningful return, additional coverage adds little.

The Qini curve therefore translates CATE estimates into a practical budgeting tool. A policy-maker can read off the expected gain at any feasible coverage level and weigh it against cost.

Section 9: From a Black Box to Simple Rules --- Policy Trees

The causal forest produces a personalised CATE for every individual, mapping a high-dimensional covariate vector $X$ to a number $\hat{\tau }(X)$. Useful as that forecast may be, it stops short of telling us what to do. The function itself is too tangled (thousands of overlapping splits) to translate directly into a policy.

The policytree algorithm bridges that gap by collapsing the forest's $\hat{\tau }(X)$ values into a single, shallow decision tree whose depth you choose. Each split is chosen to maximise expected benefit (Sverdrup et al. 2024). In this course we cap the depth at two, for three reasons:

1. At most three yes/no questions per rule, so the logic fits on a slide you can present to policy-makers.
2. Each leaf still contains enough observations to yield a stable effect estimate.
3. Deeper trees increase computational complexity faster than they improve payoffs.

Policy tree example: effect of hours socialising on social belonging

The decision tree for social belonging first splits participants by self-esteem at $-0.925$ (original scale: 3.958). For those with self-esteem at or below this threshold, the next split is by neuroticism at $0.642$ (original scale: 4.228). Within that subgroup, individuals with neuroticism at or below the threshold are recommended control, while those above are recommended treated.

For participants with self-esteem above $-0.925$, the second split is by social belonging at $0.776$ (original scale: 5.972). In this subgroup, individuals with social belonging at or below the threshold are recommended treated, while those above are recommended control.

The corresponding policy map plots each individual in the two-dimensional covariate space defined by the tree's splitting variables, with colour indicating the recommended treatment assignment. The vertical and horizontal lines mark the split thresholds, carving the space into four rectangular regions. This visualisation makes the tree's logic immediately legible: one can see at a glance which combinations of covariates lead to a treatment recommendation and which do not. Because predictions are generated out of training sample, the map also serves as an informal check on how well the rule generalises.

Section 10: Ethical and Practical Considerations

There is no guarantee that statistical optimality will align with social optimality. A rule that maximises expected health gains might still be unaffordable for a public agency, unfair to a protected group, or opaque to those asked to trust it. We all have our notions of fairness, and we cannot be expected to ignore them. Moreover, the estimation of CATE is always sensitive to which variables we include in our model (see the caveats in Week 6 regarding evidence for effect modification relative to the inclusion of other variables).

We should not consider CATE an absolute guide to practice. Caution is warranted.

Yet the very same CATE machinery that powers targeting also helps science move past a one-size-fits-all mindset. By mapping treatment effects across a high-dimensional covariate space, we can test whether our favourite categories (gender, age group, clinical severity) actually capture the differences that matter. Sometimes they do; often they do not, revealing that nature is not carved at the joints of our folk classifications. Discovering where the forest finds meaningful splits can generate fresh psychological hypotheses about who responds, why, and under what circumstances, even when no policy decision is on the table. Over the coming weeks we shall return to this point with examples.

Summary and Next Steps

Our workflow answers three questions in sequence:

1. Is there substantial heterogeneity? Reject $H_0:\tau (x)$ constant if RATE AUTOC or RATE Qini is positive and statistically reliable.
2. Does targeting pay at realistic budgets? Inspect the slope of the Qini curve around plausible coverage levels.
3. Can we express the targeting rule in a few defensible steps? Fit and validate a shallow policy tree.

In the lab section you will reproduce each stage on a simulated dataset.

Lab materials: Lab 9: Policy Trees

Week 10: Classical Measurement Theory from a Causal Perspective

Date: 13 May 2026

Overview

By the conclusion of this session, you will gain proficiency in:

1. Exploratory Factor Analysis (EFA),
2. Confirmatory Factor Analysis (CFA),
3. Multigroup Confirmatory Factor Analysis,
4. Partial Invariance (configural, metric, and scalar equivalence).

We will learn these concepts by working through an analysis.

Part 1: Factor Analysis and Measurement Invariance

Focus on Kessler-6 Anxiety

The code below will:

1. Load required packages.
2. Select the Kessler 6 items.
3. Check whether there is sufficient correlation among the variables to support factor analysis.

Select a scale to validate: Kessler-6 Distress

# get synthetic data
library(margot)
library(tidyverse)
library(performance)

# select the columns of the kessler-6
dt_only_k6 <- df_nz |>
  filter(wave == 2018) |>
  select(
    kessler_depressed,
    kessler_effort,
    kessler_hopeless,
    kessler_worthless,
    kessler_nervous,
    kessler_restless
  )

# check factor structure
performance::check_factorstructure(dt_only_k6)

Practical definitions of the Kessler-6 items

The df_nz dataset is loaded with the margot package. It is a synthetic dataset. We take items from the Kessler-6 (K6) scale: depressed, effort, hopeless, worthless, nervous, and restless (Kessler et al. 2002; Kessler et al. 2010). The Kessler-6 is used as a diagnostic screening tool for depression by physicians in New Zealand.

The Kessler-6 (K6) is a widely used diagnostic screening tool designed to identify levels of psychological distress that may indicate mental health disorders such as depression and anxiety. Physicians in New Zealand use it to screen patients quickly (Krynen et al. 2013). Each item on the Kessler-6 asks respondents to reflect on their feelings and behaviours over the past 30 days, with responses provided on a five-point ordinal scale:

1. "...you feel hopeless": This item measures the frequency of feelings of hopelessness. It assesses a core symptom of depression, where the individual perceives little or no optimism about the future.

2. "...you feel so depressed that nothing could cheer you up": This statement gauges the depth of depressive feelings and the inability to gain pleasure from normally enjoyable activities, a condition known as anhedonia.

3. "...you feel restless or fidgety": This item evaluates agitation and physical restlessness, which are common in anxiety disorders but can also be present in depressive states.

4. "...you feel that everything was an effort": This query assesses feelings of fatigue or exhaustion with everyday tasks, reflecting the loss of energy that is frequently a component of depression.

5. "...you feel worthless": This item measures feelings of low self-esteem or self-worth, which are critical indicators of depressive disorders.

6. "...you feel nervous": This question is aimed at identifying symptoms of nervousness or anxiety, helping to pinpoint anxiety disorders.

Response options

The ordinal response options provided for the Kessler-6 are designed to capture the frequency of these symptoms, which is crucial for assessing the severity and persistence of psychological distress:

• 1. "None of the time": The symptom was not experienced at all.
• 2. "A little of the time": The symptom was experienced infrequently.
• 3. "Some of the time": The symptom was experienced occasionally.
• 4. "Most of the time": The symptom was experienced frequently.
• 5. "All of the time": The symptom was constantly experienced.

In clinical practice, higher scores on the Kessler-6 are indicative of greater distress and a higher probability of a mental health disorder. Physicians use a sum score of 13 to decide on further diagnostic evaluations or immediate therapeutic interventions.

The simplicity and quick administration of the Kessler-6 make it an effective tool for primary care settings in New Zealand, allowing for the early detection and management of mental health issues. Do the items in this scale cohere? Do they all relate to depression? Might we quantitatively evaluate "coherence" in this scale?

Exploratory Factor Analysis

We employ performance::check_factorstructure() to evaluate the data's suitability for factor analysis. Two tests are reported:

Bartlett's test of sphericity

Bartlett's Test of Sphericity is used in psychometrics to assess the appropriateness of factor analysis for a dataset. It tests the hypothesis that the observed correlation matrix is an identity matrix, which would suggest that all variables are orthogonal (i.e., uncorrelated) and therefore factor analysis would be inappropriate.

The outcome:

• Chi-square (Chisq): 50402.32 with degrees of freedom (15) and a p-value < .001

This result (p < .001) confirms that the observed correlation matrix is not an identity matrix, substantiating the factorability of the dataset.

Kaiser-Meyer-Olkin (KMO) measure

The KMO test assesses sampling adequacy by comparing the magnitudes of observed correlation coefficients to those of partial correlation coefficients. A KMO value nearing 1 indicates appropriateness for factor analysis. The results are:

• Overall KMO: 0.87. This value suggests good sampling adequacy, indicating that the sum of partial correlations is relatively low compared to the sum of correlations, thus supporting the potential for distinct and reliable factors.

Each item's KMO value exceeds the acceptable threshold of 0.5, so the data are suitable for factor analysis.

Explore factor structure

The following R code allows us to perform EFA on the Kessler-6 data, assuming three latent factors.

# exploratory factor analysis
# explore a factor structure made of 3 latent variables
library("psych")
library("parameters")

# do efa
efa <- psych::fa(dt_only_k6, nfactors = 3) |>
  model_parameters(sort = TRUE, threshold = "max")

print(efa)

What is "rotation"?

In factor analysis, rotation is a mathematical technique applied to the factor solution to make it more interpretable. Rotation is analogous to adjusting the angle of a camera to get a better view. When we rotate the factors, we are not changing the underlying data, just how we are looking at it, to make the relationships between variables and factors clearer and more meaningful.

The main goal of rotation is to achieve a simpler and more interpretable factor structure. This simplification is achieved by making the factors as distinct as possible, by aligning them closer with specific variables, which makes it easier to understand what each factor represents.

There are two types:

Orthogonal rotations (such as Varimax) assume that the factors are uncorrelated and keep the axes at 90 degrees to each other. This is useful when we assume that the underlying factors are independent.

Oblique rotations (such as Oblimin) allow the factors to correlate. This is more realistic in psychological and social sciences because constructs such as stress and anxiety may naturally correlate with each other, so oblique rotation is often a better option.

Results

Using oblimin rotation, the items loaded as follows on the three factors:

• MR3: Strongly associated with kessler_hopeless (0.79) and kessler_worthless (0.79). This factor might capture aspects related to feelings of hopelessness and worthlessness, often linked with depressive affect.
• MR1: Mostly linked with kessler_depressed (0.99), suggesting this factor represents core depressive symptoms.
• MR2: Includes kessler_restless (0.72), kessler_nervous (0.43), and kessler_effort (0.38). This factor seems to encompass symptoms related to anxiety and agitation.

The complexity values indicate the number of factors each item loads on "significantly." A complexity near 1.00 suggests that the item predominantly loads on a single factor, which is seen with most items except for kessler_nervous and kessler_effort, which show higher complexity and thus share variance with more than one factor.

Uniqueness values represent the variance in each item not explained by the common factors. Lower uniqueness values for items like kessler_depressed indicate that the factor explains most of the variance for that item.

Variance explained

The three factors together account for 62.94% of the total variance in the data, distributed as follows:

• MR3: 28.20%
• MR1: 17.56%
• MR2: 17.18%

This indicates a substantial explanation of the data's variance by the model, with the highest contribution from the factor associated with hopelessness and worthlessness.

Consensus view?

There are different algorithms for assessing the factor structure. The performance package allows us to consider a "consensus" view.

n <- n_factors(dt_only_k6)
n
# plot
plot(n) + theme_classic()

Output:

The choice of 1 dimensions is supported by 8 (50.00%) methods out of 16 (Optimal coordinates, Acceleration factor, Parallel analysis, Kaiser criterion, Scree (SE), Scree (R2), VSS complexity 1, Velicer's MAP).

The result indicates that a single dimension is supported by half of the methods used (8 out of 16). However, science is not a matter of voting. Also, does it make sense that there is one latent factor here?

Confirmatory Factor Analysis (ignoring groups)

CFA validates the hypothesised factor structures derived from EFA.

• One-factor model: assumes all items measure a single underlying construct.
• Two-factor model: assumes two distinct constructs measured by the items.
• Three-factor model: assumes three distinct constructs measured by the items.

1. Data partition

First, we take the dataset (dt_only_k6) and partition it into training and testing sets. This division helps in validating the model built on the training data against an unseen test set, enhancing robustness of the factor analysis findings.

set.seed(123)
part_data <- datawizard::data_partition(dt_only_k6, training_proportion = .7, seed = 123)
training <- part_data$p_0.7
test <- part_data$test

2. Model setup for CFA

Based on the EFA results, we consider three different factor structures. We fit each model to the training data:

# one factor model
structure_k6_one <- psych::fa(training, nfactors = 1) |>
  efa_to_cfa()

# two factor model
structure_k6_two <- psych::fa(training, nfactors = 2) |>
  efa_to_cfa()

# three factor model
structure_k6_three <- psych::fa(training, nfactors = 3) |>
  efa_to_cfa()

# inspect models
structure_k6_one
structure_k6_two
structure_k6_three

• One-factor model: All items are linked to a single factor (MR1).

• Two-factor model:

  • MR1 is linked with kessler_depressed, kessler_hopeless, and kessler_worthless, suggesting these items might represent a more depressive aspect of distress.
  • MR2 is associated with kessler_effort, kessler_nervous, and kessler_restless, which could indicate a different aspect, perhaps related to anxiety or agitation.

• Three-factor model:

  • MR1 includes kessler_depressed, kessler_effort, kessler_hopeless, and kessler_worthless, indicating a broad factor possibly encompassing overall distress.
  • MR2 consists solely of kessler_effort.
  • MR3 includes kessler_nervous and kessler_restless, which might imply these are distinctive from other distress components.

Do these results make sense? Note they are different from the EFA. Why might that be?

Next we perform the confirmatory factor analysis itself:

# fit and compare models
library(lavaan)

# one latent model
one_latent <-
  suppressWarnings(lavaan::cfa(structure_k6_one, data = test))

# two latents model
two_latents <-
  suppressWarnings(lavaan::cfa(structure_k6_two, data = test))

# three latents model
three_latents <-
  suppressWarnings(lavaan::cfa(structure_k6_three, data = test))

# compare models
compare <-
  performance::compare_performance(one_latent, two_latents, three_latents, verbose = FALSE)

# select cols we want
key_columns_df <- compare[, c("Model", "Chi2", "Chi2_df", "CFI", "RMSEA",
                               "RMSEA_CI_low", "RMSEA_CI_high", "AIC", "BIC")]

# view as table
as.data.frame(key_columns_df) |>
  kbl(format = "markdown")

Metrics

• Chi2 (Chi-Square Test): A lower Chi2 value indicates a better fit of the model to the data.
• df (Degrees of Freedom): Reflects the model complexity.
• CFI (Comparative Fit Index): Values closer to 1 indicate a better fit. A value above 0.95 is generally considered good.
• RMSEA (Root Mean Square Error of Approximation): Values less than 0.05 indicate a good fit, and values up to 0.08 are acceptable.
• AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion): Lower values are better, indicating a more parsimonious model, with BIC imposing a penalty for model complexity.

Model selection

• The three latents model shows the best fit across all indicators, having the lowest Chi2, RMSEA, and the highest CFI. It also has the lowest AIC and BIC scores.

However:

• The CFI for the two-factor model is 0.990, which is close to 1 and suggests a very good fit. This is superior to the one-factor model (CFI = 0.947) and slightly less than the three-factor model (CFI = 0.995).
• The RMSEA for the two-factor model is 0.044, which is within the excellent fit range (below 0.05). It improves upon the one-factor model's RMSEA of 0.099 and is only slightly worse than the three-factor model's 0.031.
• BIC is not very different.

The two-factor model strikes a balance between simplicity and model fit. It has fewer factors than the three-factor model, making it potentially easier to interpret while still capturing the variance in the data.

Does anxiety appear to differ from depression?

Measurement Invariance in Multigroup Confirmatory Factor Analysis

When we use tools like surveys or tests to measure psychological constructs (such as distress, intelligence, satisfaction), we often need to ensure that these tools work similarly across different groups of people. This is crucial for fair comparisons.

Levels of measurement invariance

Measurement invariance tests how consistently a measure operates across different groups, such as ethnic or gender groups. Consider how we can understand it through the K6 Distress Scale applied to different demographic groups in New Zealand:

1. Configural invariance

The measure's structure is the same across groups. For the K6 Distress Scale, both Maori and New Zealand Europeans use the same six questions to gauge distress. However, how strongly each question predicts distress can vary between the groups. (Note we are using "prediction" here; we are only assessing associations.)

2. Metric invariance

The strength of the relationship between each question and the construct (distress) is consistent across groups. If metric invariance holds, a unit change in the latent distress factor affects scores on questions (such as feeling nervous or hopeless) equally for Maori and New Zealand Europeans.

3. Scalar invariance

Beyond having the same structure and relationships, everyone starts from the same baseline. If scalar invariance holds, the actual scores on the distress scale mean the same thing across different groups. If a Maori participant scores 15 and a New Zealander of European descent scores 15, both are experiencing a comparable level of distress.

Concept of "partial invariance"

Sometimes, not all conditions for full metric or scalar invariance are met, which could hinder meaningful comparisons across groups. This is where the concept of "partial invariance" comes into play.

Partial invariance occurs when invariance holds for some but not all items of the scale across groups.

• Metric partial invariance: Most items relate similarly to the underlying factor across groups, but one or two do not. Researchers might decide that there is enough similarity to proceed with comparisons of relationships (correlations) but should proceed with caution.

• Scalar partial invariance: Most but not all items show the same intercepts across groups. It suggests that while comparisons of the construct means can be made, some scores might need adjustments or nuanced interpretation.

In practical terms, achieving partial invariance allows for some comparisons but signals a need for careful interpretation. For instance, if partial scalar invariance is found on the K6 Distress Scale, researchers might compare overall distress levels between Maori and New Zealand Europeans but should acknowledge that differences in certain item responses might reflect measurement bias rather than true differences in distress.

Understanding these levels of invariance helps ensure that when we compare mental health or other constructs across different groups, we are making fair and meaningful comparisons.

Running multigroup CFA

The following script runs multigroup confirmatory factor analysis (MG-CFA) to assess the invariance of the K6 distress scale across two ethnic groups: European New Zealanders and Maori.

# select needed columns plus 'ethnicity'
# filter dataset for only 'euro' and 'maori' ethnic categories
dt_eth_k6_eth <- df_nz |>
  filter(wave == 2018) |>
  filter(eth_cat == "euro" | eth_cat == "maori") |>
  select(kessler_depressed, kessler_effort, kessler_hopeless,
         kessler_worthless, kessler_nervous, kessler_restless, eth_cat)

# partition the dataset into training and test subsets
# stratify by ethnic category to ensure balanced representation
part_data_eth <- datawizard::data_partition(dt_eth_k6_eth,
  training_proportion = .7, seed = 123, group = "eth_cat")

training_eth <- part_data_eth$p_0.7
test_eth <- part_data_eth$test

# configural invariance models
# run cfa models specifying one, two, and three latent variables
# without constraining across groups
one_latent_eth_configural <- suppressWarnings(
  lavaan::cfa(structure_k6_one, group = "eth_cat", data = test_eth))
two_latents_eth_configural <- suppressWarnings(
  lavaan::cfa(structure_k6_two, group = "eth_cat", data = test_eth))
three_latents_eth_configural <- suppressWarnings(
  lavaan::cfa(structure_k6_three, group = "eth_cat", data = test_eth))

# compare model performances for configural invariance
compare_eth_configural <- performance::compare_performance(
  one_latent_eth_configural, two_latents_eth_configural,
  three_latents_eth_configural, verbose = FALSE)

# metric invariance models
# run cfa models holding factor loadings equal across groups
one_latent_eth_metric <- suppressWarnings(
  lavaan::cfa(structure_k6_one, group = "eth_cat",
              group.equal = "loadings", data = test_eth))
two_latents_eth_metric <- suppressWarnings(
  lavaan::cfa(structure_k6_two, group = "eth_cat",
              group.equal = "loadings", data = test_eth))
three_latents_eth_metric <- suppressWarnings(
  lavaan::cfa(structure_k6_three, group = "eth_cat",
              group.equal = "loadings", data = test_eth))

# compare model performances for metric invariance
compare_eth_metric <- performance::compare_performance(
  one_latent_eth_metric, two_latents_eth_metric,
  three_latents_eth_metric, verbose = FALSE)

# scalar invariance models
# run cfa models holding factor loadings and intercepts equal across groups
one_latent_eth_scalar <- suppressWarnings(
  lavaan::cfa(structure_k6_one, group = "eth_cat",
              group.equal = c("loadings", "intercepts"), data = test_eth))
two_latents_eth_scalar <- suppressWarnings(
  lavaan::cfa(structure_k6_two, group = "eth_cat",
              group.equal = c("loadings", "intercepts"), data = test_eth))
three_latents_eth_scalar <- suppressWarnings(
  lavaan::cfa(structure_k6_three, group = "eth_cat",
              group.equal = c("loadings", "intercepts"), data = test_eth))

# compare model performances for scalar invariance
compare_eth_scalar <- performance::compare_performance(
  one_latent_eth_scalar, two_latents_eth_scalar,
  three_latents_eth_scalar, verbose = FALSE)

Configural invariance results

The table represents the comparison of three MG-CFA models conducted to test for configural invariance across different ethnic categories. Configural invariance refers to whether the pattern of factor loadings is the same across groups. It is the most basic form of measurement invariance.

Looking at the results:

1. Chi2 (Chi-square): A lower value suggests a better model fit. The three-factor model performs best.
2. GFI (Goodness of Fit Index) and AGFI (Adjusted Goodness of Fit Index): Values range from 0 to 1, with values closer to 1 suggesting a better fit. All models are close.
3. NFI, NNFI (TLI), CFI: Range from 0 to 1, with values closer to 1 suggesting a better fit. The two- and three-factor models have the highest values.
4. RMSEA: Lower values are better, with values below 0.05 considered good and up to 0.08 considered acceptable. Only the two- and three-factor models meet this threshold.
5. AIC and BIC: The three-factor model has the lowest values.

Analysis of the results:

• One latent model: Shows the poorest fit among the models with a high RMSEA and the lowest CFI. The Chi-squared value is significantly high, indicating a substantial misfit with the observed data.
• Two latents model: Displays a much improved fit compared to the one-latent model, as evident from its much lower Chi-squared value, lower RMSEA, and higher CFI.
• Three latents model: Provides the best fit metrics among the three configurations.

Metric equivalence

Metric equivalence (also known as weak measurement invariance) tests whether factor loadings are equal across groups. The three-factor model again performs best.

Scalar equivalence

Scalar equivalence tests whether both factor loadings and intercepts are equal across groups. This is the strongest form of invariance and is required for meaningful mean comparisons across groups.

Overall, there is good evidence for the three-factor model of Kessler-6, but the two-factor model is close.

Consider: when might we prefer a two-factor model? When might we prefer a three-factor model? When might we prefer a one-factor model?

Conclusion: understanding the limits of association in factor models

This discussion of measurement invariance across different demographic groups underscores the reliance of factor models on the underlying associations in the data. These models are fundamentally descriptive, not prescriptive; they organise the observed data into a coherent structure based on correlations and assumed relationships among variables.

Factor analysis assumes that the latent variables (factors) causally influence the observed indicators. This is a strong assumption that can profoundly affect the interpretation of results. Understanding and critically evaluating our assumptions is essential when applying factor analysis to real-world scenarios.

The assumption that latent variables cause the observed indicators, rather than merely being associated with them, suggests a directional relationship that can affect decisions made based on the analysis. For instance, if we believe that a latent construct like psychological distress causally influences responses on the K6 scale, interventions might be designed to target this distress directly. However, if the relationship is more complex or bidirectional, such straightforward interventions might not be as effective.

Part 2: How Traditional Measurement Theory Fails

Overview

By the end of this section you will:

1. Understand the causal assumptions implied by the factor analytic interpretation of the formative and reflective models.
2. Be able to distinguish between statistical and structural interpretations of these models.
3. Understand why VanderWeele argues that consistent causal estimation is possible using the theory of multiple versions of treatments for constructs with multiple indicators.

Two ways of thinking about measurement in psychometric research

In psychometric research, formative and reflective models describe the relationship between latent variables and their respective indicators. VanderWeele discusses this in the assigned reading (VanderWeele 2022).

Reflective model (factor analysis)

In a reflective measurement model, also known as an effect indicator model, the latent variable is understood to cause the observed variables. In this model, changes in the latent variable cause changes in the observed variables. Each indicator (observed variable) is a "reflection" of the latent variable: they are effects or manifestations of the latent variable.

The reflective model may be expressed:

$$X_i={\lambda }_i\eta +{\epsilon }_i$$