Measurement And Selection Biases
Effect-Modification on Causal Directed Acyclic Graphs
The primary function of a causal directed acyclic graph is to allow investigators to apply Pearlâs backdoor adjustment theorem to evaluate whether causal effects may be identified from data, as shown in ?@tbl-terminologygeneral. We have noted that modifying a causal effect within one or more strata of the target population opens the possibility for biased average treatment effect estimates when the distribution of these effect modifiers differs in the analytic sample population (Bulbulia 2024b).
We do not generally represent non-linearities in causal directed acyclic graphs, which are tools for obtaining relationships of conditional and unconditional independence from assumed structural relationships encoded in a causal diagram that may lead to a non-causal treatment/outcome association (Bulbulia 2024a).
Table 1 presents our convention for highlighting a relationship of effect modification in settings where (1) we assume no confounding of treatment and outcome and (2) there is effect modification such that the effect of A on Y differs in at least one stratum of the target population.
To focus on effect modification, we do not draw a causal arrow from the direct effect modifier F to the outcome Y. This convention is specific to this article (refer to Hernan and Robins (2020), pp. 126â127, for a discussion of ânon-causalâ arrows).
Part 1: How Measurement Error Bias Makes Your Causal Inferences weird {#id-sec-1}
(wrongly estimated inferences due to inappropriate restriction and distortion)
Measurements record reality, but they are not always accurate. Whenever variables are measured with error, our results can be misleading. Every study must therefore consider how its measurements might mislead.
Causal graphs can deepen understanding becauseâas implied by the concept of ârecordââthere are structural or causal properties that give rise to measurement error. Measurement error can take various forms, each with distinct implications for causal inference:
- Independent (undirected) / uncorrelated: Errors in different variables do not influence each other.
- Independent (undirected) and correlated: Errors in different variables are related through a shared cause.
- Dependent (directed) and uncorrelated: Errors in one variable influence the measurement of another, but these influences are not related through a shared cause.
- Dependent (directed) and correlated: Errors in one variable influence the measurement of another, and these influences are related through a shared cause (HernĂĄn and Cole 2009; VanderWeele and HernĂĄn 2012).
The six causal diagrams presented in Table 2 illustrate structural features of measurement error bias and clarify how these structural features compromise causal inferences.
Effect-Modification Examples of measurement error bias
Understanding these structural features will help explain why measurement error bias cannot typically be evaluated with statistical models, and will prepare us to link target-population restriction biases to measurement error.
Summary
In Part 1, we examined independent, correlated, dependent, and correlatedâdependent forms of measurement error bias. These structural features clarify why such biases threaten causal inference and often cannot be resolved with statistical adjustment alone (VanderWeele and HernĂĄn 2012).
We return to measurement error in Part 4.
Part 2: Target Population Restriction Bias at the End of Study {#id-sec-2}
Suppose the analytic sample matches the target population at baseline. Attrition (right-censoring) may bias causal effect estimates by:
1) opening biasing pathways (distortion), or
2) restricting the analytic sample so it is no longer representative (restriction).
Selection-bias over timen Five examples of right-censoring bias.
Example 1: Confounding by common cause of treatment and attrition
Table 3 \mathcal{G}_1 illustrates confounding by common cause of treatment and outcome in the censored such that the potential outcomes of the population at baseline Y(a) may differ from those of the censored population at the end of study Y'(a), so Y'(a) \neq Y(a). Suppose investigators ask whether religious service attendance affects volunteering, and an unmeasured variable (loyalty) affects attendance, attrition, and volunteeringâopening a backdoor path.
We have encountered this bias before: the structure matches correlated measurement errors (Table 2 \mathcal{G}_3). Attrition may exacerbate measurement error bias by opening a path A \;\associationred\; U \;\associationred\; U_{\Delta A} \;\associationred\; Y'.
Example 2: Treatment affects censoring
Table 3 \mathcal{G}_2 illustrates bias in which the treatment affects the censoring process. Here, the treatment causally affects the outcome reporter but not the outcome itself.
Example: In a meditation trial with no true effect on well-being, Buddha-like detachment increases attrition and also changes how well-being is reported. This opens a path A \;\associationred\; U_{\Delta{A\to Y}} \;\associationred\; Y' (not confounding; no common cause of A and Y). Structurally, this is directed uncorrelated measurement error (Table 2 \mathcal{G}_4). Results risk distortion via end-of-study restriction.
Example 3: No treatment effect when outcome causes censoring
Table 3 \mathcal{G}_3 shows outcome-driven censoring under the sharp null. In theory, the ATE may remain unbiased, though the analytic sample is restricted. This corresponds to undirected uncorrelated measurement error (Table 2 \mathcal{G}_1). In practice the sharp-null assumption is untestable and rarely known in advance.
Example 4: Treatment effect when outcome causes censoring and a true effect exists
Table 3 \mathcal{G}_4 shows that if the outcome affects censoring in the presence of a true effect, bias arises (at least on one effect scale). This structure is equivalent to measurement error bias and can occur without confounding. See the worked example in Part 4.
Example 5: Treatment effect and effect-modifiers differ in censored group (restriction bias without confounding)
Table 3 \mathcal{G}_5 represents a setting with a true treatment effect, but the distribution of effect modifiers differs at study end. If missingness is at random and models are correctly specified, inverse probability weighting or multiple imputation can recover valid estimates (Cole and HernĂĄn 2008; Leyrat et al. 2021; Shiba and Kawahara 2021). If not (e.g., MNAR or model misspecification), causal estimation is compromised (Tchetgen Tchetgen and Wirth 2017; Malinsky, Shpitser, and Tchetgen Tchetgen 2022).
Note that Table 3 \mathcal{G}_5 resembles Table 2 \mathcal{G}_2. Replacing unmeasured effect modifiers \circledotted{F} and U_{\Delta F} by \circledotted{U_Y} shows the link to uncorrelated independent measurement error âoff the nullâ.
In this setting there may be a common cause of A and Y, and, additionally, the end-of-study analytic sample is an undesirable restriction of the target population: marginal effects differ between the restricted sample and the target (see Supplement S4 for a simulation). Hence results can be weird due to inappropriate restriction.
Summary
Right-censoring can bias effect estimates by changing the distribution of effect modifiers between baseline and study end. Investigators should ensure the end-of-study potential outcomes distribution aligns with the target population. Methods such as inverse probability weighting and multiple imputation can mitigate this bias (Bulbulia 2024c), subject to their assumptions.
The take-home message: attrition is nearly inevitable; if unchecked, it yields weird results (wrongly estimated inferences due to inappropriate restriction and distortion). See Supplement S3 for a formal explanation and S4 for a simulation.
Part 3: Target Population Restriction Bias at the Start of Study
Targetârestriction bias occurs when the analytic sample at baseline differs from the target population in the distribution of confounders and/or treatmentâeffect modifiers. Misalignment may arise if the source population does not match the target, or if study selection alters distributions. Alignment cannot generally be verified from data (see Supplement S3).
ColliderâRestriction Bias at Baseline
In Table 4 \mathcal{G}_1, unmeasured health awareness (U_1) influences both activity (A) and participation (S=1), and unmeasured SES (U_2) influences both heart health (Y) and S=1. Conditioning on S=1 opens paths: - U_1: Overârepresentation of active individuals, overstating benefits. - U_2: Confounding path via SES, inflating effect estimates.
Adjusting for U_1, U_2, or proxies can block these paths (Table 4 \mathcal{G}_2).
Restriction Bias Without Collider Stratification
Example 1: WEIRD Sample, NonâWEIRD Target
If effect modifiers differ between a WEIRD sample and general population (Table 5 \mathcal{G}_{1.1}), estimates may be biased without confounding . Structure matches Table 3 \mathcal{G}_5 and Table 2 \mathcal{G}_2. Known effectâmodifier distributions allow weighting , but mapping from restricted to target effects (f_W) is usually unknown .
Example 2: Overly Broad Sample for Narrow Target
If the target is a restricted stratum (e.g., NZ men > 40 without vasectomy) but sampling is broader (Table 5 \mathcal{G}_{2.1}), bias mirrors rightâcensoring with effect modifiers. Correct restriction (Table 5 \mathcal{G}_{2.2}) aligns sample with target.